This problem is not unique to neuroethology. For example during angiography one injects dye into the cardiac circulation to visualize obstructions of the coronary arteries. The diameters of these arteries is also determined by the state of contraction of the cardiac muscle. Thus the image of the arteries must be obtained at a particular point in the cardiac cycle which is defined by the electrical signal the electrocardiogram. Thus angiographic analysis could be enhanced by multi-media techniques. Another application of these techniques is the analysis of performance in sports medicine.
Neuroethological analysis presents a unique problem since the most interesting behavior is typically spontaneous. In other words, under the most carefully designed and controlled experimental conditions the animal will exercise its own volition. Thus the experimenter can only create conditions which will optimize the occurrence of the behavior in question in the sampled data. As a result, acquisition devices must be capable of acquiring large amounts of image and neuro-physiological data.
Mulitmedia neuroethological experimentation is, at present, a multi-stage process involving acquisition, browsing, cataloging, digitization, analysis and ultimately visualization displays. Videotaped data must first be browsed and interesting epochs identified. To be subject to automated analysis, the interesting epochs must be digitized. The kinematic raw data can be stored as digitized "movies" while analog data is typically stored as binary files. Digital movies must be subjected to frame-by-frame analysis of the relevant kinematic parameters and the electrophysiological data may be subjected signal processing to extract relevant electrophysiological parameters or viewed in raw format. Finally, the results of kinematic analysis and signal processing must be presented in graphical displays which allow visualization of the interrelations between electrophysiology and behavior.
The above discussion was intended to introduce the fundamentals of neuroethological analysis . I will now address current technology and how we expect to to proceed in the future. The optimal device for acquisition is the High-8 video tape transport. Current technology acquires high resolution (>400 lines horizontal resolution) video images and digital or pulse-code modulated stereo audio. During neurophysiological experimentation. one can simultaneously record the kinematic information on the video signal and electrophysiological data on the analog tracks. The analog recording circuits of most consumer grade transports damp transients of less than 40 hz so if one wants to use this technology to record slower phenomena there are two alternatives. One approach is to use voltage to frequency converters to convert the slowly varying signal to a frequency. During playback, a frequency to voltage converter can be utilized to decode the frequency back to a voltage. Several vendors have developed multichannel versions of this technology which can also record multiple analog channels on the video signal of the tape. An alternative technology is the use of digital meters which superimpose a digital version of the voltage level on the video signal.
Browsing and cataloging of videotape-based data can be readily accomplished in the Macintosh environment. Abbate Video Consultants has developed a Hypercard-based videotape logging and assembly system which operates through the Control-L or VISCA interfaces of Sony VCR's and CamCorders and has been extended to a broad variety of transports and, in fact the Windows environment. The VideoToolkit environment allows interactive browsing of videotape data using both mouse and keypad controls to establish a Hypercard database of the interesting epochs within a tape. This system provides a ready solution to the most fundamental problem of neuroethological analysis; i.e. keeping track of sparce results in a sea of data.
As a platform for the analysis of video-based neuroethological data, we extended the NIH Image program to support color acquisition and video tape control. The color segmentation and VCR control capabilities of an earlier version of ColorImage developed in collaboration Garth Fletcher have been described in an earlier article in Advanced Imaging (Nov, 1990). An example of the sort of direct kinematic analysis supported by ColorImage is indicated in Figure. 1. In this experiment we quantified the dyamics of the movement of the digestive apparatus of a larval lobster. The quantified parameter was the length of the pyloric region of the lobsters stomach. ColorImage allowed us to digitize a videotape of lobster feeding behavior, create a digital movie and to determine the length of the pylorus using a click-drag procedure in each of the frames of the movie.. The results of this analyis are presented graphically in Fig. 1.
The final stage of neuroethological analysis is the visualization of multimedia data. Figure 3 indicates a graph of three media, two channels of electromyograms digitized from the stereo audio channels, a kinematic analysis of the progress of flexions (here regions of curvature maxima) down the body and the instantaneous swimming thrust derived from a digital meter as in figure 2. ColorImage acquired the video and analog data using the rewind and search mechanism described above. It then automatically traced the shape of the lamprey and entered the thrust from each frame. A second program performed curvature analysis on the lamprey shapes and generated a file of the locus (position) of flexions and the instantaneous thrust in each frame of the movie. A third program read the analog electromyogram files and the locus/thrust file and generated figure 3 as a MacPaint bitmap file.
One disadvantage of QuickTime as presently implemented is the constriction of small image sizes and inherent resolution in kinematic measurements. DataTranslation has recently announced a product, the Media 100 non-linear editing syhstem which transcends the disadvantages inherent in small windows allowing full motion video and analog with 640x480 windows at video frame rates. The combination of this product with ColorImage may provide the ultimate solution for multimedia neuroethology.
Joseph Ayers is the Director of the Marine Science Center, Northeastern University, East Point, Nahant, MA 01908, (781) 581-7370. His research in motion analysis is supported by NSF grant DIR 8910-7532 while his research in the neuroethology of larval lobsters is supported by NSF grant NS IBN-9121224