Exploratory Gesture Visualizations

Most gesture research articles show gestures as video snapshots, tracings of video snapshots, or time-aligned blocks as seen in the multimedia annotation software like ELAN or ANVIL. Not much has changed since the beginning of gesture research using microfilm. Partially to support our hypotheses and partially to see what’s going on in a broad visual sweep, I designed a growing set of visualizations to show trajectory movement, to find patterns in gesture phasing, and to investigate gestural alignment with the transcript. These encompassed a variety of tools and skills that I have refined over the years. They include: HTML, CSS, Perl, Processing (see processing.org), hand illustration, and video manipulation.

Gesture Phases

Gesture phases simplifed as time-aligned polygons. The preparation phase is an upwards sloping triangle, the static phases of holds and relaxeds are long rectangles, and the relaxation or recovery phase is a downwards sloping triangle. The stroke is a large rectangle with a small circle in the center to draw the eyes to them. The blue bars on top show preliminary grouped gesture labels, while the light orange background behind each set of phases show the stroke-defined gesture as a whole.

Discourse and Timing

This visualization shows the perceptual gesture groupings in the text. The text below takes the transcribed syllables–which is why most words appear misspelled–and checks whether the syllable is in a grouping (alternating colored red and teal) or in a region of video where the speaker is occluded by a presentation slide (grey). Since this is an HTML file, there is some degree of interactivity. You can hover over each syllable to get information about the location of the syllable in the video, and which grouping it overlaps with. I included a 99 millisecond margin of error, which is the equivalent of 3 frames of video in a 30fps video. The data is exported from ELAN as a tab-delimited text file, edited, and I used Perl to write it into HTML.

Hand Tracking Video

We explored the possibility of using the velocity of the hands to help easily label gesture features such as different phases, the curvature of the motion, among other possibilities. Using ProAnalyst, we tracked the index finger on both hands, subtracted the body movement, extracted the positioning data, plugged it into a Processing script for the visualization, added a line of code to turn each 1/30 second frame into an image, used a time-lapse utility to turn it into a video . It’s beautiful.