Short Version Gesture Coding Manual

//Short Version Gesture Coding Manual
Short Version Gesture Coding Manual 2018-01-19T03:35:34+00:00

Setup

Make sure you know the basics of getting started with and using ELAN (Google ELAN Max Planck). Most labeling is done with the sound turned off unless stated otherwise.

  • Creating a new ELAN file
  • Creating tiers
  • Annotating
  • Using Transcription mode (optional)

Tier Naming Conventions

It’s important to have a naming convention for your tiers so you can keep track of everything in an organized manner before it gets out of hand. This is especially true when you have multiple labelers and more than just one hypothesis to test. Our main categories for tiers are kinematics (abbreviated to “kin”), phases, pgg, sdg, referential, video, and analysis. You will learn more about each later on.

The category name is followed by the item you are labelling for in that category, then notes like “reliability” for reliability labels or “comment” for comments tier, and finally followed by the labeler’s initials. You can add additional information such as date first created with the month and year in two digits each. This helps to get a sense of who did what and when and if you have changed conventions, you can easily find what needs to be updated. Also, if you have questions about a particular tier or annotation, you would know who to ask. So for May 2017, we would put in 0517. Between each part of this tier name, we use a dash. Some examples:

  • phase-preparation-reliability-AR-0516
  • referential-comments-CL-0415
  • analysis-tobi-strokeswithpitchaccent-AR-0416
  • video-occlusions-SJ-0113

Labeling Gestural Strokes

This is vital for most gesture investigations.

  1. 1. Pass 1: Create a tier named “No-movement.” Go through the entire video labeling areas where there is no movement. You don’t have to put in any text labels. This is easier than marking areas of movement.
  2. 2. Pass 2: Create a tier named “Loose groupings.” Go through the entire video labeling general ranges of gesture strokes, whether individual or groups. For the groups, you can put in a label of how many gesture strokes are in the group so you can break it down later.
  3. 3. Pass 3: Create a tier named “Strokes.” Break down all the groupings into individual gesture strokes until you get every one of them. If you’re uncertain about whether a motion counts as a stroke, create a comments tier and say why on that tier. For continuous looping gestures, split it up by individual loops.
  4. 4. Pass 4. On the “Strokes” tier, for each stroke, adjust the begin and end position of the stroke. This takes a long time to do. Doing this after labeling the phases can help make it easier. For movements that do not look like strokes, put them in a separate tier called “non-counted movements” with description of what’s going on. Some include: fidgeting, bounce, small movement, turning page, self-touch, rubbing fingers, etc. If any of these occur quite frequently in your corpus, create a dedicated tier for them.
  5. 5. Pass 5. If your speaker does a lot of shakes, determine whether they are large enough to be individual strokes. If they are, label each one as a stroke. If not, play the selection with the audio on to determine whether they are prominence/rhythm related (see section below on Referential) and if so label it as a single stroke but comment on a comments tier something like “5 or 6 shakes”. Otherwise, put it into the “non-counted movements” tier especially if it does not appear to be significant.

Optional: Gesture Phases

Familiarize yourself with the different phases. Gestures may not have all phases.

  1. Preparation
    A dynamic phase, usually upwards, preparing for action
  2. Pre-stroke hold
    A static phase, non-moving, preceding stroke
  3. (Stroke) already labeled by this point
  4. Post-stroke hold
    A static phase, non-moving, after stroke
  5. Recovery/Relaxation
    A dynamic phase, after stroke
  6. Relaxed
    Static phase, between gestures, does not really count as a phase but include depending on your hypotheses tested

You can create one tier for all phases (and add in the stroke phase to this tier) or create a separate tier for each phase and label in multiple passes by phase type. The latter helps you better recognize different forms of the same phase.

Optional: Combine all phase tiers into one tier: Stroke-defined Gestures

  1. Create a tier named “SDGs.” In this tier, group up into a single annotation all the phases accompanying a stroke (including the stroke).
  2. This tier will be used as the main “gesture” tier, and will be duplicated for most other labeling tiers.

On labeling child gesture phases: In our experience with child gestures, we find that individual phases may be repeated with long time gaps between them. You may find lone preparation phases, long hold durations, and plenty of fidgeting.

Labels based on SDGs

  • Make a copy of the SDG tier and delete the annotation labels but not the annotations themselves.
  • Rename the tier based on what you will be labeling.
  • Since the start and end of each annotation has been already done, you can use ELAN’s transcription mode to do the following labeling dimensions.

Handedness

Handedness refers to which hand or hands are being used in gesticulation. Right and Left refers to the speaker’s right and left, usually the opposite of the labeler’s.

  • Both hands equal synchronous => BHES
  • Both hands equal asynchronous/alternating => BHEA
  • RH dominant (both hands are moving) => RHD
  • LH dominant (both hands are moving) => LHD
  • RH (only one hand moving)
  • LH (only one hand moving)
  • Notes on Handedness tier labeling:
  • For cases where one hand is lower than the other, but they’re both going up and stop at the same level, count that as BHES.
  • For cases where both hands are coming inwards, count it as BHEA, but am not completely sure.
  • For cases where one hand is low and the other hand is high (or left and right) and moving up and down alternately (or left and right alternately), I counted it as a definite BHEA. This makes BHEA contain both asynchronous and “alternating.” A set of these may be followed by one hand coming down (or either left/right) while the other remains still. I labeled it as BHEA/LHD or BHEA/RHD.
  • When the first hand is held and not in a relaxed state, and the second hand is moving, the second hand is dominant (RHD or LHD, not RH or LH). The single hand labels are used when the other hand is at fully or partially relaxed state.

Referential: Option A

  • ***Label referential with audio ON***
  • I = Iconic/Concrete -> gesture matches semantically to speech
  • M = Metaphoric/Abstract -> intensifying meaning -> needs another layer/level of explanation to say how the gesture connects to the speech
  • E = Emblem -> emblematic gestures like the peace sign, OK, etc that replaces or has same meaning as speech. Does not count when it’s just the handshape that is emblematic and does not refer to the speech in any way.
  • PR = Only Prominence-Lending -> no content at all > accentuating prominence (stands for prominence-rhythmic)
  • PR-M = Prominence-rhythm + metaphoric
  • PR-E = Prominence-rhythm + emblem
  • PR-I = prominence-rhythm + iconic (We have not seen many of these)
  • NSP = No speech for where there’s a gesture (SDG) but not accompanied by speech

Referential: Option B

Because PR can also occur during typical referential gestures, this can be a separate tier. Referential can just be its own tier with the labels for Iconic, Metaphoric, Emblematic

  • PR
  • Not PR
  • Kinematic Labels
  • These are also based on the SDG tier labels.

Trajectory Shape

  • S : Straight (vertical)
  • S-H : Straight horizontal
  • S-D : Straight diagonal
  • C : Curved
  • L : Loop/Looping

Hand Shape

Hand shape is labeled for the hand shape occurring during or at the end of the stroke (where the action is), not for the entire SDG even though the annotation spans the entire SDG. It is also labeled on a scaled gradient from 1-5 when you are not sure which one it is exactly. The number goes between the letters. For example if there’s a loose hand shape that has the index finger pointing out a bit, you can do L3D. Or L1D if it looks more loose than deictic. Or L5D / D1L if it looks more deictic than loose.

  • D Deictic, pointing
  • G Gun
  • F Fist
  • R Relaxed
  • O Open, fingers spread outwards
  • C Cup or claw
  • K Knife, fingers straight together and flat
  • A Angled, fingers flat together, bent 90 degrees above palm
  • H Hole, making an empty cylinder shape
  • Q Okay, iconic “okay” sign
  • T Two, twice, peace sign
  • L Loose fist, relaxed shape with fingers curled in

Two handed (rarely used)

  • S Steepled, two open hands with fingertips touching
  • W Wall, two open hands with fingertips touching, forming a vertical barrier
  • J Jailed, like wall, but with fingers spaced out
  • I Intertwined, two cup-like hands clasped, with fingers intertwined

For illustrations of these hand shapes, see: http://adainspired.mit.edu/gesture-research/

Location

Location tier is not based on the SDG tier. There are two main places to look for labeling location: the beginning of the stroke, and the end of the stroke. If necessary, add the middle of the stroke.
location graph

Analysis Process

  • Export tiers as tab-delimited text (or csv) files
  • Learn to work with pivot tables (Excel/Google Sheets)