Fingers’Dance: a gestural grammar for the navigation and annotation of dance video recordings.

5, May 2010  |  Published : Experimentations  | 

By Vincent Puig, Yves-Marie Haussonne, Thibaut Cavalié, Yves-Marie L’Hour
In collaboration with Dominique Sciamma, Ioana Ocnarescu, Huieen Kim, Annabelle Eugenia, Baptiste Lanne, Vanessa Reiser
Strate Collège Designers
30, rue Troyon
92310 Sèvres

Abstract : This short presentation deals with the first IHM project (Human/Machine Interaction) for the gestural annotation of dance video recordings, realized and analyzed by Thierry de Mey during his stay at IRI.
1) Thierry de Mey has elaborated a grammar of gestures to codify his own movies and musical pieces (for example Musique de tables). This grammar is used to identify the different choreographic moves within the show One flat thing by William Forsythe, and the annotation is processed through Lignes de temps, IRI’s software for the annotation of temporal objects. To each identified gesture of the dancers a corresponding move of the observer’s two hands is recorded via a multi-touch sensitive surface.
2) With this pre-set collection of gestures on a multi-touch device, a specific gesture can call back through Lignes de temps all the corresponding clips from the movie. For example a rotating move will send out a request for all rotating dance moves spotted in the movie. To facilitate navigation, one or several specific gestures on the sensitive table can be associated to any sequences, making their access more direct. This first experimentations was led within the artist-in-residence program at IRI with the support of Microsoft France and was publicly presented at CNAM at the ’09 New Industrial World Forum.

Download the article in pdf format.

Video demonstration.

Theoretical basis.

Our approach is both theoretical and technological. We need to re-think our relationship to video content and cinematographic works at a psychological, sensitive and critical level, by making use of the means of appropriation and perceptive perception that the digital has to offer. In the words of philosopher Gilber Simondon, this approach takes place within a process of psychic and collective individuation, by the mean of tools of transindividuation which constitute an organology, in its most open meaning. According to our previous studies in the domain of critical judgment directed toward movies, the amateurs and movie aficionados extend their experience of cinema in the digital world, through several critical practices, either social (thus the social networks relate to the role of cine-clubs) or instrumental (specialized press, tools for navigation annotation and creative use). On a technological level, we wish to develop our research toward sensorial-motricity, that is the systems enabling navigation loops, retroaction and active perception. In this cinematographic context, the tools we are developing are getting closer to the tools used to make a movie : the camera itself, but on a broader level all the tools of production (writing, editing, publishing) because they represent potential tools for active perception.
To this day we have mainly explored the annotation tools concerned with text or voice, and we are moving toward video, gesture and sight.

Choreographic grammar and cinematographic grammar

The starting point of this research project is the personal vision of director Thierry de Mey toward gesture itself, as he demonstrated with his first gestural score for three percussionists in Musique de tables (1987).

Musique de Tables

Fig1 : Musical score of Musique de tables

He explored this vision directing dancers, confronting his cinematographic language to the most contemporary choreographic expressions. Thus his residency at IRI in 2007 enabled a first analysis of One flat thing by Wlliam Forsythe, acknowledging its two dimensions, both cinematographic ( the different types of shots and framings) and choreographic (solos, groups, hitting, sliding gestures, etc) through the use of Lignes, which IRI has been developing since 2006.

Lignes de Temps De Mey

Fig2 : Analysis of One flat thing with the software Lignes de temps by Thierry de Mey

Fingers’ dance : an interface for annotation and navigation through gesture
In September 08 the idea of an interface for the annotation and navigation of this movie became self-evident within the collaboration initiated with a team of young designers from Strate Collège, with to the support of a residency program established at IRI thanks to Microsoft France. Beyond this object, we aim at the constitution of a gestural grammar generic enough to be used with a much larger selection of dance video recordings. The device chosen to record gestures toward navigation or annotation is a multi-touch table developed by Microsoft Surface. This device requires a 2D captation but enables the use of both hands and let us invent new collaborative technics, for example 2, 3 or 4 people working together around the table.

Fingers' Dance

Fig3 : Experimentation on Microsoft Surface

After a specification stage with Thierry de Mey, a bibliographical research helped us gaining access to the gestural grammars currently developed, five propositions were then made to validate the most adapted IHM (Human/Machine Interaction) to our cultural and technological specifications (ToolKit Surface). Following this stage, we worked toward a design allowing the multi-touch interface to be used by 1, 2 or 4 persons. Each of the participants uses his own visualization window, with its tools for navigation and annotation through gestures. The number of gestures/signs was the reduced with Thierry de Mey to obtain two grammars for navigation and navigation limited to several basic gestures.

Grammaire gestuelle

Fig4 : Cinematographic and choreographic gestural grammar

Example of Cinematographic Grammar :
Camera shot
– pan shot
Camera axis
-eye-level shot
-high-angle shot
-low-angle shot
-from extreme long shot to medium shot
-from close shot to long shot
-close shot
– extreme long-shot
-long shot
-medium shot
-close-up (detail of a body)
-extreme close-up (detail of a face)
-close shot )
From this basic language, the interfaces were built through the exchanges between designers and developers.

Annotation gestuelle

Fig5 : Annotation of a four dancers’ figure with a gesture on the multi-touch screen

Research at stake and projects in gestation
With this original research these are our primary goals :
Gesture acting as memory for an audiovisual stream.
A whole range of works on choreographic grammar demonstrate the enacting role of gesture and its memorizing power of spatial markers (position/postures of the dancers) while cinematographic grammar insists on temporal markers. This was self-evident in Thierry de Mey’s analysis of the traveling in One Flat Thing (they act as « counter-points » of the danced figures) and in the way we tried to put our tools in the hands of large public of amateurs. Of course cinema and dance constantly play between space and time, thus gestures are really appropriate to the articulation of the two dimensions, with a memorization and a comprehension which is complementary to our previous annotation experiences with keywords or audio and video tagging.
The grammatization of gesture : as generic as possible to reach different applications.
To reduce our research to such a specific example as One Flat Thing is not satisfying. Issues of normalization and industrial competition concerning the ways to pilot a movie through gestures are already at stake. This is why we began to expand our grammar toward dance figures, testing it on Musique de tables and later and on a larger selection of works.

The collective gesture : productivity, surprise and social well-being.
Multi-users interactions have been tested through several scenarios and have been evaluated during the workshops with the public of the VidéoDanse festival (November 09). The distribution of tasks is efficient, and the participants share an experience, a common memory of the movie, generating a productive and qualitative social interaction.
We want to thank Thierry de Mey for initiating this experimentation, Pierre-Louis Xech, Thomas Defaye and Thomas Serval from Microsoft France which have supported the whole project from conception to technical support. Thanks also to the students at EFREI, Santiago Aria, Riley Leykni, Tarari Amine and Jonathan Pamphile who have participated to the software development.

– (Puig, V., et al) Collaborative Annotation System Using Vocal Comments Recorded on Mobile Phones and Audio Guides: The Centre Pompidou Exhibition Traces Du Sacré. In J. Trant and D. Bearman (eds). Museums and the Web 2009, Published April 15, 2009. Also available

– (Puig & al.) Lignes de temps, une plateforme collaborative pour l’annotation de films et d’objets temporels, proceedings IHM, nov. 2007
- (Puig) Lignes de temps, une plateforme collaborative pour l’annotation de films et d’objets temporels, Collection A propos de. Les représentations du réel à l’écran, Ministère de l’Education Nationale/Cinémathèque française, Actes de l’université d’automne, 27-31 octobre 2007
- (Puig) Amateurs du XXième siècle, colloque Patrimoine et numérisation, novembre 2008, Culture et Recherche, nov. 2008
- (Prié, Puig) Nouveaux modes de perception active de films annotés, Colloque Figures de l’Interactivité, Poitiers, nov. 2008.

  • Share

Next Events

  • Pas d'événement.