James D. Hollan, Edwin L. Hutchins, and Javier Movellan
What conditions can facilitate rapid advances and breakthroughs in behavioral science to rival those seen in the biological and physical sciences in the past century? The emergence of cognitive science and the converging view across multiple disciplines that human behavior is a complex dynamic interaction among biological, cognitive, linguistic, social and cultural processes are important first steps. While empirical and theoretical work is rapidly advancing at the biological end of this continuum, understanding such a complex system also necessitates data that capture the richness of real-world human activity and analytic frameworks that can exploit that richness.
In the history of science, changes in technologies for capturing data, as well as those for creating and manipulating representations, have often led to significant advances. The human genome project, for example, would have been impossibly complex without automatic DNA sequencing. Recent advances in digital technology present unprecedented opportunities for the capture, storage, analysis, and sharing of human activity data.
Researchers from many disciplines are taking advantage of increasingly inexpensive digital video and storage facilities to assemble extensive data collections of human activity captured in real-world settings. The ability to record and share such data has created a critical moment in the practice and scopeof behavioral research. The main obstacles to fully capitalizing on this opportunity are the huge time investment required for analysis using current methods and understanding how to coordinate analyses focused at different scales so as to profit fully from the theoretical perspectives of multiple disciplines.
We propose to integrate video and multiscale visualization facilities with computer vision techniques to create a flexible open framework to radically advance analysis of time-based records of human activity. We will combine automatic annotation with multiscale visual representations to allow events from multipledata streams to be juxtaposed on the same timeline so that co-occurrence, precedence, and other previously invisible patterns can be observed as analysts explore data relationships at multiple temporal and spatial scales. Dynamic lenses and annotation tools hwill provide interactive visualizations and flexible organizations of data.
Our goals are to (1) accelerate analysis by employing vision-based pattern recognition capabilities to pre-segment and tag data records, (2) increase analysis power by visualizing multimodal activity and macro-micro relationships, and coordinating analysis and annotation across multiple scales, and (3) facilitate shared use of our developing framework with collaborators.
The work we propose builds on our long term commitment to understanding cognition “in the wild”, developing multiscale visualizations, and recent experience automatically annotating video of freeway driving. We propose to extend the theory and methods developed in our earlier work and integrate them with new web-based analysis tools to enable more effective analysis of human activity. As initial test domains we will focus on understanding activity in high-fidelity flight simulators and the activity histories of workstation usage and the process of writing. We will also evaluate a novel technique to assist in reinstating the context of earlier activities. Our long range objective is to better understand the dynamics of human activity as a scientific foundation for design.
Tags: Digital Ethnography, NSF