The intertwining of everyday life and computation, along with a new generation of inexpensive digital recording devices and storage facilities, is revolutionizing our ability to collect and analyze human activity data. Such ubiquitous data collection has an exciting potential to augment human cognition and radically improve information-intensive work. In this paper we introduce a system to aid the process of data collection and analysis during observational research by providing non-intrusive automatic capture of paper-based annotations. The system exploits current note-taking practices and incorporates digital pen technology. We describe the development, deployment and use of the system for interactive visualization and annotation of multiple stream of video and other types of time-based data.