Recording/Playing Both Gaze Data And Computer Interaction VNC is normally used to view and interact with the screen of a remote machine. It is similar to PC Anywhere and Reachout, except that we have source. VNC consists of a server and a viewer. The server is run on the remote machine (Windows or Unix/Linux X Window system). The viewer connects to the server and views/interacts with the remote machine. The viewer was modified to record and playback the interaction. EyeLink is an eye tracking system. It uses head mounted cameras, which record the location of the user's gaze on a screen. EyeLink consists of two machines, an operator machine which gathers the gaze data and a subject machine which performs calibration and provides the stimulus. The normal EyeLink stimulus is fixed images or text. Our stimulus is actually a third machine, a linux machine running VNC which allows the subject to interact with Windows or Unix/Linus machines, while we record the interaction. Alas, we still need the EyeLink subject machine to perform calibration and start/stop gaze data gathering. The layout looks like: monitor monitor keyboard kvm keyboard mouse switch mouse operator subject linux machine machine machine head mounted cameras The subject machine and linux machine share a keyboard, monitor, and mouse though a kvm (keyboard, video, mouse) switch. The steps are: -1) Log onto the subject and linux machines as yourself. You should have the same login on both. 0) Start a vnc server somewhere. It could be any Unix/Linux/Windows machine accessible on the network. For this example, we'll start a server on the linux machine. vncserver -geometry 1280x1024 If you haven't run the server before, it will prompt for a password. Access to the server is protected by the password. The viewer will prompt you for a password when you try to connect to the server. The server will report a "display number", such as :3,. Remember it. You must specify : when connecting a viewer to the server. Choose a geometry which fills the viewer screen. We need the viewer and eyelink screens to coincide. See the note at the end of this file about alignment issues. The resolution of the viewer and eyelink screens can differ, because playback scales the gaze data to fit. The resolution of the viewer screen is set when the X window system is started. In this case it is 1280x1024. 1) Synchronize the subject machine real time clock with the linux machine real time clock. Click right on the yellow/green little clocks in the taskbar. Select "Update now". The linux machine runs an NTP server to which the subject machine, using NTP software (nettime), can synchronize itself. After synchronization the real time clocks are usually within about 100 milliseconds. I check the synchronization by running the following on the linux machine. ntpdate -q mallet The offset is the is the difference between the real time clocks in seconds. The offset drifts. Some of the drift is due to the lack of real time resolution of Win98, 1/18 of a second, about 55 milliseconds. Almost all of the drift is due to Win98. If you compare the linux machine to a campus time standard, you'll see an offset of a couple of milliseconds, with a drift of a fraction of a millisecond. ntpdate -q ntp 2) Start background-vnc on the subject machine. This is a modified version of the background.exe example program. It's available in the ron folder on the desktop. The modification consists of four lines #include struct _timeb timeb; _ftime(&timeb); eyemsg_printf("REALTIME %d %d", timeb.time, timeb.millitm); Since the gaze data timestamps are relative to the start of eyelink on the operator machine, We needed a way to relate them to real time. The modification adds a line to the gaze data containing the real time. Since that line is timestamped too, we can relate the timestamp to real time. This assumes the subject machine clock is accurate. (See above about real time clock accuracy.) 3) Start vncviewer on the linux machine. vncviewer -encodings hextile -record -fullscreen localhost:3 \ -recordfile vncrecord.log "-encodings hextile" specifies a type of compression. By default if the viewer and server are running on the same machine, no compression is performed, resulting in a huge record file. "-fullscreen" specifies that the viewer should fill the screen, with no VNC border. We need this so that the eyelink and viewer screens are aligned. So, that the upper left and lower right corners match. See the note at the end about alignment issues. localhost:3 is the VNC server on the linux machine, but any machine running a VNC server could be specified. "-recordfile vncrecord.log" specifies the vnc record file. If you don't specify a file, it defaults to /tmp/vncrecord.log, which someone else might have accidently left, causing your attempt to fail because you can modify the other person's file. 4) Run your test. 5) Exit both programs. Exit vncviewer by hitting F8, then chosing "Quit Viewer". Exit background-vnc by hitting ESC. 6) On the subject machine convert the gaze data EDF file to an ASC file. The EDF file format is a secret, so we're forced to work with the ascii file. Using Start Programs/MS-DOS Prompt, cd to the directory containing the EDF file, then edf2asc backgd Since writing this, I was informed by Bob Kemp that EDF is an open format (see www.hsr.nl/edf). 7) Copy the ASC file to the linux machine. The linux machine, hoopla, should be in the Network Neighborhood, so you can drag-and-drop the ASC file into your home on hoopla. 9) Simplify the ASC file. The ASC file isn't suitable for quick parsing during playback, so we preform some conversion on it. It's easier to parse text with awk than C, so the first run a awk script. gaze.awk backgd.asc > gazedata 10) Convert the gaze data from ascii to binary. Binary can be parsed faster during playback. makegaze gazedata > vncrecord.log.gazedata What's important here is that the VNC record file and binary gaze data file are in the same directory and the name of the gaze data file is the same as the VNC record file with ".gazedata" suffix. 10) Playback vncviewer -playback -recordfile vncrecord.log "-recordfile vncrecord.log" specifies which record file to playback. vncviewer defaults to playing back /tmp/vncrecord.log, which might not be your file, so specify the file. vncviewer will find the gaze data by appending .gazedata to the record file name you specified. 11) Kill the server. You can use the server repeatedly. In fact, if you connect to it again, you'll find it looks the same as when you left it. When you're done for the day kill the server. vncserver -kill :3 Replace :3 with the for your server. Summary of the Programs background-vnc - Modified version of background.exe example program to include real time for relating timestamps to real time. vncviewer - Modified version to record and playback vnc, and to overlay gaze data if it is available. checkvnc - sanity check a vnc record file. If you need to parse a vnc record file, take a look at fbsnxt().nxt().nxt().nxt(). gaze.awk - convert .asc file to simpler format expected by makegaze. makegaze - convert the ascii gazedata to binary format for faster parsing during playback. checkgaze - sanity check a binary gaze data file. The format of the vnc record file is * MODIFIED TO INCLUDE STARTTIME * The FBS (framebuffer stream) file format is this: * * ::- * ::- 32-bit number, seconds since Jan 1 1970, big-endian * ::- 32-bit number, milliseconds part of realtime, big-endian * ::- 'FBS 001.000\n' * ::- * | * ::- 32-bit number of bytes, big-endian * ::- data received at timestamp , which is * bytes in length, padded to multiple * of 32-bits * ::- 32-bit number of milliseconds since beginning of * capture, big-endian * * occurs before and after to permit playing backwards * If you need to parse a VNC record file, take a look at checkvnc.c. It parses the file to the point of recognizing the type of the rectangle. Then look at HandleRFBServerMessage() in vnc_unixsrc/vncviewer/rfbproto.c. That is the function which parses the headers of the rectangle updates. Finally look at vnc_unixsrc/vncviewer/{corre,hextile,rre}.c. Those contain the functions which parse and unencode the rectangles. vnc_unixsrc/include/rfbproto.h contains the definitions of the structures used for parsing. The format of the binary gaze data file is * Gaze file format * ::- * ::- * | * ::- 32-bit number, seconds since Jan 1 1970, big-endian * ::- 32-bit number, milliseconds part of realtime, big-endian * ::- 16-bit number, screen width, big-endian * ::- 16-bit number, screen height, big-endian * ::- 32-bit number of milliseconds since beginning of * capture, big-endian * ::- 16-bit number, big-endian * ::- 16-bit number, big-endian The source for the vncviewer, checkvnc, gaze.awk, makegaze, and checkgaze are available using CVS. From any linux machine in the DCOG-HCI cvs login cvs checkout vnc_unixsrc cvs logout This will create a vnc_unixsrc directory containing the source for the current versions. If you want to see what changes I made to vnc, then cvs diff -r start will show diff's of the files I've changed. cvs (concurrent versions system) is a program for maintaining source code (actually any ascii text). See http://www.gnu.org/software/cvs/ for more information about cvs. Further documentation about VNC is available from http://www.uk.research.att.com/vnc/ Note About Alignment of EyeLink and VNC Screens Placement of the gaze data over the VNC playback requires reproducing the alignment of the Eyelink and VNC screens at playback time. The simplest solution is running both fullscreen during recording. An alternative would be to perform some gaze calibration of VNC to identify the upper left and lower right of the VNC screen in gaze coordinates. An X application which fills the VNC screen, displaying markers at opposite corners (maybe markers at four corners to allow for correction any trapezoidal distortion between EyeLink and VNC screens), and a button box to signal Eyelink to record the current gaze. We might want to do this even if eyelink and VNC are full screen, to improve the placment of the gaze during VNC playback. Alignment hasn't been as good as I expected. Often off by 1/2" to 1". I suspect most of that is due to the head mount slipping. If you want to experiment with the tightness of the head mount, try running the Eyedata experiment available in the Eyelink Experiment Kit folder. It displays the location of the gaze on the screen in real time. Compare the accuracy of the gaze indicator before and after raising your eyebrows, openning your mouth, or moving your head slightly. The only solution is to tighten the head mount ... ouch! Ron Stanonik Thu Nov 30 11:31:36 PST 2000