Main content
Top content
Instantaneous visualization of gaze data, here show as yellow dots: (a) visualization of gaze data on a certain frame with VLC player — (b) like normal subtitles one can easily change between the gaze data of subjects — (c) exemplary gaze data on another video sequence.
Eye Tracking Data in Multimedia Containers for Instantaneous Visualizations
ETVIS16 - Demonstration Video and Datasets
Nowadays, the amount of gaze data records of subjects associated with video sequences increase daily. These eye tracking data are unfortunately stored in separate files in custom-made data formats, which reduces accessibility even for experts and makes the data effectively inaccessible for non-experts. Consequently, we still lack interfaces for many common use cases, such as visualization, streaming, data analysis, high level understanding, and semantic web integration of eye tracking data. To overcame these deficits, we want to promote the use of existing multimedia container formats to establish a standardized method of incorporating content videos with eye tracking metadata. This will facilitate instantaneous visualization in standard multimedia players, streaming via the Internet, and easy use without conversion. Using our prototype software, we embed gaze data from eye tracking studies and the corresponding video into a single multimedia container, which can be visualized by any media player. Based on this prototype implementation, we discuss the benefit of our approach as a possible standard for storing eye tracking metadata including the corresponding video.
Source Code
VLC 3.0.0 patch
Modified version of the subsusf.c. Necessary for the playback of the USF files.
USF to ASS translation
XSL file for USF to ASS translation usf2ass.xsl.
Eye tracking data to USF converter
Currently, the eye tracking data to USF converter is only available as command line implementation. A version with a user interface is planned to release in the upcoming months. If you want to get the source code of the command line converter nevertheless, please write an email to Julius Schöning.
VLC Visual Analytics Plugins
To install the extensions under Linux, please copy the SimSub.lua and/or MergeSub.lua into ~/.local/share/vlc/lua/extensions/ for the current user or /usr/lib/vlc/lua/extensions/ for all users.
SimSub: Visualization of different eye tracking datasets in multiple windows
MergeSub: Visualization of different eye tracking datasets in a single window
Converted Eye Tracking Data Sets with Instantaneous Visualizations
The following data sets are provided for research purposes. By using these data sets in the proposed multimedia container format, please cite [S], [S1], [S2], or [S3] and also the original dataset [A], [B], [C], [K], or [R].
- Real World Visual Processing [S3] - 1 video
- Açik et al. [A] dataset - 216 videos
- Sundberg et al. [B] dataset - 7 videos
- Coutrot & Guyader [C] dataset - 60 videos
- Kurzhals et al. [K] dataset - 11 videos
- Riche et al. [R] dataset - 24 videos
Coutrot & Guyader [C]
ID | Images | Stimulus | Note |
C1 | Video 1: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C2 | Video 2: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C3 | Video 3: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C4 | Video 4: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C5 | Video 5: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C6 | Video 6: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C7 | Video 7: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C8 | Video 8: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C9 | Video 9: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C10 | Video 10: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C11 | Video 11: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C12 | Video 12: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C13 | Video 13: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C14 | Video 14: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C15 | Video 15: ASS USF | Observers: 18 per all four auditory condition Class: one moving object | |
C16 | Video 16: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C17 | Video 17: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C18 | Video 18: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C19 | Video 19: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C20 | Video 20: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C21 | Video 21: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C22 | Video 22: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C23 | Video 23: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C24 | Video 24: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C25 | Video 25: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C26 | Video 26: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C27 | Video 27: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C28 | Video 28: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C29 | Video 29: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C30 | Video 30: ASS USF | Observers: 18 per all four auditory condition Class: several moving objects | |
C31 | Video 31: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C32 | Video 32: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C33 | Video 33: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C34 | Video 34: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C35 | Video 35: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C36 | Video 36: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C37 | Video 37: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C38 | Video 38: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C39 | Video 39: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C40 | Video 40: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C41 | Video 41: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C42 | Video 42: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C43 | Video 43: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C44 | Video 44: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C45 | Video 45: ASS USF | Observers: 18 per all four auditory condition Class: landscapes | |
C46 | Video 46: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C47 | Video 47: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C48 | Video 48: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C49 | Video 49: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C50 | Video 50: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C51 | Video 51: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C52 | Video 52: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C53 | Video 53: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C54 | Video 54: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C55 | Video 55: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C56 | Video 56: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C57 | Video 57: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C58 | Video 58: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C59 | Video 59: ASS USF | Observers: 18 per all four auditory condition Class: faces | |
C60 | Video 60: ASS USF | Observers: 18 per all four auditory condition Class: faces |
Kurzhals et al. [K]
ID | Images | Stimulus | Setting | Task | Induced Patterns |
K1 | Car Pursuit (Rectangle) ASS USF | Panning camera follows a red car while it was going through a roundabout. | Follow the red car. | Potential smooth pursuit with long time spans of attentional synchrony on the red car. | |
Car Pursuit (Polygon) ASS USF | |||||
K2 | Turning Car (Rectangle) ASS USF | Camera follows turning car. The movement of the car describes the shape of an eight. | Recognize the shape that is described by the movement of the car. | Attentional synchrony on the car with potential smooth pursuit eye movement. | |
Turning Car (Polygon) ASS USF | |||||
K3 | Dialog ASS USF | Two persons talk to each other in front of the camera. | Follow the dialog attentively. | Switching focus between the faces of both persons. Label on shirt (right person) attracts additional attention. | |
K4 | Thimblerig ASS USF | A thimblerig with three cups and a marble. | Find the cup with the marble. | Attentional synchrony mainly on the cup with the marble. | |
K5 | Memory ASS USF | A 4x4 memory game. Pairwise flipping of cards is performed until all pairs are found. | After one card is flipped, focus on the corresponding card of the pair. | Increasing attention on matching cards after several turns and switching focus during the search. | |
K6 | UNO ASS USF | Two persons play UNO card game until the right player wins. | For each player's turn, focus on the playable cards on the hand. | Switching focus and attention mainly distributed between both hands and the stack of played cards. | |
K7 | Kite ASS USF | Person on a meadow steers a kite. The kite repeatedly leaves the field of view. | Follow the flight path of the kite if possible. | Smooth pursuit if the kite is visible. Otherwise, the participants either tried to estimate the position of the kite, or focused on the person. | |
K8 | Case-Exchange ASS USF | Various persons crossing the field of view while a text ribbon in the lower part is showing further information. | Task is provided by the text ribbon: Look for metal case. | Attentional synchrony on the text ribbon until the metal case appears and the task is readable. | |
K9 | Ball Game ASS USF | Three players with orange shirts and one player with a white shirt pass a ball around. | Task group A: Count ball contacts of the white player. Task group B: Count passes between orange players. | Attentional synchrony often on the ball, independent from the task. | |
K10 | Bag Search ASS USF | Various persons carrying different bags are crossing the field of view. | Look for a specfic bag. Two groups: two different search targets, presented before the video started. | Switching focus on new bags in the scene. Depending on the group, the search targets attract more attention. | |
K11 | Person Search ASS USF | People with different clothing cross the field of view. | Task group A: Find the person with a hooded sweater. Task group B: Find the person with a red shirt and a headgear. | Switching focus on new persons. After identification, search targets become less important than new persons. |
Riche et al. [R]
ID | Images | Stimulus | Note |
R1 | Video 1: ASS USF | Observers: 10 Class: Surveillance | |
R2 | Video 2: ASS USF | Observers: 10 Class: Abnormal | |
R3 | Video 3: ASS USF | Observers: 10 Class: Surveillance | |
R4 | Video 4: ASS USF | Observers: 10 Class: Abnormal | |
R5 | Video 5: ASS USF | Observers: 10 Class: Surveillance | |
R6 | Video 6: ASS USF | Observers: 10 Class: Moving | |
R7 | Video 7: ASS USF | Observers: 10 Class: Noise | |
R8 | Video 8: ASS USF | Observers: 10 Class: Crowd | |
R9 | Video 9: ASS USF | Observers: 10 Class: Surveillance | |
R10 | Video 10: ASS USF | Observers: 10 Class: Crowd | |
R11 | Video 11: ASS USF | Observers: 10 Class: Noise | |
R12 | Video 12: ASS USF | Observers: 10 Class: Crowd | |
R13 | Video 13: ASS USF | Observers: 10 Class: Noise | |
R14 | Video 14: ASS USF | Observers: 10 Class: Crowd | |
R15 | Video 15: ASS USF | Observers: 10 Class: Noise | |
R16 | Video 16: ASS USF | Observers: 10 Class: Abnormal | |
R17 | Video 17: ASS USF | Observers: 10 Class: Noise | |
R18 | Video 18: ASS USF | Observers: 10 Class: Abnormal | |
R19 | Video 19: ASS USF | Observers: 10 Class: Moving | |
R20 | Video 20: ASS USF | Observers: 10 Class: Abnormal | |
R21 | Video 21: ASS USF | Observers: 10 Class: Crowd | |
R22 | Video 22: ASS USF | Observers: 10 Class: Moving | |
R23 | Video 23: ASS USF | Observers: 10 Class: Noise | |
R24 | Video 24: ASS USF | Observers: 10 Class: Moving |
References
[S] | J. Schöning, C. Gundler, G. Heidemann, P. König & U. Krumnack. Visual Analytics of Gaze Data with Standard Multimedia Players. Journal of Eye Movement Research, 10(5) : 1-14, 2017. | DOI | URL | BibTeX |
[S1] | J. Schöning, P. Faion, G. Heidemann & U. Krumnack. Eye Tracking Data in Multimedia Containers for Instantaneous Visualizations. In IEEE VIS Workshop on Eye Tracking and Visualization (ETVIS), pages: 74-78, 2016. IEEE. | PDF | DOI | URL | BibTeX |
[C] | A. Coutrot & N. Guyader. How saliency, faces, and sound influence gaze in dynamic social scenes. Journal of Vision, 14(8) : 5-5, 2014. Association for Research in Vision and Ophthalmology (ARVO). | DOI | BibTeX |
[K] | K. Kurzhals, C.F. Bopp, J. Bässler, F. Ebinger & D. Weiskopf. Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli. In ACM Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), pages: 54-60, 2014. ACM Press. | DOI | BibTeX |
[R] | N. Riche, M. Mancas, D. Culibrk, V. Crnojevic, B. Gosselin & T. Dutoit. Dynamic Saliency Models and Human Attention: A Comparative Study on Videos. Lecture Notes in Computer Science, pages: 586-598, ISBN: 978-3-642-37431-9, 2013. Springer Science + Business Media. | DOI | BibTeX |