hollywood2_em/README.md

In this repository we provide a partial hand-labelled ground-truth eye movment annotation of the large Hollywood2 data set [1].

ALGORITHM EVALUATION

Below we provide preliminary evaluation results for some popular eye movement classification algorithms. The reported F1 scores are computed through the quality evaluation functionality of the sp_tool [2].

		Sample F1	Sample F1	Sample F1	Event F1	Event F1	Event F1
Model	average F1	Fixation	Saccade	SP	Fixation	Saccade	SP
1D CNN-BLSTM: speed + direction	0.787	0.872	0.827	0.680	0.808	0.946	0.588
sp_tool smoothed	0.755	0.853	0.816	0.617	0.820	0.905	0.516
REMoDNaV [3]	0.748	0.779	0.755	0.622	0.784	0.931	0.615
sp_tool [2]	0.703	0.819	0.815	0.616	0.587	0.900	0.483
(Dorr et al. 2010) [4]	0.685	0.832	0.796	0.373	0.821	0.884	0.403
(Larsson et al. 2015) [5]	0.647	0.796	0.803	0.317	0.807	0.886	0.274
(Berg et al. 2009) [6]	0.601	0.824	0.729	0.137	0.845	0.826	0.243
I-VMP	0.564	0.726	0.688	0.564	0.503	0.563	0.338
I-KF	0.523	0.816	0.770	0.000	0.748	0.803	0.000
I-VDT	0.504	0.813	0.700	0.136	0.557	0.559	0.263
I-HMM	0.480	0.811	0.720	0.000	0.646	0.700	0.000
I-DT	0.473	0.803	0.486	0.000	0.744	0.802	0.000
I-VT	0.432	0.810	0.705	0.000	0.520	0.555	0.000
I-VVT	0.390	0.751	0.705	0.247	0.061	0.555	0.023
I-MST	0.385	0.793	0.349	0.000	0.590	0.576	0.000

REFERENCES

[1] Mathe, S., & Sminchisescu, C. (2012, October). Dynamic eye movement datasets and learnt saliency models for visual action recognition. In European Conference on Computer Vision (pp. 842-856). Springer, Berlin, Heidelberg.

[2] Startsev, M., & Agtzidis, I, & Dorr, M. (2019). Characterising and Automatically Detecting Smooth Pursuit in a Large-Scale Ground-Truth Data Set of Dynamic Natural Scenes. Journal of Vision

[3] Dar, A. H., Wagner, A. S., & Hanke, M. (2019). REMoDNaV: Robust Eye Movement Detection for Natural Viewing. BioRxiv, 619254.

[4] Dorr, M., Martinetz, T., Gegenfurtner, K. R., & Barth, E. (2010). Variability of eye movements when viewing dynamic natural scenes. Journal of vision, 10(10), 28-28.

[5] Larsson, L., Nyström, M., Andersson, R., & Stridh, M. (2015). Detection of fixations and smooth pursuit movements in high-speed eye-tracking data. Biomedical Signal Processing and Control, 18, 145-152.

[6] Berg, D. J., Boehnke, S. E., Marino, R. A., Munoz, D. P., & Itti, L. (2009). Free viewing of dynamic stimuli by humans and monkeys. Journal of vision, 9(5), 19-19.