Background & Summary

Eye movement research relies on accurately classifying episodes of fixations, smooth pursuits, saccades, and other eye movements (see Lappi, 2016 for an overview1). Fixations actively maintain the gaze on or near stationary targets, while smooth pursuits are active attempts at “fixating” moving targets by matching their velocity1. In terms of the samples (consecutive gaze coordinates) collected during eye tracking, both fixations and smooth pursuits are characterized by low sample-to-sample velocities1,2,3. In contrast, saccades are ballistic eye movements (high sample-to-sample velocity) that rapidly shift the gaze between locations1. While fixations and saccades are readily distinguished based on their velocity, many state-of-the-art event classification algorithms fail to reliably distinguish smooth pursuits from fixations and sometimes confuse smooth pursuits and saccades3,4,5,6,7,8,9,10. Similarly, human annotators find smooth pursuits difficult to classify and frequently disagree on their classification2. Furthermore, it is questionable whether expert annotations constitute a gold standard2,11. This lack of accurate automatic smooth pursuit classification is a serious shortcoming because, among other applications, smooth pursuits are used to investigate schizophrenia12, traumatic brain injury13, and neurodegenerative diseases14,15.

Real-world eye movement classification involves complex factors like head, body, and binocular dynamics, and naturalistic stimuli1. In line with this, some of the most prominent datasets for training, evaluating and comparing classification algorithms are based on eye movements in response to naturalistic stimuli and human annotation (see https://michaeldorr.de/smoothpursuit/ for an overview). However, here we address the core difficulty of reliably distinguishing smooth pursuits from other eye movements by deliberately collecting data under highly controlled conditions (fixed-head, monocular eye movements, moving dots). This procedure enabled us to avoid manual annotation and instead establish plausible benchmark labels for fixations, saccades, and smooth pursuits only based on experimental manipulation and dynamic velocity thresholds. Our objective was to collect data that makes it easy to identify statistical features that can improve the classification of smooth pursuits. This approach may yield a reliable classification foundation that can then be generalized and validated in more complex, naturalistic datasets such as GazeCom16.

This dataset consists of nearly four hours of eye movement recordings collected at 1000 Hz from ten university students using a fixed-head paradigm. Participants were instructed to focus on small moving circles without moving their head. These circles, moved according to different patterns, speeds, and trajectories and were designed to evoke fixations, saccades, and smooth pursuits. As the head and body remained stationary, gaze and target positions were collected in a screen-relative (world) coordinate system. Importantly, each trial was limited to include maximally one type of slow eye movement events (i.e., fixations OR smooth pursuits) and one type of fast eye movements (i.e., saccades). This way we aimed to prevent fixations and smooth pursuits – which are frequently confused during annotation – from co-occurring; thus, enabling us to label eye movements based on a combination of the experimental design and adaptive velocity thresholds, without relying on human annotation. Additionally, this separation of fixations and smooth pursuits should make it easier to identify novel statistical features (feature engineering) that are better at distinguishing between these eye movements than velocity or other commonly used features. In addition to sharing the raw eye movement recordings, we also provide the full experimental code and target trajectories, enabling researchers to collect comparable data and investigate if incorporating target information into algorithms can improve classification performance. We think that this dataset complements existing datasets well and we encourage other researchers to use it for feature engineering, to train, validate, and benchmark their algorithms, and to investigate data-driven approaches other than dynamic velocity thresholds for assigning benchmark labels.

Data were randomly assigned to training (n = 7) and test sets (n = 3) and are available on the Open Science Framework (OSF; https://osf.io/zx7hc/)17. The OSF repository contains raw tracking data, participant demographics, target locations, calibration and validation performance, and a video showcasing the stimuli. We also offer a companion package on GitHub (https://github.com/lukekorthals/pursuing-smooth-pursuits-data). This repository contains the code and instructions for downloading, preprocessing, annotating, and plotting the data. A second GitHub repository (https://github.com/lukekorthals/pursuing-smooth-pursuits-experiment) contains the experimental code and a script for interactively visualizing the different stimuli.

Methods

The study was approved by the Ethics Review Board of the Faculty of Social and Behavioural Sciences at the University of Amsterdam, reference number FMG-2710. Participants were recruited via direct email and provided written, informed consent prior to taking part in the study. They agreed to the participation and sharing of their data for research purposes. The retained and published data does not personally identify the participants. The consent procedure included an explanation of the study aims, procedures, voluntary nature of participation, right to withdraw, and data-handling practices.

Participants

Twelve psychology students from the University of Amsterdam participated as a favor to the researchers. Two participants could not be calibrated because the eye tracker tracked their pupil jerkily and frequently lost it completely; thus, we collected no data for them. The remaining ten participants were between 22 and 27 years old (M = 24.2). Most identified as female (7 female, 2 male, 1 other), and had brown eyes (brown = 7, blue = 3). Eight participants were nearsighted; half of them used vision correction during data collection (glasses = 1, contact lenses = 3, no correction = 4). The remaining two participants had no known vision impairments. All participants indicated that they could clearly discern the stimuli.

Technical Setup

Participants stabilized their head using a chin rest, positioning their eyes 70 cm from the center of a widescreen monitor (61.4 × 36.5 cm; 2560 × 1440 pixels) operating at a 144 Hz refresh rate. Monocular eye movements of the right pupil were collected at 1000 Hz with an EyeLink 1000 Plus18 using ellipse tracking without drift correction. The EyeLink was mounted on the table below the center of the monitor.

Calibration and Validation

All calibration and validation results are accessible in the data repository. Participants self-calibrated using the manufacturer-provided nine-point calibration by focusing on circles and pressing the Enter key on a keyboard. Each calibration attempt was validated using the same nine-point procedure. We tried to achieve fair or good validation according to the SR Research manual19 for all participants and had them repeat the procedure multiple times if it indicated poor calibration. Eight participants achieved fair or good calibration quality, with a mean average error of 0.45 degrees (°) and a mean maximum error of 1.18°. Two participants only achieved poor calibration (average/max errors = 1.08°/3.16° and 0.50°/2.13° respectively). Degrees here denote the angular distance on the retina between the true target and the measured gaze position – sometimes referred to as degrees of visual angle (dva)20. Throughout this manuscript, we use degrees to express not only calibration error but also stimulus size (the angle subtended at the eye) and stimulus velocity in degrees per seconds (°/s).

Calibration and validation were supposed to be repeated between trials whenever participants removed their head from the chin rest during a break or if the eye tracker lost their pupil. This happened once for two participants each. They achieved good and fair validation during recalibration (average/max errors = 0.8°/1.22° and 0.79°/1.65° respectively). After investigating the logs, we realized that participant 21db28aa took a two-minute break after trial 81 but was not recalibrated, causing the gaze in all subsequent trials to be severely misaligned with the targets (see “Target Alignment” in Table 1). Consequently, we suggest removing trials 82 to 144 when analyzing data for this participant based on the trial_number column.

Table 1 Measures indicate good data quality for all trials apart from 82–144 of participant 21db28aa.
Full size table

Stimuli

Stimuli were created based on self-testing by two of the researchers (LK and SK). During each trial a small grey circle (#808080; radius = 0.2°) was presented on a black background (#000000). Participants were instructed to focus on these circles (targets) without moving their head and ideally without blinking. Circles moved across the screen and their movement pattern, speed, and trajectory were varied between trials.

Movement Pattern

Circles followed one of three patterns to evoke specific eye movements. Each pattern was designed to elicit only one type of fast (i.e., saccades) and one type of slow eye movements (i.e., fixations or smooth-pursuits) per trial. “Moving circles” moved consistently at target speed to elicit smooth pursuits (slow eye movements). “Jumping circles” jumped along multiple fixation locations to induce series of fixations (slow eye movements) and saccades (fast eye movements). Fixation locations were spaced evenly according to target speed and circles stood still for one second at each location. “Back-and-forth circles” moved consistently at target speed while jumping back and forth along the axis of their trajectory at one second intervals to produce smooth pursuits (slow eye movements) interspersed by (catch-up) saccades (fast eye movements).

Movement Distance and Target Speed

We selected the movement distance and speed based on self-testing to satisfy the following requirements.

  • Participants should accurately track targets at all speeds without moving their head. This was to ensure that the collected eye movements could be reliably classified relative to the world coordinate system.

  • Participants should comfortably complete each trial without or with minimal blinking. This was necessary to ensure the collected gaze trace clearly represents the intended eye movements and maximizes the data completeness for classification algorithms.

  • We wanted to collect data for very slow and very fast smooth pursuits. We aimed for this because statistical features for distinguishing smooth pursuits from fixations and saccades may vary across different pursuit speeds.

  • Smooth pursuits at all speeds should be clear without intermittent fixations or catch-up saccades. Avoiding fixations is required to classify eye movements based on experimental manipulations and dynamic velocity thresholds. We also wanted to avoid catch-up saccades during “moving circle” trials to ensure that pure smooth pursuit features may be extracted from our data. In contrast, “back-and-forth circle” trials were designed to include catch-up saccades.

  • Trials had to last at least one second long. This ensured that “jumping circle” and “back-and-forth circle” trials would include at least a single saccade.

We first selected a movement distance of 20° because it falls within normal eye movement ranges21 and felt comfortable without moving the head during self-testing. Then we selected target speeds to cover slow, medium, and fast smooth pursuit speeds. Maximum smooth pursuit speeds without catch-up saccades are reported to be around 30°/s1. At this speed, targets would clear the distance in less than one second, which would be too fast, as it would not allow for even a single saccade. To our knowledge, no minimum speeds for smooth pursuits are reported in the literature. Therefore, we piloted different speeds for “moving circle” trials. We found it difficult to smoothly track targets slower than 1°/s and faster than 6°/s. This is consistent with Drewes et al. (2018)22, who found that smooth pursuits slower than 1°/s included fixations, and faster than 6°/s included saccades. Based on self-testing we selected speeds of 1°/s, 3°/s, and 6°/s to represent slow, medium and fast smooth pursuits respectively. At these speeds, targets took 20 s, 6.66 s, and 3.33 s respectively to clear 20°. With these settings we could comfortably track targets without moving our heads and with minimal blinking. They also ensured that smooth pursuits would be collected at varying speeds and “jumping circle” and “back-and-forth circle” trials would include at least three saccades. Therefore, these settings satisfied all our requirements.

Inadvertently, the code for converting degrees to pixels did not correctly account for the monitor dimensions. As a result, targets did not move the intended 20° at speeds of 1°/s, 3°/s, and 6°/s. Instead, horizontal movements covered approximately 19.29° at 0.96°/s, 2.89°/s, and 5.78°/s, vertical movements covered approximately 20.38° at 1.02°/s, 3.06°/s, and 6.11°/s, and diagonal movements covered approximately 28.06° at 1.4°/s, 4.21°/s, and 8.41°/s. The actual distance and speed for each trial are included in the data (actual_distance and actual_speed columns). For transparency, we share the functions that were used during the experiment (PREPROCESSOR._PIX2DEG_FAULTY(), PREPROCESSOR._DEG2PIX_FAULTY() and the functions we used to calculate the actual distances and speeds (PREPROCESSOR._PIX2DEG(), PREPROCESSOR._DEG2PIX()) here: https://github.com/lukekorthals/pursuing-smooth-pursuits-data/blob/main/eyemovement_data/preprocessor.py.

Target Trajectory

Targets moved along eight straight trajectories; specifically, cardinal and ordinal directions (north ↑, south ↓, west ← , east → , north-west ↖ , north-east ↗ , south-west ↙ , south-east ↘ ). North and south represent up- and downwards movements, while east, and west represent right-, or leftwards movements. We selected this variety of horizontal, vertical, and diagonal movements because they are each controlled by different muscle pairs23. Additionally, we avoided circular movements because plots reported by Drewes et al. (2018)22 indicate they are more difficult to track.

Task Battery

We presented every participant with each pattern-speed-trajectory combination twice for a total of 144 trials (i.e., pattern [3] × speed [3] × trajectory [8] × block [2]). Trials took 20 s, 6.66 s, and 3.33 s for targets moving approximately 1°/s, 3°/s, and 6°/s respectively. Therefore, about 24 minutes of eye movements were collected for each participant and almost four hours were collected in total. Note that slow targets were overrepresented in this dataset because their trials took longer. This makes this dataset particularly useful for studying how to distinguish slow smooth pursuits from fixations. Trials were organized in two blocks. Within each block, trials were ordered from slow to fast while movement pattern and trajectory were randomized. We did not randomize speed because adjusting to slower targets after tracking multiple fast targets appeared difficult during pilot testing.

Data Acquisition Procedure

Data for all participants was collected by one of the researchers (LK) in June 2023 in an eye tracking lab at the University of Amsterdam. Upon entering the lab, the experimenter informed participants about the procedure, and they signed participation forms. Participants sat in front of the screen, rested their head in a chin rest, and calibrated the eye tracker using nine-point calibration. Calibration was repeated multiple times in an attempt to achieve fair or good validation. After calibration the experimenter instructed participants to focus on all targets without moving their head and to avoid blinking as much as possible. Participants completed a tutorial trial (moving circle, 2°/s, east →) and could ask questions. Afterwards, participants started each trial by pressing the Space bar. Between trials, they could take as many breaks as they wanted. Before the first trial, the experimenter emphasized the importance of collecting clear data and encouraged them to take frequent breaks. To keep participants focused, the experimenter told them they were doing well and encouraged them to keep it up after the first trial and intermittently throughout data collection. Additionally, to ensure the quality of collected eye movements the experimenter constantly monitored if the eye tracker was working and if participants were accurately following the targets. If a participant blinked frequently or lost track of the target, the experimenter reminded them to focus and take breaks. If participants removed their head from the chin rest or the eye tracker lost their pupil, calibration was repeated before the next trial. After the last trial, the experimenter thanked and debriefed participants. Each session took between 45 and 60 minutes, depending on the time spent for breaks and calibration.

Data Records

The OSF repository (https://osf.io/zx7hc/) includes an mp4 video showcasing some stimuli and an R script that is used to extract relevant data from raw data files. It also contains two folders named “raw” and “clean”, each with subfolders “train” and “test”. These contain data folders for seven and three participants respectively. For “raw” data, each participant folder holds five raw data files and for “clean” data, each participant folder holds six csv files17.

Files under “raw”:

  • <id>.EDF: The raw EyeLink data.

  • <id>.asc: ASCII conversion of the EDF file.

  • <id>_participant.csv: Participant age, sex, and eye condition.

  • <id>_targets.csv: Target coordinates for each trial, experiment and trial time to match with gaze data.

  • <id>_trials.csv: Movement pattern, distance, and speed for each trial.

Files under “clean”:

  • <id>_blinks.csv: EyeLink blink data extracted from the ASCII file inside “raw”.

  • <id>_eyetracker.csv: EyeLink settings extracted from the ASCII file inside “raw”.

  • <id>_gaze.csv: EyeLink gaze coordinates extracted from the ASCII file inside “raw”.

  • <id>_participant.csv: A copy of the raw <id>_participant.csv without the experiment_name column.

  • <id>_targets.csv: A copy of the raw <id>_targets.csv without the experiment_name column.

  • <id>_trials.csv: A copy of the raw <id>_trials.csv without the experiment_name column and convenient trajectory labels.

Technical Validation

Data quality calculations can be replicated by following the instructions inside the data_quality.ipynb script. We looked at five indicators to assess data quality for each participant:

  • Average validation error: Reported by the EyeLink. Indicates how accurately the pupil position was tracked. Should ideally be below 0.5 degrees for adults with normal vision19.

  • Maximum validation error: Reported by the EyeLink. Indicates how accurately the pupil position was tracked. Should ideally be below 1.0 degree for adults with normal vision19.

  • Missing data: Percent of gaze samples with NA values. Indicates how prevalent blinks and tracking problems are in the data. Should be as low as possible.

  • Fixation precision: Root Mean Squared Error (RMSE) of gaze samples relative to the centroid of each classified fixation group in “jumping circle” trials. Reflects the spatial dispersion of gaze within fixations. Lower values indicate more stable, tightly clustered fixations on stationary targets.

  • Target alignment: Cosine similarity between gaze and target trajectories. Indicates how diligently participants followed targets with their eyes. Should be as close to 1 as possible.

Table 1 shows the results for all participants. We calculated the five indicators for all trials that followed (re)calibration of a given participant. Overall, the indicators suggest good data quality with the exception of trials 82 to 144 for participant 21db28aa, who took a two-minute break after trial 81 and was not recalibrated.

Usage Notes

In addition to the raw data, we share a companion Python package. Follow the installation instructions on GitHub (https://github.com/lukekorthals/pursuing-smooth-pursuits-data) and run the quickstart.ipynb script. This will make it easy to download, preprocess, and annotate the data; and includes basic plotting functions (see Fig. 1). In the following, we provide a brief overview of the preprocessing and classification protocols.

Fig. 1
Fig. 1
Full size image

These plots show one trial for each of the three movement patterns. They were created with the PARTICIPANT.PLOT_TRIAL() function and include participant and trial information, gaze and target positions, and the annotated benchmark labels. We used https://davidmathlogic.com/colorblind/ for colorblind friendly colors.

Cleaning

Raw data is cleaned by calling the CLEAN_RAW_DATA() function. Cleaning refers to extracting all relevant data from raw EyeLink and PsychoPy files, storing it as csv, removing unnecessary columns (e.g., experiment_name), setting convenient column names (e.g., xp becomes gaze_x), and setting convenient trajectory values (e.g., “ver_up” becomes “north”).

Preprocessing

Clean data is preprocessed by calling the PARTICIPANT.PREPROCESS_CLEAN_DATA() function. By default, the PREPROCESSOR.ORIGINALPREPROCESSOR(PREPROCESSOR) class is used to implement the procedure outlined below. You can adjust some parameters like the window size for blink removal or create your own subclass of the PREPROCESSOR.PREPROCESSOR() to implement your own preprocessing protocol.

Correcting Target positions for back-and-forth circle trials

Due to a bug, target locations collected during “back-and-forth circle” trials were shifted along their jumping positions (e.g., target at position A but position B collected). Due to another bug, the starting position of these targets was reversed for participants 06b8d2d3, 6cde27b5 and 21db28aa (e.g., target started at position B instead of position A). Consequently, the correct target positions are recalculated according to trial time, target speed, and trajectory.

Transforming gaze and target positions

Eye and target positions were initially collected in pixels; here they are converted to degrees and rounded to three decimals. Time measurements of eye and target positions are converted to seconds and rounded to three decimals. Eye positions were collected relative to the center of the screen, while target positions were collected relative to the top left corner. Therefore, eye positions are rescaled to match target positions. While eye positions were collected at 1000 Hz, target positions could only be collected at the monitor refresh rate of 144 Hz; consequently, missing target positions are filled with previous target positions.

Removing EyeLink delays

To ensure that the EyeLink was tracking during each entire trial, tracking started 100 ms before trial onset and ended 100 ms after trial offset. Consequently, all gaze samples before target onset and after target offset are removed.

Removing blinks

Blinks are characterized by scattered, fast samples when the eye tracker loses the pupil24. These artifacts could affect the velocity threshold approach we propose to distinguish saccades from fixations or smooth pursuits because of their high speed. Therefore, all samples the EyeLink classified as blinks were set to NA. Additionally, in line with the approach by Lüken et al.3, all samples 50 ms before and after the blinks classified by the EyeLink are also set to NA as they consist of very fast samples that may interfere with dynamic velocity thresholding. Like other parameters, users can adjust the window size for blink removal when using our companion package.

Classifying Plausible Benchmark Labels

As indicated above, stimuli were designed such that each trial should only include saccades (fast eye movement), and fixations OR smooth pursuits (slow eye movements). While we cannot rule out that (stationary) fixations occurred during smooth pursuit trials, because participants may have occasionally “zoned out”, we are confident that they generally followed the instructions because we provided continuous encouragement and monitored the eye-tracker signal. Evidence in the literature indicates that fast and slow eye movements can be distinguished based on their velocity2,3,4,5,8,10,16,25,26,27. This should be especially the case for the current dataset because the fastest target speed for smooth pursuits is around 6°/s while saccades typically show velocities upwards of 100°/s1. As long as participants diligently followed the targets with their eyes, “moving circle” trials should include smooth pursuits and (possibly) unintended saccades. “Back-and-forth circle” trials should include smooth pursuits and saccades. “Jumping circle” trials should include fixations and saccades. Based on these assumptions we developed the following protocol to create benchmark labels through dynamic velocity thresholds.

Note that this dataset can also be used without establishing benchmark labels in purely data driven endeavors. Additionally, we encourage other researchers to improve our protocol for establishing benchmark labels and/or developing different protocols altogether. For convenience, the following protocol is included in our companion Python package and can be used by calling the PARTICIPANT.CLASSIFY_PREPROCESSED_DATA() function as described in quickstart.ipynb.

Calculating dynamic thresholds

A dynamic velocity threshold to distinguish slow from fast eye movements is calculated for each trial. For a given trial the velocity is first calculated as the Euclidean distance divided by time. Then the threshold is determined by taking the median velocity and adding 1.5 times the 75th percentile velocity (see Equation (1)). This approach assumes that most gaze movements were slow — either due to stationary targets or consistently moving targets designed to elicit fixations and smooth pursuits. In contrast, saccades (either induced by abrupt target jumps or arising from participant inattention or blinks) were expected to produce brief bursts of higher gaze velocity, exceeding the computed threshold. We landed on the median, 75th percentile, and a scaling factor of 1.5 based on visual inspection of plots from arbitrarily selected trials of participants f7bb2338 and 6cde27b5.

$$\,{\rm{Threshold}}={\rm{Median\; Velocity}}+1.5\times {\rm{75th\; Percentile\; Velocity}}$$
(1)

Distinguishing fast and slow eye movements

All samples with velocities exceeding the threshold are classified as saccades. The remaining samples are classified as fixations for “jumping circle” trials, or as smooth pursuits for “moving circle” and “back-and-forth circle” trials.

Relabeling short events

Lappi (2016)1 reports that the shortest microsaccades have durations of 10 ms. We also use 10 ms as a very conservative estimate for minimum fixation and smooth pursuit durations to allow for short events. Fixations and smooth pursuits shorter than 10 ms are reclassified as saccades. Finally, any remaining saccades shorter than 10 ms are relabeled as fixations for jumping circle trials or smooth pursuits for moving circle and back-and-forth circle trials; thus, concluding classification.