GestureTag: Combining Gaze with Touch Gestures to Acquire Targets for People with Motor Impairments

Intro clip:

Eye gaze tracking is a promising input alternative for people with motor impairments using computing systems. Although gaze pointing seems straightforward, activating (“clicking”) is not, and often causes unintentional clicks. Other alternative solutions such as performing eye gesturing also rose concern of eye fatigue issues while using these alternative gaze-based interactions in a significant period of time.

As a solution, we proposed GestureTag, a novel selection technique which aids motor impaired users in selecting targets on desktops by combining gaze pointing and simple touch swipe gestures for click activation in an adequate period of time, which could reduce the unintentional clicks as well.

Pantograph (a) A user is trying to share a video on the YouTube webpage with GestureTag, which annotates targets on the screen covered by gaze area cursor with a simple touch gesture. (b) Buttons covered by the target area (blue circle) are assigned different gestures icons (yellow circular icons). (c) The user can then select the desired target by performing the corresponding swipe gesture on the trackpad.

We included two mechanisim in our input technique:

User study

We recruited 7 participants with motor impairments and 10 able- bodied participants and conducted a repeated-measures full factorial within-subject design study with three independent variables, including selection technique ((a) GestureTag: gazing at the desired area to trigger tag for each potential target and then perform the swipe gesture with the correct direction. (b) Gaze Dwell: gazing at the desired target over a certain of time to trigger the selection. (c) Gaze Smooth Pursuit: gazing at the desired area to trigger orbits for each potential target and then following one of orbiting dots with eyes to acquire the corresponding target), target size (16, 32, 48 dps), and target spacing (no spacing around the target, half-target width, full-target width spacing).

In each condition, the user is asked to complete at least 10 unassisted trials in a row as quickly and correctly as possible. A Likert scale questionnaire was issued after the each selection technique type is completed. Finally, overall preference questionnaire was presented.


Error rate:
Average error rates were 6.64% for GestureTag, 42.72% for Gaze Dwell, and 4.12% for Gaze Smooth Pursuit. Results of pairwise comparisons showed Gaze Dwell was significantly worse than Gaze Smooth Pursuit and GestureTag.

GestureTag reduced mental and physical demand compared to Gaze Smooth Pursuit and Gaze Dwell.

Selection Time:
Average of completion time were 2.39 seconds for GestureTag, 3.26 seconds for Gaze Dwell Selection, and 6.63 seconds for Gaze smooth pursuit. the average acquisition rate of Gaze Smooth Pursuit was significantly slower than GestureTag, and Gaze Dwell.


Competition achievement

This work won the 2nd enterprise place for the 2018 undergraduate research competition held by CS department at National Taiwan University and was funded by the Ministry of Science and Technology (MOST) Taiwan.