GestureTag: Combining Gaze with Touch Gestures to Acquire Targets for People with Motor Impairments



Intro clip:


Eye gaze tracking is a promising input alternative for people with motor impairments using computing systems. Although gaze pointing seems straightforward, activating (“clicking”) is not, and often causes unintentional clicks. Other alternative solutions such as performing eye gesturing also rose concern of eye fatigue issues while using these alternative gaze-based interactions in a significant period of time.

As a solution, we proposed GestureTag, a novel selection technique which aids motor impaired users in selecting targets on desktops by combining gaze pointing and simple touch swipe gestures for click activation in an adequate period of time, which could reduce the unintentional clicks as well.

(a) A user is trying to share a video on the YouTube webpage with GestureTag, which annotates targets on the screen covered by gaze area cursor with a simple touch gesture. (b) Buttons covered by the target area (blue circle) are assigned different gestures icons (yellow circular icons). (c) The user can then select the desired target by performing the corresponding swipe gesture on the trackpad. (a) A user is trying to share a video on the YouTube webpage with GestureTag, which annotates targets on the screen covered by gaze area cursor with a simple touch gesture. (b) Buttons covered by the target area (blue circle) are assigned different gestures icons (yellow circular icons). (c) The user can then select the desired target by performing the corresponding swipe gesture on the trackpad.


We included two mechanisim in our input technique:

(a) Targets within the region of interest (ROI) were assigned different gestures based on the angular coordinate of the target in the polar coordination where the center of the ROI is the pole. Swipe right gesture: 315°-45°(angular coordinate); swipe up gesture: 46°-135°; swipe left gesture: 135°-225°; swipe down gesture: 225°-315°. (b) The gesture assigned to the target changed when the ROI shifts.

The late trigger error caused by looking down to the trackpad while performing the gesture. The user uses a single tap to lock the ROI, so he can still select to the desired target while the gaze is out of the screen.

User study

We recruited 7 participants with motor impairments and 10 able- bodied participants and conducted a repeated-measures full factorial within-subject design study with three independent variables, including selection technique ((a) GestureTag: gazing at the desired area to trigger tag for each potential target and then perform the swipe gesture with the correct direction. (b) Gaze Dwell: gazing at the desired target over a certain of time to trigger the selection. (c) Gaze Smooth Pursuit: gazing at the desired area to trigger orbits for each potential target and then following one of orbiting dots with eyes to acquire the corresponding target), target size (16, 32, 48 dps), and target spacing (no spacing around the target, half-target width, full-target width spacing).

Different selection techniques we evaluated in the experiment.(a) GestureTag: gazing at the desired area to trigger tag for each potential target and then perform the swipe gesture with the correct direction. (b) Gaze Smooth Pursuit: gazing at the desired area to trigger orbits foreach potential target and then following one of orbiting dots with eyes to acquire the corresponding target. (c) Gaze Dwell: gazing at the desired target over a certain of time to trigger the selection.

In each condition, the user is asked to complete at least 10 unassisted trials in a row as quickly and correctly as possible. A Likert scale questionnaire was issued after the each selection technique type is completed. Finally, overall preference questionnaire was presented.

Results

Top part: Average error rate for the four selection techniques with different target size and spacing. Bottom left: Average selection time for the four selection techniques regarding different size of target. Error bars represent standard error. Bottom right: Users’ average ratings for different selection techniques. For physical, mental, effort, frustration and temporal, higher score represents a heavier workload. As for performance, ease of use and ease of learning, higher score means better preference. All ratings scale from 1 to 7. Error bars represent standard error.

Error rate:
Average error rates were 6.64% for GestureTag, 42.72% for Gaze Dwell, and 4.12% for Gaze Smooth Pursuit. Results of pairwise comparisons showed Gaze Dwell was significantly worse than Gaze Smooth Pursuit and GestureTag.

Workload:
GestureTag reduced mental and physical demand compared to Gaze Smooth Pursuit and Gaze Dwell.

Selection Time:
Average of completion time were 2.39 seconds for GestureTag, 3.26 seconds for Gaze Dwell Selection, and 6.63 seconds for Gaze smooth pursuit. the average acquisition rate of Gaze Smooth Pursuit was significantly slower than GestureTag, and Gaze Dwell.

Conclusion


Competition achievement

This work won the 2nd enterprise place for the 2018 undergraduate research competition held by CS department at National Taiwan University and was funded by the Ministry of Science and Technology (MOST) Taiwan.