01 Overview

AI Gesture
Recognition
Vehicle Signals · ML Interaction

What if your car could read your hand gestures and speak to the world around it? This project applied machine learning to create a bidirectional communication system between driver intent and the vehicle's external environment.

ML · Gesture Recognition
Interaction Design
2024

My Role

Led end-to-end UX design for both communication flows: external awareness displayed in the cabin, and gesture-triggered external vehicle signaling.

Outcome

A functional prototype deployed and tested on a real vehicle, covering ML object detection, in-cabin gesture capture, and synchronized external light and projection output.

Impact

Directly informing next-generation intelligent vehicle experiences and establishing new patterns for human-machine interaction strategies at Ford.

02 Objectives

Redefining Vehicle-to-Environment Interaction

This project redefined vehicle-to-environment interactions, focusing on enhancing safety and clarity between the vehicle, pedestrians, and other drivers. Intelligent systems interpret driver intent and external conditions, enabling the vehicle to communicate unambiguously. A key scenario: a driver making an in-cabin "yield" hand gesture at a crosswalk, which the vehicle recognizes and translates into a clear external signal for others.

01

Leverage Machine Learning for object detection (humans and animals) and gesture recognition to facilitate intuitive inside-outside vehicle communication.

02

Reduce confusion and enhance safety by enabling clear, unambiguous external signaling of driver intent and vehicle awareness of surroundings.

03 Vehicle → Driver

External Awareness

The first communication flow surfaces what the vehicle detects outside and brings it clearly into the cabin, keeping the driver aware of nearby pedestrians and animals without taking their eyes off the road.

Outside Vehicle

Environment

Pedestrians · Animals

ML Detection

Vehicle

Sensor Fusion

Visual + Audio Cues

Inside Cabin

In-Cabin UI

Real-time Alerts

Driver

Informed & Aware

04 Driver → World

Gesture-Triggered Intent Communication

The second flow goes in the opposite direction: the system reads the driver's in-cabin hand gestures using ML recognition and translates them into external signals that communicate intent clearly to pedestrians and other drivers.

Hover a step to see more detail.

05 Prototype

Real-Vehicle Demonstration

The prototype went beyond conceptual: it was deployed and rigorously tested on a real vehicle, providing close-to-real-world experience of both ML object detection and gesture-triggered external signaling working in tandem.

06 Impact

Informing the Future of Intelligent Vehicles

This project culminated in a highly functional design prototype that was deployed and rigorously tested on a real vehicle, providing invaluable close-to-real-world experience. The insights gained are directly informing and accelerating future explorations into intelligent vehicle experiences and human-machine interaction strategies at Ford.

ML Integration

Demonstrated real-time object detection and gesture recognition as viable inputs for production vehicle UX.

Bidirectional Communication

Established patterns for simultaneous vehicle-to-driver awareness and driver-to-world intent signaling.

Real-World Validation