2025

Project Info

Client:Apple Academy 5th Project
Duration:1 month
Technologies
Swift
SwiftUI
ARKit
SwiftData
iPadOS
AAC

Table of Contents

Eyespeak | Assistive Tech

An iPadOS AAC (Augmentative and Alternative Communication) device that empowers people with motor neuron diseases to communicate again. Uses advanced ARKit facial recognition to control a full communication board through natural expressions—winks, eyebrow raises, puckers, and more. Features combo inputs for enhanced accessibility and SwiftData for persistent storage.

Eyespeak | Assistive Tech - Image 1

The Challenge

Creating an accessible communication solution for individuals with motor neuron diseases who have lost the ability to speak or use traditional input methods. The challenge was to develop an intuitive system that allows users to communicate effectively using only facial expressions, while ensuring reliability, speed, and ease of use. The app needed to work seamlessly on iPadOS and provide a full-featured AAC board experience.

Our Solution

Built an iPadOS app using SwiftUI and ARKit to detect and interpret facial expressions in real-time. Implemented a sophisticated combo input system that combines multiple facial gestures (winks, eyebrow raises, lip puckers) to create a comprehensive control scheme. Integrated SwiftData for persistent storage of user preferences, custom phrases, and communication boards. Created an intuitive AAC board interface with customizable phrases, quick replies, caregiver prompts, and emoji support. The facial recognition system processes natural expressions without requiring controllers or external devices, restoring independence and voice to users.

My Role

I served as the iOS Engineer.

Technical Highlights

  • •ARKit Facial Recognition: Real-time detection and interpretation of facial expressions including winks, eyebrow raises, and lip gestures
  • •SwiftData Integration: Persistent storage for user preferences, custom AAC boards, and communication history
  • •Combo Input System: Advanced gesture combination system allowing multiple facial expressions to trigger different actions
  • •iPadOS Optimization: Designed specifically for iPad with optimized UI layouts and touch interactions
  • •SwiftUI Architecture: Modern declarative UI framework for responsive and accessible interface design
  • •Accessibility Features: Full VoiceOver support and customizable interaction settings for different user needs