Shaping the future of pilot training with AI-driven maneuver analysis

The challenge

The U.S. Department of Defense faced a critical issue: how to accurately assess mission readiness of pilots.

• Current approach: Post-flight debriefs that heavily rely on instructor feedback
• Missed opportunity: Each aircraft generates millions of data points during flight, but this data wasn’t being fully leveraged to evaluate and track performance.

To address this the 333rd Fighter Squadron at Seymour Johnson Air Force Base partnered with Crowdbotics. I joined a cross-functional team of developers, data scientists, and project managers to redesign the Data Driven Readiness (DDR) software. Our mission was to enhance how pilots analyze maneuvers and seamlessly integrate flight data into their daily workflow.

Research kick-off

To understand the problem we conducted usability tests of the current DDR software with five pilots. Pain points we uncovered:

• Unclear grades -> Ambiguous scoring made it hard for pilots to know how well they performed
• Awkward controls -> Limited descriptions made it difficult to view maneuvers from the right angles
• Inaccurate timestamps -> Maneuvers on the timeline didn’t match actual flight events, requiring tedious scrubbing
• Isolated notes -> Missing collaboration features made notes only useful as personal reminders

Key insights

Pilots valued DDR’s potential but needed usability improvements before they could fully rely on it. Our research revealed four design priorities:

1. Transparent grading system: Define a clear criteria with detailed explanations and visual indicators
2. Intuitive camera controls: Simplify the interactions and guide users
3. Accurate timestamps: Automatic alignment with actual maneuvers
4. Collaborative notes: Ability to tag team members, add context, and foster discussion

Mapping mouse and keyboard controls

The DDR viewport simulates the aircraft and maneuvers in the browser. Through our user testing it was revealed that:

• Default mouse controls were restrictive and disorienting
• No keyboard shortcuts were available

I reassigned the mapping of keyboard and mouse controls to:

• Simplify the controls for better accessibility
• Added keyboard shortcuts to quickly navigate the timeline

Reducing cognitive load

Promoting consistency reduced cognitive load and improved predictability across the platform. We reorganized DDR’s core layout into clear, repeatable sections:

1. Header -> Global navigation and system actions
2. Side panel -> Supporting details
3. Viewport -> Maneuver visualization
4. Timeline -> Performance metrics

Clarity to maneuver grades

Instead of vague sliding scales, we introduced parameter-level pass/fail grades. This helped pilots pinpoint areas for improvement without ambiguity.

Collaborative notes

Pilots can now tag teammates, trigger email notifications with mentions, and timestamp notes to reference exact moments in the timeline. These features increase the collaboration between pilots and instructors.

Real-time performance visualization

We layered user-generated and AI-driven notes directly onto the timeline reel. Pilots could now:

• View note indicators directly on the timeline
• Access to layered real-time aircraft performance data
• Cross-check AI evaluations using human judgment

Impact

We transformed DDR from being difficult to use into a collaborative tool that pilots trust to evaluate and improve their performance, while still fitting into their demanding workflows. The improvements that made the biggest impact for pilots were:

• Quickly identify performance gaps through transparent grading
• Navigate flight simulations more efficiently with intuitive controls
• Collaborate with instructors with interactive notes
• Cross-evaluations with human analysis from the enhanced visualizations

Final thoughts

This project was as much about adapting design methods to constraints as it was about building a better tool:

• Adapt to user constraints -> Pilots limited availability required leaner, flexible research methods
• Embrace the learning curve -> The aviation industry is highly technical. Building domain knowledge gave me confidence to drive design decisions
• Organize research better -> A dedicated repository would have made user research insights easier to share with stakeholders
• Bridge design and development -> By learning GitHub workflows and contributing to CSS tokens, I accelerated implementation and improved team collaboration