Problem

Air Force pilots conduct post-flight debriefing sessions to assess their performance and mistakes, but analog tools such as whiteboards and writing notes hinders their ability to effectively learn from missions by relying on memory.

Business goals

• Utilize avionics data collected on military aircraft
• Leverage AI for the identifying and grading of aerial maneuvers
• Plan for missions based on squadron or pilot proficiency

Research

User interviews

I conducted interviews with 4 pilots stationed at Seymour Johnson Air Force Base to understand their work habits, pain points, and expectations of flight debriefing in a digital workspace. The pilots are familiar with their jargon and onboard flight instruments so they didn’t want to relearn how to do simple things. Pilots wanted to know at a glance what maneuvers were flown and how well they performed each one.

Typically at the end of the debrief the student keeps some notes, but we erase our tapes. We erase our shot sheets and all the data gets flushed.

User flow

The primary flow we considered is uploading the acquired flight data, viewing the maneuver analysis, and adding comments about the sortie.

Usability testing

Releases were pushed at the end of every two-week sprint which gave us time to incrementally make changes and gather feedback from pilots such as:

• Renaming labels to be more familiar with Air Force pilots and instructors
• Organizing large sums of data to be more easily scannable
• Inclusive design patterns to accommodate for users mental models

It allows them to get an objective view of what is happening and how they flew and be able to make incremental, sometimes significant changes to how they're flying to be safer, more effective or more lethal

Technical limitations

While striving towards an ideal user experience, our team encountered a number of challenges throughout the development process:

• Our team used Google’s Material design library to jumpstart the implementation of our design system in Figma
• Some design components were made to suit our specific use case that are not a part of publicly accessible open-source design libraries
• Our team used CesiumJS to visualize the geospatial data in an interactive and 3-dimensional space

Design

Maneuvers

Detected maneuvers are classed by a passing, acceptable, or failed grading scale. Maneuvers that can’t be identified with certainty are considered possible detections. Selecting a maneuver jumps to that timestamp in the viewer to be replayed.

Analysis

Pilots can replay their recently flown sorties allowing them to pause, rewind, speed up, or jump to any maneuver on the timeline. The viewport enhances the pilot's ability to objectively break down their flight by simulating the flight as it happened, not as they remember it happening.

Pilot profile

Every pilot is placed into a percentile within their squadron based on the flight performance of their aircraft over a period of time. These grades can then be collected by pilot instructors to more accuractely assess the mission readiness of pilots.

Conclusion

Results

• Expanded software testing with 59 Test and Evaluation Squadron at Nellis Air Force Base
• Crowdbotics was the first company to successfully leverage the Maneuver Identification Challenge to advance the field of AI