Challenge

The U.S. Department of Defense knew it had a problem determining the proficiency level of pilots. While debriefs are held to evaluate the pilot’s most recent flight, there are also millions of data points recorded onboard their aircraft being wasted that could be used to measure and track pilot performance.

The 333rd Fighter Squadron at Seymour Johnson Air Force Base began looking at technologies and apps that address these problems, when Julian Capps from Crowdbotics recommended a larger research and design effort to understand usability problems with the Data Driven Readiness (DDR) software design. Along with Julian and myself, the core team during this phase included the development team, data scientists, and project management.

Kick-off

Identifying pain points in the maneuver analysis experience

At the start of the project, we didn’t have clear goals for improving the maneuver analysis experience. To gain insights, I teamed up with our data scientists and key stakeholders to learn how pilots were using the software.

We tested the current DDR application with five participants at Seymour Johnson Air Force Base to understand their experiences. Our goal was to identify the challenges they faced, such as:

• Grades are unclear: Pilots found it hard to understand how well they performed maneuvers because the grades were ambiguous.
• Camera controls are awkward: Without clear descriptions pilots struggled to change the camera to the desired angles.
• Timestamps are off: When a maneuver starts and ends on the timeline isn’t when maneuvers were actually performed. An additional amount of scrubbing was required to find the exact start time of a maneuver.
• Notes feel isolated: The notes feature was only used for personal use, offering little in terms of collaboration or having discussions.

"It allows them to get an objective view of what is happening and how they flew and be able to make incremental, sometimes significant changes to how they're flying to be safer, more effective or more lethal."

The discovery

Streamlining DDR integration into pilots workflow

While some usability issues were uncovered, pilots appreciated DDR’s ability to provide immediate feedback from flight data and personalized input from instructors. They want to keep using DDR in their workflow but prefer a design that doesn’t require much effort to relearn. As the platform grows, so have their expectations.

These insights were crucial. If even our early adopters—highly skilled and educated pilots—faced usability challenges, users in other contexts would likely struggle even more.

From our research, a few key patterns emerged:

• Clear and transparent grading system: Develop a grading system with clear criteria, detailed explanations, and visual indicators to help pilots better understand their performance.
• Easy-to-use camera controls: Design user-friendly camera controls with intuitive descriptions, tooltips, and an interactive tutorial to make navigation easier.
• Accurate and automated timestamps: Introduce automatic, precise timestamping that aligns with the start of each maneuver, making it quicker to locate events on the timeline.
• Collaborative and interactive notes: Expand the notes feature to include collaborative tools like tagging team members to foster teamwork and discussions.

Human-computer interaction

Mapping out the mouse and keyboard controls

Since the viewport simulates the aircraft and maneuvers in the browser we included this as part of our testing with participants. We learned that the default mouse controls were difficult to orient the camera view with and that no actions were bound to the keyboard.

Based on the feedback received we simplified the mouse controls to be easier to approach. We also introduced keyboard controls to timeline actions for quick access by power users.

Content guidelines

Refining the information architecture

Defining areas of the page into sections allowed our team to communicate and limit design decisions while customers can safely expect to see these patterns reused in different ways throughout the product experience.

This helps reduce cognitive load by displaying information that's visible, predictable, and consistently applied. We further enhanced the page template by including a header section to support global navigation links and system-level actions that are accessible from all pages across the product.

Bringing clarity to aircraft maneuver grades

Instead of assigning grades on a sliding scale to each maneuver, a simple pass or fail grade is assigned to each parameter to aid pilots in narrowing down areas of improvement. We focused on making content easier to read so that information can be processed more efficiently.

Stay connected to your team members

We made it easier to keep everyone in the loop by tagging team members and sending notifications when mentioned. The addition of note timestamps was also introduced so that pilots are able to mark the exact point on the timeline being referred to. While these changes make notes more interactive it also encourages social engagement between pilots and instructors.

Visualizing real time aircraft performance

Maneuver and note indicators are now both shown on the timeline reel to give an overview of where events have happened at. Real time information of the aircraft's performance is also visualized on the timeline to raise the level of information awareness, which opens the opportunity for humans to manually inspect maneuvers graded by artificial intelligence.

Final thoughts

• Adapt your research methods to user constraints Talking to pilots provided valuable insights, and their limited availability highlighted the need for a more flexible research approach. Adapting methodologies to fit our users constraints could have made our process more effective.
• Embrace the learning curve: The project's complexity was a challenge, and gaining industry-specific knowledge was key to building up my confidence. Knowing how to navigate through uncertainty became an essential part of solving design problems and taking the lead.
• Organize a research repository: While Figma worked well for design assets, research and documentation grew harder to access over time. A better system for organizing research would have made sharing insights with stakeholders easier.
• Be a GitHub contributor: Working with developers taught me how CSS frameworks leverage design tokens. Implementing tokens in our design system allowed me to then make small updates to front-end code, speeding up the development and enhancing team collaboration.

“Hiring Chris brought us instant relaxation. Having someone dedicated to solving usability and design problems freed up other team members to do their jobs. Before working with Chris I didn’t realize how much it would take to do the research sitting down with pilots to better solve their problems.”