Pilot, safety expert, consultant, and aviation journalist Stuart “Kipp” Lau writes about flight safety and airmanship for AIN.

Contributor - Safety

Stuart “Kipp” Lau

Researchers at Southern Methodist University (SMU) are developing an innovative approach that combines biometrics with machine learning techniques to reshape the future of flight training. The goal is to measure physical reactions of the pilot to provide—in real-time—a more objective and automated determination of performance to make flight training more personalized, effective, and efficient. Flight training programs have historically relied on subjective observations and post-flight analysis from an instructor to determine proficiency and mastery of a maneuver.
Teamed with simulator manufacturer and training provider CAE, researchers from SMU’s AT&amp;T Center for Virtualization are entering the fourth year of a project to develop and test methods to measure situational awareness and cognitive load sensing using biometrics and machine learning. The goal is&nbsp;to capture how pilots react to various scenarios in a flight simulator.
Machine learning is the study of algorithms that can improve automatically through experience and gathering data; it is seen as a part of artificial intelligence. The SMU project measures multiple physical reactions such as visual gaze patterns, pupil size, and heart rate to determine a pilot’s level of engagement, workload, situational awareness, stress, or fatigue. Remarkably, some of the early automated biometric test results align closely with the assessments of highly experienced human evaluators.
"Our theory is that biometrics during the simulation will result in much more objective and accurate measurements than asking users a few questions after the simulation to measure their experience,” said&nbsp;Suku Nair, director of the Center for Virtualization at SMU.
Eric Larson is the principal investigator on the study and&nbsp;is SMU’s associate professor of computer science and&nbsp;a recognized expert in machine learning with more than 53 patents and published papers. “Accelerating learning with biometric sensing is a difficult, unproven hypothesis," he said.&nbsp;"This research seeks to understand how sensing can be used to understand a learner’s mastery level in a difficult task, like flying an aircraft. We hope to advance the research field by being the first group to show whether personalized, automated learning can show efficacy in an actual learning scenario.” 
<figure style="margin:0;">
<div style="margin-bottom:5px;"><img alt="Biometric data captured during the experiment took place in a mixed-reality simulator setup using a virtual reality headset equipped with an integral eye-tracker." data-entity-type="file" data-entity-uuid="f16354ba-398e-4d56-8290-902e2d186ee1" src="/sites/default/files/heatmapbv2.jpg" width="2000" height="880" loading="lazy"></div>
<figcaption style="font-size:1.5rem;line-height:1.2;font-style:italic;">Biometric data captured during the experiment took place in a mixed-reality simulator setup using a virtual reality headset equipped with an integral eye-tracker.</figcaption>
</figure>

Much of the early interest in the project was in support of a Department of Defense priority to automatically measure mission-critical, higher-order cognitive constructs, such as situational awareness, to accelerate training of complex skills and support multi-domain warfare. In 2019, SMU and CAE (then L3Harris Technologies) first demonstrated that machine learning based on biometric data could yield real-time accurate performance results. During these demonstrations, pilot eye scan techniques—using heat maps—were used to determine levels of situational awareness and mental workload. &nbsp;&nbsp;
CAE was drawn to SMU for its expertise in biometric sensing and machine learning. “Our research will yield the first real-time measure of situational awareness, a critical high-order cognitive construct for dynamic, high stakes domains such as military aviation,” said&nbsp;Sandro Scielzo, a principal human systems scientist at CAE. “For example, our machine learning classifiers could identify a breakdown in perception allowing the remediation of poor visual scans.&nbsp;A breakdown in comprehension could also be mitigated by ensuring students remain within the zone of maximal adaptability via real-time training complexity adaptation. Thus, mission readiness could be achieved more effectively and rapidly.”&nbsp;
To date, much of the data collection effort consisted of a repeated measures experiment using 40 test subjects with various backgrounds and experience levels flying a mixed reality flight simulator in a controlled environment. The simulator uses virtual reality (VR) to replicate a military fighter jet and incorporates visualization, head-up displays, and high-precision hand tracking.
Data collection equipment included a VR headset with an integral eye-tracker and a wearable wrist-worn device to collect other biometric parameters. The eye-tracking system collected gaze patterns, pupillary response, and eye blinks. The wrist device collected heart rate measurements, galvanic skin responses, electrodermal activity (EDA), and wrist acceleration. These biometric data points are then correlated&nbsp;through computer analysis to determine levels of cognitive load or mental effort (workload), stimulation, or stress.
A&nbsp;lot can be learned from collecting this biometric data. As an example, a “poor” gaze pattern, depending on the phase of flight, can be indicative of a high workload. A “correct” gaze pattern would show a higher level of attention and performance. Likewise, fewer eye blinks or blinks of shorter duration can be correlated to tasks requiring greater attention. Heart rate and heart rate variability can be used for tracking effort while performing a mental task. In addition to looking at a singular biometric parameter, SMU’s research performed comparative analysis to identify if there was a higher level of correlation between different parameters. 
<figure style="margin:0;">
<div style="margin-bottom:5px;"><img alt="Measuring pilot workload was a key objective of this research" data-entity-type="file" data-entity-uuid="6a04552d-0ec4-4230-a591-6e12df0ca8f8" src="/sites/default/files/aggregate-high-resolution-heatmap.jpg" width="2000" height="880" loading="lazy"></div>
<figcaption style="font-size:1.5rem;line-height:1.2;font-style:italic;">Measuring pilot workload was a key objective of this research, and the idea is to determine if the pilot has excess capacity to perform additional tasks or is becoming overwhelmed.</figcaption>
</figure>

Key to this research is the ability to measure pilot workload and if there is excess capacity to perform additional tasks. Using biometrics and the machine learning algorithms, researchers could determine if the pilot subject was “loaded” or “unloaded.” Likewise, this project creates an automated means to objectively evaluate a student’s performance and the “level of ease” required to fly a maneuver.
Outside of the laboratory, the SMU/CAE team spent time at Edwards Air Force Base to demonstrate the feasibility and utility of the physiological sensor system as a flight test data source to objectively assess pilot workload. This test involved two&nbsp;flight test sorties in a Boeing C-17A jet and included aerial fueling maneuvers and lateral offset landings. A total of 33 maneuvers were recorded with good results. The study was deemed exploratory, and a hypothesis could be made that a “real flight” would capture a higher workload than the simulator data.
The use of biometrics and machine learning in a flight training environment may ultimately change the way pilots are trained. Physical reactions from a student may be a more reliable indicator of proficiency and mastery of a maneuver than the subjective view of an instructor or evaluator. While the SMU/CAE research validates the use of biometrics and machine learning on military aircraft, these processes, and techniques may be useful in civil aviation. The use of gaze patterns, as an example, may help train pilots to improve in areas such as active flight path monitoring, flight mode awareness, and the effectiveness of pilot monitoring duties.

News about simulators and training procedures, as well as workforce.

SMU and CAE Apply Biometrics to Flight Training

By measuring pilots' physical reactions during flight training, SMU and CAE researchers believe performance can be more objectively determined. 

2024 Corporate Aviation Leadership Summit

Matt Thurber

News and issues relating to business, corporate and private aviation, primarily regarding turbine-engine-powered airplanes and helicopters. Subjects include aircraft, engines, maintenance, operations, personnel, acquisitions, accidents, safety, security, and training.

News and issues concerning general aviation, specifically airplanes and helicopters powered by piston and alternative engines (i.e., non-turbine powered aircraft). Subjects include aircraft, engines, personnel, acquisitions, accidents, safety, security and training.

Why not teach new pilots to go around before learning how to land?

AINsight: You Can (and Should) Always Go Around

Kerry Lynch

News and developments about airplanes, helicopters and unmanned air vehicles used in all varieties of operations, including airlines, military, business, commercial, general aviation and private.

Most major manufacturers will be at static this year

NBAA-BACE Taking Shape with Expanded Program, Most OEMs

News and issues about aviation maintenance, repair and overhaul (MRO), covering all segments of the aviation industry; and modifications for aircraft and engines. Subjects include product support of aircraft, engines and avionics, MRO company mergers, acquisitions, announcements and personnel, Airworthiness Directives, Supplemental Type Certificate approvals and other topics.

News and issues relating to international air transport and cargo carriers, national airlines and regional airlines, including aircraft, engines, personnel, acquisitions, accidents, safety, security and training.

First step toward licensing could encourage youth to seek aviation careers

GAO Wants High Schoolers To Take Mechanic General Test

Charlotte Bailey

Essential news and analysis on the defense aerospace business. Your trusted source for new and ongoing programs and requirements in military aircraft (manned and unmanned), engines, avionics, weapons and associated systems, including ground-based air defense.

Evolving AI-augmented simulator-based approach to ‘gamify’ learning 

Saab Shaking Up Gripen Training with Project Skynapse 

Howard Hardee

News and issues relating to the aerospace industry that are related to the science, engineering and business of flight in space involving civil, industrial and military applications, and similar topics related to flight in the atmosphere of the Earth that are not covered more directly by other AINonline subject channels. In other words, both astronautical and aeronautical subjects, except for those aeronautical topics that are specifically covered elsewhere on this website.

Spirit AeroSystems workers give Leeham News and Analysis the inside track 

Cutting 'Traveled Work' Delays At Spirit Critical To 737 Output

Jessica Reed

Michel Denis was named CEO of Daher,  Steve Varsano president of Flexjet, and Robert Sumwalt was the next Wright Trophy recipient

People in Aviation —September 2026

Charles Alcock

News and issues regarding all manner of civil and military rotorcraft and their powerplants, including helicopters, tilt rotors and unmanned air vehicles.

News and information on safety procedures and concerns.

Austrian manufacturer has yet to say which rotorcraft types it will support 

Axis Announces Plans for Helicopter Training Systems

AIN Staff

In a Corporate Aviation Leadership Summit (CALS) roundtable on retention, an aviation staffing expert and flight department leaders traded hard-won lessons on money, culture, and knowing who you’re really competing with.

More In Training and Workforce