Synthetic Data Automation: A Use Case for Pilot Training
This isn't your AI's synthetic data.
While the domain of pilot training could benefit from the development of novel analytics and artificial intelligence innovations, much of the data required to feed such applications are restrained by security and privacy requirements, especially in the DoD.
But what if the data were synthetically generated, right? GenAI is an option. Though AI-driven synthetic data is often... problematic. If there were only a better approach that was fast, cost-effective, transparent, secure, and accurate...
A different approach...
In our approach, all of the patterns and sequences of synthetic activity are modeled in an xAPI Profile -- a semantic data profile that can be designed by a human, run by a machine, and which is defined by open global standards.
The profile is fed into DATASIM where the researcher chooses parameters such as the number of learners being observed, the duration of the training session, and whether the activity should be weighted for increased stress and difficulty. The profile and the variable modifications are saved as a simulation specification and are reusable. So unlike black box solutions the behavior here is transparent, explainable, and able to be audited and modified either by humans or machines.
While attempts have been made to model airplane behavior and pilot activities in the cockpit, relatively little attention has been paid to modeling the pilot instructors themselves. We chose to create data profiles representing the act of evaluating pilot trainees. To do this, we created a digital representation of a pilot evaluation form and then modeled the behaviors and activities of pilot instructors during an evaluation including not only how they score pilots on different items, but things like how long it takes them to complete portions of the assessment, in what order they observe things happening, and when they choose to change initial observations based on later observations.
Human-Computer Synthesis
By modeling the human pilot training experience in data, we are able to feed the machine with the information it needs to create a simulated world in which the learning activity is carried out according to the patterns and sequences of possible and coherent activity described in the data profile.
The data produced by that simulated learning experience is then captured and validated just as though it were coming from a real-life human scenario. The key is to simulate the instrumentation of the digital interface points and human observations points that will allow for a high-fidelity simulation to occur. As all of the mechanics are available in a machine-readable format, there is no need for the simulation to occur in real-time -- but ideally, the synthetic data output from DATASIM is the same as if the simulated activity occurred in the real world.
What were the results?
We evaluated the technical conformance of the synthetic data to the xAPI data standard and found it to be 100% conformant at four levels of scale from one hundred data statements created to one million data statements created.
The synthetic data generation occurs over an amount of time that we would consider viable for ad hoc use.
We then evaluated the synthetic JSON code against a manually created baseline. DATASIM took advantage of optional fields and decisions available as xAPI data attributes (such as choosing different, but conformant ways to represent the identity of the pilot trainee or the instructor or by automating the creation of new session and registration IDs). Overall, DATASIM produced deterministic results entirely consistent with the concepts and patterns -- and temporal conditions -- coded into the xAPI Profile representing patterns of instructor behavior.
Finally we evaluated the semantic intelligibility of the results. While we designed a method for human evaluation, we found it to be too unwieldy to use at scale. So we've been experimenting with feeding resultant synthetic data into a GPT model and having it produce a human readable narrative based on the data. Currently, the GPT is able to turn our synthetic JSON code into a human readable narrative of basic consistency. And it is beginning to be able to identify gaps and inconsistencies in narrative flow -- such as if actions occur out-of-order or something breaks the logical flow of events.
We would like to continue the research -- refining the modeling capabilities while testing on a broader variety of training domain use cases.
What difference could this make?
In the near term, this data could be mined to better understand and predict instructor behaviors that improve pilot training. The result could be novel analytics that increase insight into the process of training pilots. The longer term implications are even more intriguing.
These analytics could leverage advances in learning science such as our understanding of spaced repetition, personalization, and optimal zones of learning development to increase the quality and efficiency of learning experiences. This is data that would enhance the design of AI instructors who could augment the capability of human instructors to meet the demands of a workforce that is experiencing generational change.
It's also an approach that can be applied to all training -- every training event could be represented in a digital data profile. The synthetic data available through those profiles would increase the ability to experiment with and improve training systems at all levels and in any vertical -- including creating connections between activity occurring during a learning experience and the assertion of competency.
Background
This project was sponsored by the Air Force Research Laboratory. Initial development of DATASIM was sponsored by the Advanced Distributed Learning Initiative. The project was presented in a poster session at the 2024 Emerging Technologies for Defense Conference and Exhibition (Emerging Technologies Institute / NDIA).
Prior Relevant Research and Standards Publications
Blake-Plock, S. (2024). DATASIMx for Pilot Training Generation of Synthetic Exemplar Data with Relevant Tactics. Final report. Air Force Research Laboratory. FA238423C0005.
Blake-Plock, S. (2021). DATASIM Option Year 1 Final Report (TRL 5). Advanced Distributed Learning. HQ003419C0061. DTIC: AD1141220.
Blake-Plock, S. and Casey, C. (2020). DATASIM and xAPI Profile Validation. iFEST 2020. National Training and Simulation Association. Alexandria, VA.
Blake-Plock, S. (2019). DATASIM: Data and Training Analytics Simulated Input Modeler for xAPI and xAPI Profiles. iFEST 2019. National Training and Simulation Association. Alexandria, VA.
IEEE 9274.1.1-2023 IEEE Standard for Learning Technology--JavaScript Object Notation (JSON) Data Model Format and Representational State Transfer (RESTful) Web Service for Learner Experience Data Tracking and Access. https://standards.ieee.org/ieee/9274.1.1/7321/
IEEE P9274.2.1 Standard for JavaScript Object Notation for Linked Data (JSON-LD) for Application Profiles of Learner Experience Data. https://standards.ieee.org/ieee/9274.2.1/10570/
Robson, R., Goldberg, B., Blake-Plock, S., Casey, C., Hoyt, W., Hernandez, M., Ray, F. (2022). Mining Artificially Generated Data to Estimate Competency. Proceedings of the 15th International Conference on Educational Data Mining. Durham University. Durham, UK.