Research Student — LLM Drone Safety

COSMOS is the California State Summer School for Mathematics and Science. I was in UC Santa Cruz’s secure autonomous systems cluster as a rising junior, mostly surrounded by seniors. The first half was cybersecurity and the second half was AI. On day 3 there was a nine-challenge security competition, from OSINT to steganography, and I won it. The prize was a 3D-printed pair of rotor ciphers.

The question

LLM-based drones are increasingly trusted to make their own safety calls. But how consistent are those calls when the sensor and weather data are ambiguous? If a model’s “is it safe to fly?” answer flips under borderline conditions, that instability is a vulnerability in itself. An attacker who can nudge the inputs can nudge the decision.

The framework

I built an automated pipeline on Cosys-AirSim (a fork of Microsoft’s AirSim) that generated 200 scenarios: 8 visual weather conditions crossed with 5 METAR aviation-weather patterns. Each scenario went to OpenAI’s o1-mini along with simulated GPS, barometer, IMU and magnetometer readings, and the model had to decide whether flying was legal and safe under FAA Part 107. A custom uncertainty score measured how often the answer changed.

What it found

The inconsistency peaked at about 75%, but not where I expected. Bad weather got consistent, correct refusals, because at the extremes the right answer is obvious. The model became unstable in the middling conditions, the most normal-looking ones. So the better the flying conditions looked, the easier the decision was to push around.

The recommendation that came out of it carries over to any safety-critical LLM system: pair the model’s reasoning with deterministic, rule-based overrides for ambiguous inputs instead of trusting its judgment alone.

COSMOS ended with a symposium, but my interest in autonomous systems kept going. It’s what led to the V2X intrusion-detection project that winter.