← back

Boston University Security Lab (SeclaBU), RISE

Research Intern — VibeAuthBench

Jun 2026 – Aug 2026

Boston, MA · Advisors: Prof. Gianluca Stringhini, Prof. Manuel Egele · Supervisor: Khosro Moeini

  • research

Boston University Security Lab (SeclaBU), RISEVibeAuthBench repo

highlights.md
  • Conceived and built VibeAuthBench (VAB), a benchmark that measures the vulnerabilities and misconfigurations AI coding agents introduce when adding a feature (Sign-In with Google) to existing open-source applications with Python, Go and Node.js backends.
  • Designed a two-stage pipeline: functional verification of the sign-in flow, then an LLM-as-judge security review against OWASP and Google OAuth guidelines, scored with CVSS 4.0 weights.
  • Served open-weight models (up to 480B parameters) with vLLM on BU's Shared Computing Cluster across 30 runs. Agents produced a working sign-in in only 5 of 30 runs; 163 of 172 security findings were introduced by the new code, and security varied by language.
  • Presented at the BU RISE Symposium (Aug 2026); poster accepted at IEEE ACSAC 2026. Selected as one of 85 interns from 3,800+ applicants (2.2%).

Where the idea came from

I once came close to shipping private account keys inside a function in one of my own iOS apps. That near-miss made me wonder how much developers should trust code they didn’t write line by line, including code from AI agents. A coding agent can write a clean snippet in isolation, but real use means dropping it into an existing codebase and asking it to add a whole feature. That’s the question I took to Boston University’s Security Lab for the summer.

Before the program started, I’d asked Prof. Stringhini what a project could look like. The lab works across mobile security, software security and AI agent security, and I picked agents. I read through the lab’s earlier work, including CVE-Genie, which uses agents to reproduce and test CVEs, then did a literature review of the existing secure-code benchmarks. Most of them test small snippets with no surrounding context. The few that use real repositories tend to stick to one language. I worked through about nine candidate ideas, and by day 9 of the 42-day program VibeAuthBench was fully formed.

What VAB does

The task is something developers really do ask agents for: “Integrate Google OAuth into my app.” The agent gets one attempt, a plain prompt, and no re-prompting.

  • Targets. I programmatically parsed the awesome-selfhosted dataset and narrowed 423 candidate apps down to 3 with comparable functionality, no existing OAuth/OIDC login, and three different backends: Baby Buddy (Python), Traggo (Go) and Karaoke Eternal (Node.js). The no-existing-auth rule was the key scoping call, because it forces the agent to build sign-in from scratch.
  • Agents. Two open-weight models, Qwen3-Coder-480B and Qwen3.6-27B, served with vLLM on BU’s Shared Computing Cluster and run inside the Claude Code harness through an Anthropic-compatible endpoint. Each app ran in its own Docker container.
  • Stage 1: does it work? I checked every run by hand. Did the agent give the redirect URI to register with Google, and did a fresh account actually sign in end to end?
  • Stage 2: is it secure? A separate LLM-as-judge agent reviewed each modified codebase against rubrics built from the OWASP OAuth 2.0 Cheat Sheet and Google’s OAuth best practices. Findings are rated high, medium or low, marked as new or as pre-existing issues the change made reachable, and scored with CVSS 4.0–based weights. Lower is better.

What it found

I expected the agents to do much better than they did.

  • 5 of 30 runs produced a working end-to-end sign-in, even though 28 of 30 gave the developer sensible next steps.
  • 172 security findings, 163 of them in the agents’ own new code. Agents were far more likely to introduce new vulnerabilities than to expose dormant ones.
  • Language mattered. Both models scored best on the Python app (23.14 and 24.04) and noticeably worse on Go and Node.js (up to 39.71).
  • The most common mistakes were over-privileged access tokens, missing PKCE, and careless handling of client credentials and user tokens. In one run the agent committed the .env file with the secrets and never added it to .gitignore, apparently assuming the app would never leave a developer’s laptop.

The pipeline is modular on purpose. You can swap the harness, the feature request, the base app or the rubric and rerun it. Next steps are frontier models and other harnesses, deterministic SAST tools alongside the judge, and re-prompting on failure to look more like real “vibecoding.”

What I took from it

This was the first time I designed an entire project myself instead of being handed the criteria, with professors who pushed back on my ideas. The most useful advice was to think about generalizability and to start with a minimal version before scaling up. I’m fairly sure I’d have buried myself without it.

I presented VAB at the RISE Symposium in August. Afterwards I wrote up and submitted a poster proposal to IEEE ACSAC 2026 on my own, and it was accepted for the poster session in Los Angeles this December.

↑↓ navigate · ↵ select · esc close