I built vexproof to answer the question dependency scanners skip: can your Python code actually reach the vulnerable function? Across 231 alerts in two Django projects, it cleared 30 with evidence and proved 29 reachable with verified call paths. When I ran both projects and recorded what executed, none of the 30 clears was contradicted. The model reads what code can’t, and deterministic checks stop it from inventing a function, a quote, or a call step.
I built it during the Meta Global AI Developer Hackathon, which ran from October 26 to November 4, 2026. Entry was by application, and the event admitted up to 20,000 developers. I was accepted into the Indie Developer track and made the first commit on October 27. vexproof was one of 5,367 projects submitted, and you can find it in the hackathon’s project gallery.
At a glance
- Problem: scanners match package versions against advisories, so teams get long alert lists and no way to tell which alerts matter.
- What I built: a scanner and web app that use static analysis and Muse Spark to clear an alert, prove it reachable, or leave it open with a reason.
- Result: 0 of 30 clears contradicted by the running projects, 0 of 1,246 scanned Python modules that loaded were missing from the analysis, and 25 of 29 reachable functions confirmed by running them.
- Stack: Python 3.12, Muse Spark Open through Ollama, Syft, OSV, CISA KEV, FIRST EPSS, OpenVEX, FastAPI, and Docker.
- Source: vexproof on GitHub, a 3-minute demo video, and the hackathon gallery entry.
The problem
A security advisory says, in effect, “Pillow versions before 8.1.2 have a bug in the function that reads ICO images.” A scanner checks only the first half of that sentence. If your project installs Pillow 4.3.0, you get an alert. Nothing checks whether your code ever calls the function with the bug.
Answering that by hand means reading the advisory, finding the fix commit, and tracing calls through the framework, for every alert. Three things make it hard to automate:
- Advisories are prose. They describe the bug in words and rarely name the function at fault.
- Fix commits are broad. Pillow’s commit
756fff33fixes ICO, ICNS, and BLP bugs together, so “the function this commit changed” is often the wrong answer. - Python is dynamic. Frameworks load code by name, and method calls resolve at run time. Plain static analysis either misses paths or invents them.
An LLM helps with the first two. It’s also the wrong thing to trust with a security verdict: if a model tells a team that a reachable vulnerability doesn’t matter, the team has lost more than the tool ever saved them. That tension shaped the design.
Requirements and constraints
- Must: back every verdict with evidence that someone else can check, down to the file and line.
- Must: treat a wrong “not affected” as the one error the tool can’t make. Failing to find a call path never clears an alert.
- Must not: run any code from the project it scans, since that code is untrusted.
- Must not: let a model answer change a verdict before it passes deterministic checks.
- Non-goal: other languages. I chose depth on Python over a shallow claim about several ecosystems.
- Non-goal: clearing every alert. An honest “I can’t decide, and here’s why” beats a confident guess.
Architecture

Here’s one alert going through the pipeline. GHSA-95q3-8gr9-gm8w is a denial-of-service bug in Pillow’s ICO decoder, flagged in a small Django shop:
- Collect. Syft lists the shop’s dependencies, and OSV reports the advisory against Pillow 4.3.0. vexproof fetches the fix commit with
git, maps the changed lines to functions, and confirms they exist in the installed version. That leaves two candidates. - Pick. Muse Spark reads the advisory and both diffs, picks
IcoFile.frame, and quotes the advisory: “reported size of a contained image is not properly checked for an ICO container.” The verifier confirms the name is a real candidate and the quote is really in the advisory. - Prove. A search finds a path from the shop’s admin code to that function. The admin’s upload form validates images with Django’s
ImageField, which callsImage.open, which tries each decoder Pillow has registered. - Verify. The verifier checks all seven steps at their cited lines. Every step is certain, meaning the code itself shows what gets called, so this path needs no model judgment.
- Report. vexproof marks the alert fix now, because it’s reachable and high severity in an internet-facing service, and adds it to the upgrade plan and the OpenVEX export.
Key decisions and trade-offs
Decision 1: Who gets to decide a verdict?
- Options: an agent that reads the code and decides, static analysis alone, or a model that only proposes answers for code to check.
- Chose: the model proposes and a verifier proves. The model chooses only among functions that deterministic analysis already found, in both the fix commit and the installed code. Each pick must quote the advisory, and the verifier rejects any invented name or quote that isn’t there.
- Why: the model can’t invent a function, a quote, or a call step, so the most obvious LLM failures can’t reach a verdict. Advisory text and diffs are also untrusted input, which makes prompt injection a real risk. vexproof wraps them in data tags that the text can’t close, with a test for that, and every answer still has to pass the verifier. That limits what injected text can do, but it can’t stop it from steering the model toward a wrong real candidate.
- What this doesn’t stop: the verifier checks grounding, not meaning. The model can still pick the wrong real candidate with a real quote, and because clears follow the picked function’s module, a wrong pick could support a wrong clear if the fix also touches a loaded module. So I score picks against hand-checked answers, and I checked this case directly: in the recorded scans, none of the 30 clears depended on a pick, because no other module changed by those fixes was loaded.
- Cost: 66 of the shop’s 94 alerts stay open. That’s the price of never guessing, so vexproof reports the rate instead of hiding it.
Decision 2: How much can you safely clear?
- Options: clear at the function level when no path is found, or clear only when nothing can load the module that holds the bug.
- Chose: module-level clears. If anything loads code by a name vexproof can’t see, it refuses to clear.
- Why: function-level “not reachable” is unsound in Python. A call such as
getattr(self, request.method)can reach code that no static search can rule out. - Cost: fewer clears, and frameworks need hand-written rules for how they load code. A framework without rules means more open alerts, which is the safe way to fail.
Decision 3: How do you find a call path through dynamic code?
- Options: a name-based search, which finds absurd shortcuts such as
Product.objects.filter()“reaching” Pillow’sImage.filter. Or a model asked to find the whole path, which is slow and can invent steps. - Chose: a search proposes, the model reviews, and the verifier checks. The search treats a hop as certain when the code shows the callee and as uncertain when it doesn’t. The model reviews only the uncertain hops, and it can steer the search but can’t add a hop of its own.
- Why: deterministic search covers most of the work, and the model’s cost and risk are limited to the steps that really need judgment. In the shop scan, 11 of the 14 reachable paths needed no model review at all.
- Cost: a verified path shows that each call is in the code, not that every condition along the way holds at run time. It doesn’t prove that an exploit works.
Decision 4: How do you evaluate a tool that refuses to run code?
- Options: hand-label results, which can’t scale to hundreds of module loads, or run the project and compare.
- Chose: a separate tool that runs the project in a throwaway environment under a scripted workload, then records every loaded module and every function called. The scanner reads only the resulting JSON, so it still never runs target code.
- Why: a wrong clear is the worst possible error, so it needs an independent check rather than my own opinion.
- Cost: a run can disprove a clear but never prove one, because another workload might load that module. I also wrote the workloads myself, so their coverage is only as good as my imagination, although anyone can extend them.
How I know it works
I scanned two projects and then ran each one, with its exact pinned packages, under a workload covering browsing, cart, checkout, admin login, and image uploads. One is a tutorial shop that I published in December 2017 and January 2018, and the other is the third-party justdjango/django-ecommerce.
| Measure | Django shop | justdjango | Together |
|---|---|---|---|
| Alerts from version matching | 94 | 137 | 231 |
| Cleared with evidence | 14 | 16 | 30 |
| Reachable, with a verified call path | 14 | 15 | 29 |
| Clears contradicted by the run | 0 of 14 | 0 of 16 | 0 of 30 |
| Loaded modules missing from the analysis | 0 of 470 | 0 of 776 | 0 of 1,246 |
| Reachable functions that ran | 13 of 14 | 12 of 15 | 25 of 29 |
| Open alerts | 66 | 106 | 172 |
The module count excludes 90 loaded modules with no Python source file to scan, such as compiled extensions and aliases. The second-to-last row is the strongest test in the table: every module that loaded had to be one the scan said could load.
I also measured what the model adds by comparing the shop scan with what the same pipeline does without it:
| In the shop scan, Muse Spark… | Alerts | Without the model |
|---|---|---|
| Reviewed an uncertain step, so vexproof could prove the alert reachable | 3 | The alerts stay open |
| Picked the one function the advisory is about, out of several the fix changed | 11 | Paths lead to any changed function, including unrelated ones |
| Found that none of the changed functions matches the advisory | 4 | vexproof could report a path that has nothing to do with the bug |
| Changed a clear | 0 | Clears rest on module loading alone |
Function picks matched my hand-checked answers in 22 of 22 comparisons, using 14 labels applied across both projects. That’s a small set, so I read it as “hand-checked and reproducible”, not as a general accuracy rate. The verifier rejected no picks in the shop scan, which means it worked as a guarantee here rather than something I can show catching a bad answer.
The rest of the engineering is covered by 355 offline tests at 96% coverage, strict type checking with mypy, and CI. Every model answer is recorded, keyed by model, prompt version, and inputs, so a re-run returns the same result. You can check the recorded scan yourself without a model or an API key:
$ uv run vexproof show examples/demo/findings.json GHSA-95q3-8gr9-gm8w
picked: PIL.IcoImagePlugin.IcoFile.frame (muse-spark-open:cloud / symbol_pick.v1)
advisory says: "reported size of a contained image is not properly checked for an ICO container"
call path from your code (all hops certain)
shop.admin shop/admin.py:20
django.forms.fields.ImageField.to_python django/forms/fields.py:626 via django_admin_upload
PIL.Image.open PIL/Image.py:2549 via call
PIL.ImageFile.ImageFile.__init__ PIL/ImageFile.py:102 via pillow_open_registry
PIL.IcoImagePlugin.IcoImageFile._open PIL/IcoImagePlugin.py:264 via call
PIL.IcoImagePlugin.IcoImageFile.load PIL/IcoImagePlugin.py:267 via call
PIL.IcoImagePlugin.IcoFile.getimage PIL/IcoImagePlugin.py:150 via call
PIL.IcoImagePlugin.IcoFile.frame PIL/IcoImagePlugin.py:153 via callWhen the workload uploaded an ICO image through the admin, IcoFile.frame ran, which confirms the path.
Results
- A shorter list: across both projects, 59 of 231 alerts (26%) go from “maybe” to a decision with evidence, and 29 of them are the ones to look at first.
- A concrete fix: for the shop, Pillow 13.0.0 or later and Django 4.2.26 or later close every reachable alert.
- Evidence other tools can use: the OpenVEX export carries the evidence with each statement, so a “not affected” claim can be checked by someone else.
- A plausible cost: I estimate $0.95 in model cost for a fresh shop scan and $3.61 for justdjango. These are estimates at per-token rates I set myself, because I ran the model through Ollama’s cloud service, which doesn’t bill per token.
The honest headline is the other 74%. Most alerts stay open, usually because there’s no fix commit to read, the fix is in native code, or the search found no path. vexproof narrows the list and proves what it can. It doesn’t replace a person reviewing the rest.
How the build went
I built vexproof as a solo developer, and the commit history records every step. The order mattered as much as the code:
- October 27: I set up CI, dependency ingestion, and advisory lookups, and wrote the first decision records. The first rule was that vexproof never runs the code it scans.
- October 28: I built the import analysis and the fix-commit parsing. No model was involved yet, so this layer gave me facts the model couldn’t change.
- October 29: I connected the model, wrote the answer key, and built the call-path search and verifier. The evaluation against the running project also started this day, and it found the migration bug described in the next section. The hosted API’s billing setup wouldn’t complete, so I ran Muse Spark Open through Ollama, which the client’s OpenAI-compatible interface made possible.
- October 30: I built the web app and Docker image, then scanned projects I hadn’t tuned for. That produced the wrong clear described in the next section and the second evaluation project.
- November 2-3: I resolved the answer key disagreement and wrote the documentation and demo video for people who had never seen the project.
Connecting the model on day three is the decision I’d repeat. By then, the facts its answers had to agree with already existed.
What went wrong and what I learned
My first evaluation run found a real hole. Django imports every migration file by name, and a filename such as 0001_initial.py isn’t a valid Python identifier, so my source reader skipped those files. The analysis missed 14 modules that loaded at run time. Unit tests hadn’t caught it. Running the real project did.
A Flask scan produced a wrong clear, the exact error the tool exists to avoid. It cleared SQLAlchemy even though Flask-SQLAlchemy imports it. SQLAlchemy 1.1 keeps its source under a lib/ folder, so vexproof read its modules as lib.sqlalchemy..., and import sqlalchemy matched nothing. Fixing that one layout wasn’t enough, because the next unusual layout would cause the same error. So I also added a rule: if an import goes unmatched and looks like a package’s name, vexproof refuses to clear that package.
The model disagreed with my answer key, and the model was right. I had labeled a Pillow BLP advisory as “none” because the shop’s Pillow 4.3.0 has no BLP decoder. On justdjango’s Pillow 6.2.2, Muse Spark picked BLP1Decoder._decode_jpeg_stream, and the evaluation marked it wrong.
Changing a label because a model disagreed with it is how evaluations quietly stop meaning anything, so I checked the facts without the model first. The 4.3.0 wheel on PyPI has no BlpImagePlugin module, and the 6.2.2 wheel does. The fix commit adds the missing size check to that exact function. The right answer depends on the installed version, so each label now names the versions it applies to. I made the change in its own commit so that anyone can review it. Without that correction, the score would be 21 of 22.
The lesson I took from all three: evaluating against something independent of your own assumptions finds what tests can’t, and it matters most when the result is one you’d like to believe.
Limitations
- Python and PyPI only, for local directories and public GitHub repositories.
- Two Django projects evaluated. Rules exist for Django, Pillow, django-allauth, and the Django debug toolbar. Other frameworks have no run-time check yet and will leave more alerts open. Endor Labs, Snyk, and Semgrep already offer commercial reachability analysis, and I haven’t compared vexproof against them on the same projects. What vexproof adds is the verified model step, the run-time evaluation, and evidence attached to every OpenVEX statement.
- Bugs in native code stay open, because there’s no Python function to trace.
- Some leads remain. Across both projects, 9 open alerts had a picked function that ran under the workload, so the search has paths still to find. Four reachable paths weren’t exercised, so the run couldn’t confirm them.
- The hosted Meta Model API is untested. I couldn’t complete its billing setup during the event, so every result uses Muse Spark Open through Ollama.
What I’d do next
- Close the wrong-pick gap. When a fix also changes a loaded module that the model didn’t pick, the alert should stay open. It didn’t affect these results, but a verdict this important shouldn’t rest on a model’s choice alone.
- Measure against an independent benchmark. The largest gap in my evidence is that I picked the projects and wrote the workloads. Vulnerabilities with known exploits in projects I didn’t choose would be a much harder test.
- Reduce the open rate safely. Following up the 9 known leads and adding rules for Flask and FastAPI would have the most impact, with each new rule checked against a running project.
- Label operator-only paths. A path that starts in
manage.pyneeds operator access, and that should lower its priority. - Prepare it for production use. Running this as a shared service would need authentication, tenant isolation for scanned repositories, and spending limits, plus monitoring for how often the model’s answers get rejected.
vexproof pairs with my governed dependency pipeline: that project decides which packages get in, and vexproof tells you which known vulnerabilities matter in the packages you already have.
Patterns you can reuse
None of these depend on security or on a particular model. They apply to any feature where an LLM’s answer drives a decision:
- Give the model a closed choice. Let code find the candidates, so the model selects from real options instead of naming its own.
- Require a checkable citation. A quote that must appear in the source is cheap to verify and stops fabricated reasoning from passing.
- Know what your verifier can’t see. Checking that an answer is grounded doesn’t show it’s correct, so you still need to score correctness separately.
- Write your labels before the first model run. If you change one later, check the facts without the model and make the change in its own commit.
- Find an independent check. For vexproof, that’s the running project. Ask what could prove your system wrong without relying on your own assumptions.
- Report what the system declines to answer. An abstention rate tells a reader more than an accuracy figure on its own.
- Record every answer. Key each one by model, prompt version, and inputs, so that you can reproduce results and re-run evaluations at no cost.
Try it yourself
- Source code: vexproof on GitHub
- Demo: the 3-minute video
- Decisions: 12 architecture decision records
The recorded scan runs with no API key, no model, and no Syft:
git clone https://github.com/henrymbuguak/vexproof.git && cd vexproof
uv sync --locked --all-groups
uv run vexproof serveOpen http://127.0.0.1:8000 and choose Demo result.
I’m a software engineer in Nairobi, Kenya, most recently at Microsoft working on Microsoft Entra ID. I’m looking for AI security, agent identity, and IAM engineering roles, including developer advocacy in those areas. I’m most interested in systems where model output has to be checked before anyone relies on it. Get in touch.


