Research · 18 August 2026

What we found in 12 AI-generated codebases

We scanned twelve public projects built with Lovable, Bolt, v0 and Cursor — the kind of thing a person ships in a weekend without a developer looking at it. We were checking whether a scan of the deployed website misses things that matter. It does.

Every project whose dependencies could be resolved carried a package with a known published vulnerability. One had a database key with full read-write access sitting in its git history. None of it was visible from outside the site.

How we did it

We searched GitHub for public repositories that identify themselves as built with an AI site builder — the project descriptions say so — and took twelve, weighted toward ones with a backend, because a static page has nowhere to hide a secret. Mostly personal projects with no stars: the profile of someone shipping their own idea rather than a team with a review process.

Each was cloned with its full history and scanned with two open-source tools: gitleaks for credentials, across the working tree and every past commit, and trivy for dependencies with published CVEs. Then we classified the output, which turned out to matter more than collecting it.

Finding 1 — vulnerable dependencies were universal

Eight of the twelve projects had a lockfile, which is what makes it possible to know which versions were actually installed. All eight carried at least one dependency with a published high or critical vulnerability. Across the set: 4 critical and 76 high.

Not obscure packages. Widely used ones, in versions that were fine when the model learned them and have since had a CVE published:

protobufjs        7.4.0    critical    fix available: 7.5.5
websocket-driver  0.7.4    critical    fix available: 0.7.5
seroval           1.5.2    critical    fix available: 1.5.3
@grpc/grpc-js     1.9.15   high        fix available: 1.9.16
path-to-regexp    0.1.12   high        fix available: 0.1.13

The mechanism is simple and has nothing to do with the model being careless. A language model suggests the versions it saw during training. Time passes. Vulnerabilities are found and published against those exact versions. Nobody runs npm audit, because nobody told the person shipping the site that it exists.

Every one of these had a fix already published. That is the part worth sitting with: this is not a hard problem with a research answer pending. It is a one-line upgrade nobody knew to make.

Why a scan of your website cannot find this

A package version is a fact about your lockfile. It is not something your server tells a visitor, and no amount of probing the deployed site reveals it. You can only know by reading the project.

Finding 2 — one project had a database key in its history

One of the twelve had committed a Supabase service_role key to a .env file. That key bypasses row-level security completely: whoever holds it can read, change or delete every row in the database, whatever policies were written to protect it.

Two things make it worse than it sounds.

It is in the git history. It was committed, and it is still retrievable from every clone of that repository — which is what a public repository means. Deleting the file in a later commit does not remove it. The only fix is rotating the key at Supabase.

The file was server-side. A .env read by the backend is never sent to a browser, so scanning the deployed site — ours or anyone's — would have reported that project completely clean.

Finding 3 — the tools raised four alarms and one mattered

This is the part we did not expect, and it changed how we built our own product.

gitleaks flagged four credentials across the set. Three of them were Supabase anon keys — which are designed to sit in a browser. They are published on purpose. Row-level security, not secrecy, is what protects the data behind them. Finding one is not a leak; it is the system working.

The fourth was the service_role key. In shape it is identical — both are JWTs issued by the same project, the same length, the same prefix. The only way to tell them apart is to decode the token and read its role claim:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiYW5vbiIsIC4uLg
                                       └─ decodes to: "role": "anon"          safe in a browser
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoic2VydmljZV9yb2xl
                                       └─ decodes to: "role": "service_role"  full database access

A tool that reports all four equally hands a non-developer four red alerts where one is an emergency and three are correct behaviour. The likely outcome is that they ignore all four — and the next one too.

The same problem runs the other way in the dependency results. One of the four criticals was in vitest, a test runner that never ships to production. Real, worth fixing, and not the same class of problem as a flaw in code you serve to the internet. Presenting them as equals is noise wearing the costume of thoroughness.

Finding 4 — a third had no lockfile at all

Four of the twelve had no package-lock.json, yarn.lock or equivalent. There is no record of which versions were installed, so nothing could be checked against a vulnerability database.

That is not a clean result. It is an unknown one, and we report it as untested rather than as a pass. It also means the build is not reproducible: install today and install in a fortnight and you can get different code, with no record of the difference.

What this does not show

Twelve projects is a small sample chosen deliberately, not a random one. We weighted it toward projects with a backend, so the secret findings are likely over-represented relative to all AI-generated projects — most of which are static pages with nothing to leak.

The dependency result is the one we would expect to hold up, because its cause is structural rather than behavioural: any project pinned to model-suggested versions and left alone will drift into published CVEs on its own, without anyone doing anything wrong.

We also only found credentials matching known vendor formats — an sk_live_, an AKIA…, a decodable JWT. A password in a variable called DB_PASS is not a shape any tool can recognise with confidence, and we did not count guesses.

And none of this says the projects are insecure overall. It says these specific, checkable things were true on the day we looked.

How to check your own, free

Both tools are open source and run locally. Nothing leaves your machine:

# credentials, including everything in your history
brew install gitleaks
gitleaks detect --source . --no-banner

# dependencies with published vulnerabilities
brew install trivy
trivy fs --scanners vuln --severity CRITICAL,HIGH .

If your code is on GitHub, turn on secret scanning, push protection and Dependabot in the repository settings. They are free on public repositories, they are good, and push protection stops the commit before the key ever lands.

If you find a key, the order is: rotate it at the provider (assume it is compromised — automated scanners find credentials in public commits within minutes), set a spend cap if it is a metered key, then check the account's recent activity. Removing the file is housekeeping, not the fix.

Where this leaves our own product

We ran this before building, to find out whether the thing we wanted to build was worth building. The answer changed what we made: we had assumed leaked keys were the headline, and the data said dependencies are the near-certainty and classification is the hard part.

Preflight now runs both halves — the deployed site and, when you upload it, the project and its full history — and reports what each finding actually opens, with the change that fixes it. The scan of your live site is free. The rest is $29, once, with no account.

If you would rather run the tools yourself with the commands above, do that. It is the same underlying detection. What you are paying us for is the second half: which of these matters, and a dated record of it.

Scan your site free →