Field notes: running Cloudflare’s open-source security-audit-skill on a real codebase

What Cloudflare shipped

Cloudflare recently published a coding-agent skill that turns your agent into a security auditor. The repo is security-audit-skill, the license is MIT, and at the time of writing it sits in the top three of GitHub’s daily trending page, pulling in over two thousand stars a day. A skill, in the agent sense, is a playbook your coding agent loads on demand. This one coordinates a swarm of isolated agents through a structured security audit and returns findings you can hand to a human without squinting.

We ran it against a large open-source monorepo this week. These are our field notes: what the skill does, what it demands from your machine, and what came out the other end.

Six phases, one idea: trust nothing you did not verify

The audit runs in six phases. Reconnaissance maps architecture, trust boundaries, and input surfaces into an architecture.md plus a coverage-ledger.json that tracks every unit of the codebase worth auditing. Coverage-led hunting assigns one isolated agent per unit; each hunter records what it checked, and coverage critics hunt for gaps the hunters missed. Every unique candidate then goes to a fresh verifier whose only job is to disprove it. Survivors are written into findings.json as confirmed, needs_validation, or rejected, and a schema validator checks the file. Finally, a new set of agents verifies every record against the source, and reporting derives the human-readable documents from the verified data.

The design principle underneath all of it: the agent that checks a finding is never the agent that found it. That adversarial split is the difference between a confident model and an auditor. A confirmed record needs a complete source trace and a bounded, observed result. Anything less, and the skill files it as needs_validation with the exact unresolved fact and no severity attached.

Runs are also additive. A repeat run reuses the prior ledger to target gaps and revalidates changed source instead of starting from zero, which matters because of the coverage math we will get to.

The catch: it wants a sandbox

The requirements list looks short: a coding agent with tool use and parallel sub-agents, Node.js for the validators, and an OS-enforced sandbox for anything the target code might execute. That third item is the real cost of admission. The sandbox has to kill external networking, run the target in a sanitized, allowlisted environment, cap CPU, file size, and processes, and keep writes confined to scratch paths.

Without those controls, the workflow refuses to execute target code and parks the lead as needs_validation instead. That is the right default, and it is what separates this from a prompt that says "find vulnerabilities". A skill that treats its own execution environment as part of the threat model has read the room.

We ran it on a real codebase

We pointed it at a large open-source monorepo: the kind with Python services, Go control planes, TypeScript SDKs, and daemons that talk to each other over local sockets. Standard profile, scoped run. We audited the security-critical subset rather than the whole tree, and the report says so, in exactly those words, which is the kind of honesty you want in audit tooling.

The execution policy was the strict one: no external network, an empty allowlisted environment, a read-only target, writes limited to scratch, and CPU, file-size, and process caps enforced by the OS. The ledger seeded 59 units. Forty-six agents did the work: four on reconnaissance, the rest in hunting waves, with three coverage critics and eleven verification agents closing the loop. Most of it ran unattended.

The report also discloses its own deviations. Our record-verification phase ran as seven subsystem-grouped verifiers instead of one per record; every record still got checked by a fresh agent that had not hunted it, and the deviation is stated in the run metadata. Audit tooling that audits itself out loud is rarer than it should be.

What came out

After root-cause consolidation the audit returned 46 records: ten confirmed findings HIGH · 1MEDIUM · 4LOW · 5 and 36 leads held at needs_validation, each with the exact fact a human would need to resolve. No severity was asserted anywhere the evidence did not support it, which is why the confirmed list is short and the leads list is long.

We are not walking through the specific findings here, because the target project gets to fix them first. As classes, they will look familiar to anyone who has audited agent infrastructure: a local server handing data to any web page through wildcard CORS, credentials riding on cleartext transports, credentials embedded unencoded in share URLs, command-and-response demux races in client SDKs, and a few local file-permission issues. Individually unglamorous. In combination, exactly what ships when the people building agent frameworks are racing the market.

When this skill earns its keep

Run it when you are about to depend on, invest in, or ship something large and unfamiliar: an open-source framework you are adopting, a target in diligence, your own agent platform before your customers find its issues for you. The coverage ledger alone justifies the run. It maps trust boundaries and records what was checked, so the next run starts from evidence instead of zero.

Run it with fewer expectations than the trending page implies. The README states the coverage math itself: in their test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total. Plan on more than one run. A needs_validation record is a question, not a vulnerability, and a confirmed record still needs a human who knows the deployment to judge impact. An audit swarm is not a penetration test: it will not probe your live systems, and it should not.