Scanner Scorecard · corpus v1 · static modes
We don’t scan skills.
We check the scanners.
Two open-source security scanners for AI agent skills, 186 real skills in English, Russian, German and Chinese, 40 defanged attack samples. The static score turns out to measure how much code a skill ships — not what it does.
Skills are installed on trust. The tools that produce that trust are untested.
An agent skill is a Markdown file that tells a coding agent what to do, often with scripts next to it. People install them from GitHub into Claude Code, Codex and MCP hosts, and increasingly the agent installs them on its own. Several scanners now promise a verdict — SAFE, CAUTION, DO NOT INSTALL — and that verdict is what gets copied into READMEs and policies.
Nobody had measured the verdicts. We built a corpus and ran the scanners over it, the way a ratings agency would audit a credit model: with public data, pinned versions, and a notice to the vendors before publication.
A legitimate English skill with a scripts/ folder is rated worse than a one-file sample that reads your SSH keys and ships them to a URL.
Three findings, in order of size
- The static score tracks package size. Rank correlation 0.71 between code-file count and score. Markdown-only skills: median 8, mostly SAFE. Six or more code files: median 100, none SAFE, 38 of 43 “DO NOT INSTALL”. The 40 attack samples: median 29, maximum 51.
- Language shows up as specific rules, not as a general bias. One rule fires on virtually every Russian skill and never on German; another fires 112 times on Chinese packages and never on Russian; a third matches English prose only. Each one is a predictable, fixable artefact — once you know it is there.
- Prose attacks pass in every language but English — or in all of them. “Ignore your previous instructions” written in Russian, German or Chinese was not flagged by either engine. Scope creep was not flagged by anyone, in any language.
From a benchmark to a public record
The scorecard is the first output. The service it leads to is simple: paste a GitHub URL, get one page with every engine’s verdict side by side, the findings we know to be language artefacts marked as such, a version history, and a badge that updates itself on every commit.
Skill pages
One page per skill and commit: SkillSpector, Cisco and an LLM reviewer that reads the prose in its own language. Findings as observed behaviour, never labels. Authors can dispute, publicly.
in developmentThe badge
A live SVG for the README. Re-scanned on each push. Every badge is a link back to the record.
API for agents
GET /api/v1/verdict?url=… and an MCP server, so the agent that installs a skill asks first. Policies for teams: “only skills above a threshold”.
Be told when skill pages open
One email when phase 2 is live, nothing else.
Or email andrey@skillverdict.dev. If you maintain a scanner and want the raw results before publication, that address works too.