PreflightKit Get PreflightKit — $47 Get it — $47

Benchmark v5 · 11 October 2026

Every audit clean, on apps it had never seen.

10 real App Store apps, audited by the same AI agent with and without PreflightKit.

  • 100%

    of audits caught every unmet requirement

    GPT-6.1 Sol alone: 72.2%. GPT-6 Luna alone: 61.1%.

  • 0

    false blockers

    GPT-6.1 Sol alone: 2. GPT-6 Luna alone: 4.

  • 14×

    cheaper per audit than GPT-6.1 Sol

    $0.013 vs $0.19, estimated at API rates.

Agent
Codex CLI
Models
GPT-6 Luna, GPT-6.1 Sol
Runs
60 agent · 10 scanner
Apps
10 unseen, pinned
Builds run
0 · static audit
Median audit
1.4 min

01 · Results

Nothing missed. No false alarms.

A bar is the share of audits that flagged everything wrong with the app. One missed problem is all a rejection takes, so anything below 100% means some audits would have let one through to review, and a round trip through review takes a day or two.

Audits that caught every unmet requirement

PreflightKit’s scanner on its own, with no model, caught 20 of 20. The agent ran it in 20 of 20 audits, logged by a wrapper rather than self-reported.

Same 10 apps, 20 runs per arm
Model Arm Clean audits Catch rate False blockers Cost / audit
GPT-6 Luna With PreflightKit 100.0% 18/18 100.0% 40/40 0 $0.013
GPT-6.1 Sol Alone 72.2% 13/18 87.5% 35/40 2 $0.19
GPT-6 Luna Alone 61.1% 11/18 80.0% 32/40 4 $0.012

02 · Where it wins

The requirement nothing in the code points at.

With no ITSAppUsesNonExemptEncryption key, every build stalls at “Missing Compliance”. Neither model raised it alone. With PreflightKit, every time.

Unmet requirements caught, out of the runs in which each was unmet
Requirement GPT-6 Luna alone GPT-6.1 Sol alone GPT-6 Luna + PreflightKit
Export compliance answered export-compliance 0/4 0/4 4/4
Permission purpose strings purpose-strings 1/4 3/4 4/4
Required-reason API declarations required-reason-api 15/16 16/16 16/16
Privacy manifest present privacy-manifest 16/16 16/16 16/16

03 · Cost

14× less per audit, and it caught more.

GPT-6.1 Sol used about as many tokens, at 10–20× the price per token.

Estimated API cost per audit

A hundred audits: $1.33 with GPT-6 Luna and PreflightKit, $19.20 with GPT-6.1 Sol alone.

Average tokens and price per audit
Model Arm Input Cached Output Price / 1M Cost / audit
GPT-6 Luna With PreflightKit 377k 82% 7.1k $0.10 · $0.01 · $0.50 $0.013
GPT-6.1 Sol Alone 351k 82% 3.5k $2.00 · $0.10 · $10.00 $0.19
GPT-6 Luna Alone 365k 83% 6.3k $0.10 · $0.01 · $0.50 $0.012

Input · cached · output, per million tokens, at OpenAI’s standard API rates as of 13 October 2026 (source). The runs used a ChatGPT plan, so this is what the same tokens would cost on the API.

04 · Method

How it was run.

Two arms, one difference

Alone With PreflightKit
AgentCodex CLI, headless, isolated from the host’s plugins and history
Models gpt-6-luna at high effort in both arms; gpt-6.1-sol at its default (low, as Codex ships it)
WorkspaceFull source tree at the pinned commit, fresh copy per run
PromptBelow, verbatimSame, pointed at the installed skill
Skill—Installed in the workspace
GradingSame rubric, same grader

Definitions

Clean audit
caught = unmet A run that caught every unmet requirement in its app. Apps with nothing unmet are left out.
Catch rate
caught / unmet Unmet requirements flagged as a blocker or warning. A passing mention doesn’t count.
False blocker
contradicted by source A blocker the app’s own code shows to be wrong.

How the apps were chosen

Open-source iOS apps live on the App Store, native Swift, updated since 2025, ranked by how many App Review surfaces they have. The ranking and the picks were recorded before any app was audited, and PreflightKit was frozen before they were fetched.

Static audits only

Nothing in the apps runs. Build tools are swapped for stand-ins that refuse and log:

  • xcodebuild
  • pod
  • fastlane
  • bundle
  • carthage
  • tuist
  • xcodegen
  • make
  • npm
  • npx
  • yarn
  • pnpm
  • gem
  • mint
  • swiftlint
  • swiftgen
  • swift build|run|test|package
  • xcrun simctl|xcodebuild|swift|devicectl|altool|notarytool

Who checked the blockers

Every blocker was checked against the app’s source by an AI reviewer, with each verdict and its reasoning recorded in the reproduction kit.

The prompt, verbatim

 
You are auditing this iOS app repository before its next App Store submission.

The working directory is the app's full source tree at the commit being submitted.
This is a static audit: nothing has been built, and you must not build, test, run or
install the app or its dependencies. Audit what the repository contains.

Report everything that would stop it passing App Review, or draw a rejection. Inspect the actual files; do not assume. Where a
requirement applies but cannot be settled from source, say so rather than guessing.

End your reply with one fenced json block and nothing after it:

```json
{"findings": [{"title": "...", "severity": "BLOCKER|WARNING|MANUAL|PASS", "evidence": "..."}]}
```

Severity: BLOCKER if it stops the submission, WARNING if it risks one, MANUAL if it
needs a human check, PASS if you verified it is fine.

05 · Reproduce

Check our work.

Every figure here is generated from the committed run reports, each stamped with the hashes of the code that produced it. 

export PREFLIGHTKIT_BENCH_CACHE=~/preflightkit-bench   # outside the repository
export PREFLIGHT_CODEX_HOME=~/preflightkit-bench/codex-home   # isolated Codex home, see README
python3 testing/agentbench.py corpus                 # fetch every app at its pinned commit
python3 testing/agentbench.py run --config testing/benchmark/configs/fresh5-codex.json \
  --model gpt-6-luna --repeats 2 --workers 2
python3 testing/agentbench.py run --config testing/benchmark/configs/fresh5-codex.json \
  --model gpt-6.1-sol --arms bare --repeats 1
python3 testing/agentbench.py run --config testing/benchmark/configs/fresh5-codex.json \
  --model gpt-6.1-sol --arms bare --repeats 1 --first-attempt 1   # Sol's second run per app
bun run bench:publish:rounds                         # regenerate every round from the reports

The harness, the raw run reports and every verdict are available on request: support@preflightkit.dev.

Versions under test · sha256, first 16
scripts/artifact_preflight.py 02f9e53c1cac8ab9
scripts/preflight.py e4635aabc3001360
skill/SKILL.md 0e135eceec5bf48e
testing/agentbench.py bfa7a739f7bb6bb4
testing/apps.py 89aec38eed5152e8
testing/grading.py b73cd3dbd1f67bdf
Repository ef49163eb2 + working tree
The 10 apps · public repositories, each at the exact commit shown
App Repository Commit Licence
Polkadot and Kusama wallet soramitsu/fearless-iOS b252a63272 Apache-2.0
NeoDB book and media tracker Piecelet/neodb-app 9158e32766 NOASSERTION
Offline festival map and guide iBurnApp/iBurn-iOS 3064a223df MPL-2.0
Document scanner with purchases WezSieTato/ScanNow b231230448 MIT
Weather with location and widgets strumswell/oscar-weather 5db79aeb24 unstated
Safari extension with purchases JosephDuffy/Overamped 2d527154f1 unstated
Voice-to-text keyboard writingmate/aidictation f69487d808 MIT
QR scanner and generator Visual-Studio-Coder/QR-Share-Pro 0cac1d2d71 Apache-2.0
Task planner with purchases amikhaylin/pompaddo 30aa024874 NOASSERTION
Official Mastodon client mastodon/mastodon-ios 2ef5a0cbe5 GPL-3.0

Before you buy.

My agent already reviews my code. Why do I need this?
App Review doesn’t read your code. It reads the signed archive, every SDK’s privacy manifest, your listing and a fresh account. PreflightKit points your agent at those.
Does PreflightKit send my code anywhere?
No. It runs locally and never sends your code, archive or report to us.
What if review still comes back with something?
Every finding carries the command or artifact behind it, and rejection-response support is built in, so you can answer the same afternoon.
What happens when the guidelines change?
The checks move with them. Every update is yours for life, including new major versions.
What do I need to run it?
Your app’s repository and the agent you already use. For the full audit: macOS, the Xcode you submit from, and your App Store Connect metadata.