Study
State of quality of vibe-coded apps
What an independent AI test finds on web apps whose defects are known in advance — measured every night, published as aggregates.
Data being collected
Nightly measurement has just started. The first results will appear here after the first measured night; nothing is shown until something is measured.
Methodology
- Sample: our 2 demo apps (Cabinet Azurine, Pennylune), built by Avalyz, with 10 deliberately placed defects described before any test.
- These are not real customer apps: the figures measure the tool on a known benchmark; they do not describe any real app or any app builder.
- Every night, the engine used for customers tests both apps in read-only mode. A finding counts as a defect found when it targets the right page and contains the keywords described for that defect (fixed rule, no AI). Findings with no matching defect are reviewed by a human; they are never counted as defects found.
- A night when the engine cannot run is recorded as “not measured”, never replaced by a number. Nights not measured: 0.
Measured comparison
Other tools are run by hand on the same two demo apps, from trial accounts, and their results are entered with evidence (screenshot or export). Only measured, dated results are published.
Tools in the protocol (official websites consulted on 2026-10-12):
- VibeEval — https://vibe-eval.com
- Vibe App Scanner — https://vibeappscanner.com
- TestSprite — https://www.testsprite.com
No measured result for another tool yet.