How accurate is Code AI? We haven’t measured it yet, so here are the questions.

Everyone in this category publishes an accuracy percentage. Almost nobody publishes the questions it was scored on, and a percentage without a question set cannot be checked, reproduced or argued with — which is what makes it useful as marketing and useless as information.

So this page is the other way round. The 45 questions below, the rubric and the grading rules are published before any answers were collected, and they are in our source repository so the history shows they were not edited afterwards. When the results land, every transcript is published — ours included, and the ones we get wrong included.

The method

• 45 questions, fixed in advance, listed in full below.
• Four contestants: BuildSpec Code AI, UpCodes Copilot (Professional trial), GPT-5, Claude. Answers are collected from all four before any are graded, and presented to the grader with the product names removed.
• The grader is a named, independently paid licensed PE or code consultant with no equity in BuildSpec. Not yet engaged — the grader is named here before grading starts, not after.

The rubric

The rules we are holding ourselves to

• The grader is a named, independently paid licensed PE or code consultant with no equity in BuildSpec, and is named on the results page.
• Answers are collected from all four contestants before any are graded, and are presented to the grader with the product names stripped.
• Every transcript is published in full, including ours, including the failures.
• A question with no defensible answer is removed and the removal is published with the reason. Questions are never replaced with easier ones after results are known.
• Two scores are reported: overall, and building-code-only. If BuildSpec loses the building-code-only subset, that is published in the same size type as everything else.
• If the overall result is worse than the competitor on their own subject matter, it is published anyway, with the list of what we are fixing.

The question mix, and the obvious objection

15 building-code questions, 10 state amendment/adoption, 10 permit-cost and 10 state DOT specification.

The objection writes itself: 20 of 45 are on subjects UpCodes does not carry, so of course they score zero there. That is true, it is stated here rather than buried, and it is why two scores are published — overall, and building-code-only. If we lose the building-code-only subset, that number appears in the same size type as everything else. A benchmark that can only produce one answer is not a benchmark.

It is also a real measurement rather than a rigged one: a practitioner asking what a permit costs in Denver gets a wrong or absent answer from a tool that has no permit data, and that is a fact about the tool, not a trick question. Coverage is part of accuracy.

The 45 questions

About the 93% figure

UpCodes markets “93% weighted accuracy, 2× generic AI”. Three factual observations, recorded 27 August 2026, with no adjectives attached:

• The figure comes from a whitepaper UpCodes commissioned.
• As observed on that date it publishes no question set and no transcripts.
• Its question mix excludes permit-cost and state DOT specification questions — the two categories their product does not carry.

We are not offering a competing number until ours has been graded by someone independent, because a number we produced about ourselves would have exactly the problem we have just described. The full comparison is here.