The State of the Evidence

What is being counted

Each published entry gets one evidence grade: the strongest kind of source that supports its core claims — not a grade per figure, and not a judgement of whether the project was any good. A well-run project whose results only its operator has published is official. A failure documented by a national audit office is independent. The grade describes the state of the record, not the state of the city.

The grades are assigned under a written protocol during the source check, and they are AI-assigned like the rest of the atlas — the entries are AI-researched summaries of the sources each page lists, produced under a documented rule set rather than individually re-checked by a human. That is worth stating plainly on a page about evidence quality: this is a consistent reading of the record by one method, not an audit by a third party. The underlying data is open, so the reading can be checked.

🔬 Peer-reviewed

Impact confirmed by an independent peer-reviewed or academic evaluation.

📊 Independent

Impact confirmed by an independent non-academic evaluation (audit, EU/government review, NGO).

🏛️ Official

Impact figures self-reported by the operator or city (official report or press release).

📰 Media

Impact reported by the press only; no primary evaluation located.

• No measured figure

No independently verifiable impact figure located; the entry rests on qualitative description.

1 🏛️ Two-thirds of the record is the operator's own word

Across all 651 published entries, 68.0% rest on figures published by the body that runs the project — the city, the utility, the transport authority, the vendor. Peer-reviewed evaluation exists for 8.4%. Add every independent audit, regulator's review and NGO evaluation to the academic work and the share of the atlas that anyone outside the project has examined is 20.1%.

This is not an accusation of dishonesty. Operators publish the numbers they have, and often they are the only body positioned to collect them. It does mean that when a city compares itself against what other cities report, it is mostly comparing press releases.

2 📊 Where a result is claimed, it is usually the seller's number

375 entries state a concrete outcome figure — a percentage saved, a journey time cut, a leak rate reduced. Those are the numbers that travel: into slide decks, procurement documents and news copy. 67.5% of them come from the operator. 11.2% have been through peer review.

The atlas labels this on every project page rather than smoothing it away, which is why the count is possible at all. A figure with an operator's name on it is reported as an operator's figure, never as a measured fact.

3 🔍 The recorded failure rate depends on who was allowed to measure

Sort the atlas by who examined a project and the failure rate moves sharply. Among entries whose best source is the operator, 5.4% are recorded as failed or scaled back. Among entries examined by an independent auditor, regulator or NGO, 21.1% — roughly four times as many. Where the press is the only source that looked, 23.1%.

Read the peer-reviewed row before drawing the obvious conclusion: at 9.1% it sits below the operator-reported rate, which is the opposite of the pattern. That is a selection effect, and this corpus can show it rather than just claim it — the peer-reviewed entries have been running for a median of 11 years against 8 for the operator-reported ones, and 90.9% of them are still live. Academics study systems that lasted long enough to be worth a multi-year study and a journal cycle; the pilot that folded after eighteen months never becomes a paper. The academic subset is not a random sample of the atlas, it is a sample of survivors, and at 55 entries a handful of cases moves it. So the claim the table supports is narrower than "scrutiny finds failure": among the groups that produced a figure at all, the operator-reported share of failure is the lowest, and independent and press examination both land several times higher. Whether self-reporting omits shortfalls, or outside evaluations simply get commissioned once a project is contested, this data cannot say. Both are ordinary. Both are reasons to grade the source.

4 🕒 The newest projects are the least examined

Evaluation lags deployment, and the gap is wide. Of the entries that started before 2010, 26.4% have been examined by someone at arm's length. For projects launched since 2020, 13.7%.

This is the least surprising finding here and possibly the most consequential. A city choosing technology in 2026 is choosing from the cohort with the thinnest evidence base, and the systems with a decade of independent study behind them are the ones now being replaced. The evidence arrives, on average, after the procurement decision.

5 💸 Four out of five entries have no public price

A cost figure could be traced for 19.2% of published entries — 125 of 651. The rest are public infrastructure whose price is not in any source we could find. Where money is visible, it is visible unevenly: 40.4% of public-private partnerships carry a cost figure, against 6.9% of privately run systems.

The pattern is not that PPPs are more open by disposition; it is that a contract award has to be published and an internal budget line does not. Publication follows procurement law, not policy on transparency.

6 🗂️ The evidence is thinnest where the spending is heaviest

Independent examination is not spread evenly across the field. Mobility is the largest category in the atlas at 252 entries and one of the least independently examined; energy is the thinnest of all, with 1.7% peer-reviewed.

Environmental projects score best, which is less a compliment to city departments than a side effect of academia: air quality, heat and flooding are established research subjects with people already measuring them for other reasons. Where no research community happens to be watching, nobody is.

What this report is not

• Not a sample of the world's smart-city projects. This atlas is curated. Entries are selected for being documentable, which biases the corpus toward projects that left a public trail — and away from the ones that quietly never happened. Every share here describes this record, not the field.
• Not a quality ranking. The evidence grade says who examined a project, not how well it worked. Some of the best entries in the atlas are graded official because nobody outside the operator has ever looked.
• Failure counts are a floor. An entry is only recorded as failed or scaled back when a named source says so. Quiet abandonment rarely produces one, so the true share is higher than 8.9% — by how much, we cannot say.
• "Ongoing" is a default, not a verdict. 69.7% of entries are recorded as ongoing, which means running with no clear outcome yet. It should not be read as success and does not belong in a success rate.
• The grader is not independent of the graded. The same AI-assisted process that wrote the entries assigned the grades. That is a real limitation, stated here rather than in a footnote — the defence is that the rule set is written down, the sources are on every page, and the data is open enough for anyone to grade it differently.

Check this yourself

Nothing on this page is a private calculation. The whole corpus is published as JSON under CC BY 4.0, and every figure above is a count over two fields — evidence and outcome — plus cost, cat, fundingModel and yearStart. Download the data, group by the field, and you will get these numbers or find a mistake worth telling us about.

🔓 Open data — download the corpus →