The measurement record
This is the long version of our approach. It provides the detailed test results, withdrawn measurements, extraction incidents, corrective actions, and known limitations. We retain withdrawn results so readers can see what changed and why.
Plexus Impact helps professionals prepare and organize environmental-review work; it does not issue determinations, which remain with the responsible agency, and is not legal advice.
How we test the calculators
Completed reviews give us both the project figures and the results professionals reached from them. We run the same figures through Plexus and compare the two results.
The test has four steps:
Command: python backtest.py --all. Every result lands in a file that carries the cases behind it, so a divergence can be traced to the document that produced it.
We set the test standard before running the test. Water, sewer, and solid waste must be within 15% of the filed results. Trip generation and energy must be within 30%. A calculation without a predefined standard cannot pass.
We investigate and report differences. For example, the traffic test could not apply a site-specific transit adjustment because the test cases lacked mapped sites. The energy test also depends on assumed floor area when a filing provides units but not square feet. These limitations appear with the results below.
The one case where this worked is worth reading, because it also shows the cost. Solid waste sat 42% from the filings, which in rate-book arithmetic usually means the table is being read differently rather than that the arithmetic is wrong. The audit that followed changed the engine, not the bar: it now prices the population it publishes, and it takes a commercial employee count from a filed density rather than inventing one. The re-run came in at 6.9%. Part of that movement is also curation: a case whose program includes hotel rooms, school seats or hospital beds is now rejected rather than compared, because Table 14-1 prices those per bed and per pupil and the engine accepts neither. So the pass is real and it is narrower than it looks, and the set’s silence about institutional programs is on the record rather than hidden in it.
Small samples can give unstable results. Water and sewer passed on four cases but failed when the sample grew to eight. We therefore raised the minimum to eight cases for every rate-based calculation and show the sample size beside every result.
Measured performance
Six real New York projects were run end to end through the live product and graded against what Part 617 requires for that scale and place. Command: python eval_projects.py. Clean on all six, on 2026-08-10 and again on 2026-08-11.
Six projects is a small sample and this is not a hallucination test. It measures classification, study relevance, and whether answers differ between projects. It does not measure whether the tool predicts a determination.
Plexus no longer predicts agency determinations. Until August 28, 2026, the precedent library used outcomes from comparable reviews to estimate a likely determination. Testing showed that the estimate was unreliable, so we removed it from the calculation, the service behind it, and the screen.
What this page published. From 2026-08-11 it said the signal caught 13 of 29 Positive Declarations, 44.8%, and called 30 of 30 Negative Declarations correctly. The method behind it: hold out a real project whose determination is known, rebuild its description with every sentence naming an outcome removed, ask the evidence layer what comparable projects received, and compare.
What was wrong with it. Removing the sentences that name an outcome is not the same as removing the sentences that exist only because of one. A Draft Scope is a consequence of a Positive Declaration and cannot be written before one. Most of these records come from the state’s Environmental Notice Bulletin, which publishes a Positive Declaration bundled with the notice of its draft scope and public scoping session. The scrubber removed the first and left the second:
| Words left in the “scrubbed” description | Present in Positive Declarations (94) | Present in Negative Declarations (1,036) |
|---|---|---|
| “draft scope” | 56.4% | 0.0% |
| “scoping, any mention” | 51.1% | 0.1% |
All 1,130 determined records in the corpus, scrubbed with v1, the scrubber as it ran on 2026-08-11, then searched for the words that only appear after a Positive Declaration has been issued.
On more than half of the Positive Declarations, the description we searched with still carried the fact we were asking it to predict. The signal was not recognising projects that resemble ones agencies found significant. It was recognising the paperwork that follows the finding.
What it scores with that vocabulary removed. Same harness, the same 29 cases, one change:
| The same signal, the same 29 cases | Positive Declarations caught | Negative Declarations correct |
|---|---|---|
| As published, 2026-08-11 | 44.8% (13 of 29) | 100.0% (30 of 30) |
| With the leak closed, 2026-08-26 | 0.0% (0 of 29) | 100.0% (30 of 30) |
Both rows are runs of app/eval_determinations.py, stored as artifacts in the repository and re-derivable with python app/eval_determinations.py --reconstruct. The published run is still reachable, because a withdrawn number that cannot be reproduced is a claim about the past rather than a record of it.
The published result depended on information leakage rather than a valid project signal. The 44.8% figure is withdrawn. It remains in this record so readers can see the original result and the correction.
Our internal testing found the problem. We publish the correction because the purpose of testing is to change product claims when the evidence changes.
The replacement does not work by similarity of description. It combines three readings of a project and scores them together: how the project sits against the Part 617 thresholds, read from the same quote-verified rule catalog the classifier uses; what the record says about the kind of action and the agencies involved, each rate taken from the filed reviews with the project itself left out; and how many of the closest past reviews were Positive Declarations. Command: python scripts/fit_significance.py test.
A second leak was found and closed while building it, on the stored-review side rather than the query side. Each past review was stored as a vector of text that included its own determination and the whole bulletin notice, so the comparison was matching project descriptions against procedural notices. Re-embedded without that text, the same neighbour reading improves, and it is the single largest contributor below.
| Approach | Positive Declarations caught | Negative Declarations correct |
|---|---|---|
| argmax of the neighbour vote (today) What the product does today: take the outcome most common among the closest past reviews. | 2.0% (1 of 50) | 100.0% (527 of 527) |
| neighbour share alone (>= 8%) How many of the closest past reviews were Positive Declarations, across the reviews Plexus has read so far. | 76.0% (38 of 50) | 64.1% (338 of 527) |
| LENGTH-ONLY CONTROL (>= p57) A control that reads nothing but how long the description is. If the model lands here, the model is measuring document length. | 60.0% (30 of 50) | 58.4% (308 of 527) |
| 617.4(b)(5)-(7) size screen alone The Part 617 size thresholds alone, with no reference to past reviews. | 30.0% (15 of 50) | 93.7% (494 of 527) |
| 617.4 size OR category screen alone The size thresholds plus the categorical Type I tests, where the record makes them readable. | 48.0% (24 of 50) | 85.2% (449 of 527) |
| clean-corpus neighbour share (>= 8%) The same share, measured against past reviews with their own stated outcome removed from the text. | 72.0% (36 of 50) | 73.2% (386 of 527) |
| clean share, similarity-weighted (>= 5.3%) | 74.0% (37 of 50) | 73.1% (385 of 527) |
| The model Plexus uses now | 78.0% (39 of 50) | 77.0% (406 of 527) |
577 held-out reviews, 50 of them Positive Declarations, none of which any of this was fitted against. The split is a hash of each review’s identifier rather than a random seed, so a review stays on the same side of the line for the life of the corpus. The model was tuned on the other 553 and this half was read once. Re-derive with python scripts/fit_significance.py test. Neither rate reaches the 80% both have to reach, so the program is not finished and the readout stays off.
The control checks whether the model is only measuring document length. The length-only control reaches 60.0% and 58.4% on the same reviews. The model performs eighteen points better than that control.
The model did not meet the standard. The test requires at least 80% accuracy for both Positive and Negative Declarations. The model reached 78.0% and 77.0%. Plexus therefore removed the prediction feature. Comparable reviews remain available as context, but Plexus does not score them into an expected agency outcome.
Positive and Negative are never pooled into one accuracy figure. Of the 1,130 determined reviews this signal can search, 94 are Positive Declarations, 8.3%. Anything that always guessed Negative would score above 90% pooled while catching no significant action at all, which is the number this harness exists to refuse to print.
What we cannot test yet, said plainly because it is the half most likely to matter. The obvious way to improve this is to match on what is at the site: wetlands, flood exposure, a historic district next door, a critical environmental area. It cannot be tested on this corpus. Not one of the 94 Positive Declarations carries a coordinate precise enough to measure a distance from, so there is no geography to match on, and the records themselves describe an action rather than a place. That hypothesis is untested rather than disproven, and testing it depends on collecting reviews that carry a real address.
Use comparable projects to investigate questions on your own site. Their outcomes provide context, while the responsible lead agency makes the significance determination from the project record and site-specific conditions.
Automated document extraction can introduce errors that look credible. On 2026-08-10, testing found invented content in an extracted record. This section explains the incident and the safeguards added afterward.
A reviewer marking extracted values against their source documents opened the ecology chapter of a Findings Statement for a solar project in the Town of Marilla, Erie County. The extraction reported 39.79 acres of wetlands, described the site as freshet and tidal, and cited a source called “Wetland and Stream Delineation (2022).” None of those appear anywhere in the 29 page document. They sat beside figures that were verbatim correct, which is what makes this failure mode dangerous: the invented content reads exactly like the real content. Three things caused it. The document is fully scanned, with zero characters of embedded text, so the model read page images rather than text. Sampling temperature was left at the API default, the widest setting. And the validator kept the first of its three readings word for word while comparing only numeric fields, so the narrative field where the invention lived was never compared against the other two readings at all. Running that chapter five times in isolation produced five different narratives, two of which reproduced the invented strings exactly. Reading stopped that day and did not resume until two guards were built.
Two safeguards now apply. First, every extracted number and named item must be found in the source document before it is saved. Unsupported statements are dropped. Second, three extraction passes are compared field by field. If they disagree, Plexus withholds the field for human review instead of selecting one answer. Nine documents currently contain withheld fields.
Re-read behind both guards, the incident document came back clean. None of the invented strings reached the database and the correct figures survived. The gate is strict enough to hurt: it also quarantined seven of seven substantive fields on that document, leaving a thin record where three broadly correct readings had disagreed about wording. Loosening it to keep what a majority of readings assert is open work. Separately, the first version of the grounding guard dropped “M3-1 District” from a document that plainly contains M3-1, because the pattern split on the hyphen. That was found by reading the drops and fixed before it ran on the corpus. A guard that silently destroys real evidence is the same class of problem as one that lets invention through.
The audit. The guards protect what is written from now on. Everything stored before them was audited by re-running the grounding check over every extraction already in the database. No model calls, so it is repeatable at no cost. Command: python audit_grounding.py. Measured 2026-08-11:
That 21.9% is an upper bound on suspicious content. It is not a hallucination rate, and reporting it as one would overstate the problem as badly as hiding it would understate it. The dominant pattern is a composite the extractor assembled from a table, such as a noise reading written as “R02 66.5-70.0 dBA” or a total area summed across rows. The document holds the parts and never the assembled string, so the check cannot locate it. That is synthesis, and this method cannot separate synthesis from invention. Only a human mark can, which is what the gold set below is for. The genuinely suspicious residue is smaller: single proper names that should appear verbatim and do not. An equivalent audit on 2026-08-10, over a smaller corpus of 2,478 blocks, put the same figure at 24.2%.
One more result worth stating, because it contradicts an intuition we held. The extractor’s own self-consistency score does not predict invented content. Of 20 sampled low scorers, 58 of 58 numbers were grounded, while the one clearly ungrounded block found in that sample scored a perfect 1.00. A confidence threshold would have discarded good readings and admitted the bad one. That is why the guards check the text rather than the score.
The transportation dataset is the first one measured end to end, and this is its number. On 2026-09-08, at commit b161a66, a stratified sample of 20 documents was drawn from the extracted corpus and every figure stored against them was read back against the page it was taken from. 132 of 188 values were correct on all four of the number, the scenario, the unit and the page: 70%. On a sample of 20 documents the true rate sits somewhere between 58% and 81%, and that interval is computed by resampling documents rather than values, because the values arrive in clusters and a narrower interval would be a claim about a sample size we do not have.
What the number is made of matters more than the number. The figures themselves are string-matched back into their own evidence quote before they are stored, and that mechanism holds: a value on the screen does appear in the review. The failures are in the label the figure hangs from. One direction of a stated inbound-and-outbound pair gets stored as the project total. A figure from a project alternative gets stored as the proposal. One end of a stated range gets stored as the figure. The worst field is trip generation results, where 4 of 18 values were correct, and trip figures are the ones any comparison would be built from.
Three further measurements from the same sample, each with its own denominator. Omission: of the questions a document answers, 27 of 82 were not returned at all. Untraceable claims: 14 of 176 of the stored values could not be traced to any passage in their own document. False positives: of the documents in the sample that perform no transportation analysis, 2 of 10 reported that they did, and both were tables of contents from which a significance finding, a mitigation description and a list of impacted intersections were manufactured out of chapter and table titles. The launch gate set on 2026-09-01 asked for omission under 10%, zero untraceable claims, and no single field worse than 25%. This result meets none of the three. The full method, the sample and every demoted value are in Notes/AUDIT-traffic-accuracy-2026-09.md.
What may and may not be said about this. It may be said that the traffic dataset has a measured accuracy, that the measurement is public and unflattering, and that a figure shown beside a citation does appear on the page cited. It may not be said that traffic figures are verified, validated, checked, or reliable, and no traffic accuracy figure may be quoted without the sample size and the interval attached. The reading was done by a model against each document’s own text and not by a traffic professional, so it measures the extraction and not the traffic.
Outside transportation, finding a claim in the document still does not prove that Plexus interpreted it correctly, and no other domain has been measured this way. Human review is underway: as of 2026-08-11, 47 of 392 fields have been checked. That sample is too small for a reliable accuracy rate, so none is published for those domains yet.
The one number that already exists is worth naming precisely, because it is easy to misread. The scoring harness reads one of the two review sets and currently reports 11 of 11 fields correct with no hallucinations. That is a sample of 11. It also does not include the review set holding the Marilla document, which is where the known hallucination was found. So a clean result from that harness today would be an artifact of which folder it reads, and it is not being offered as evidence of anything.
Reading a source closely enough to compute from it turns up things that reading it for gist does not. Table 6-1a of the CEQR Technical Manual, which sizes how many children a housing project puts into publicly funded early-childhood programs, states its trigger twice and the two statements disagree. The table prints a multiplier of children under six per low or low/moderate income unit, and beside it the minimum number of such units that it says yields twenty children. Multiplying one column by the other does not produce twenty in any borough:
| Borough | Children per unit | Published trigger | Children that yields |
|---|---|---|---|
| Bronx | 0.139 | 141 units | 19.60 |
| Brooklyn | 0.178 | 110 units | 19.58 |
| Manhattan | 0.115 | 170 units | 19.55 |
| Queens | 0.140 | 139 units | 19.46 |
| Staten Island | 0.090 | 217 units | 19.53 |
Every borough lands near 19.5, which says the unit column was computed from a finer multiplier than the three-decimal one printed beside it. The gap is small and it is not harmless: a project sitting on the published unit trigger passes one of the manual's two tests and fails the other, and which test an analyst happened to apply decides whether a detailed early-childhood analysis gets scoped. Plexus screens in on either, and the topic card reports that the two tests disagree rather than resolving a contradiction that is the manual's to resolve.
The finding is a side effect of the method rather than an audit of the manual. Rates are transcribed into a versioned file with the sentence each number came from, and every sentence is string-matched back into the source PDF before it can be cited. Doing that to Chapter 6 meant multiplying the two columns against each other, which is not something a reader has any reason to do.
Library
The page loads these New York library counts from the database. If live counts are unavailable, it shows the latest dated snapshot.
Counted August 19, 2026, the last time these were measured. Plexus keeps reading, so treat these as a minimum.
Where the records come from, counted 2026-08-19, by count of review records:
They cover 81 counties, counted 2026-08-19. 16,607 of the 20,988 records carry coordinates, and each one records how those coordinates were derived: 10,332 to a parcel, 2,584 to a block, 3,691 only to a jurisdiction. That last group is the reason Plexus will not measure a distance from a jurisdiction-level point and present it as a site distance. If the precision is not good enough for the question, it says so instead of answering.
Back
Our approach explains how Plexus works, what it calculates, and which decisions remain with practitioners, specialists, and agencies. Is something here wrong or out of date? Tell us.