Confidence Scoring with Bayesian Networks

The running joke inside Brain Co. is that if you want to extend your deck in the Bay Area, just the permits will take two years. When the joke was first told, it was six months, and each retelling adds another couple of months. If you own a home, you probably understand the frustration behind the joke. Homeownership may be part of the American dream, but this particular struggle is familiar the world over.

That struggle makes it a great candidate for applying AI. In fact, using AI in construction permitting is fairly straightforward:

• Ingest the documents of a construction project: architectural blueprints, tables, and forms
• Apply safety and zoning regulations to flag violations in minutes, rather than years
• Builders and architects can use these capabilities during design to identify violations
• Governments can use these capabilities to ensure safe construction and enforce consistency where processes may be undocumented institutional knowledge

So that’s exactly what we have been working on.

On the government side, we are actively developing a permitting system designed to support progressively greater automation as confidence and reliability improve. The problem space is complex, and so we are tackling it with a land-and-expand approach. As summarized in Figure 1, we have followed a staged autonomy pathway, starting with copilot decision support and expanding toward full automation as confidence and reliability mature.

Permitting automation comes with relatively high consequences and needs high accuracy and graceful degradation out of the gate. Unfortunately, Large Language Models (and neural networks more generally) sometimes (frequently) make stuff up. A common approach for these complex problem spaces is building out additional systems to verify candidate solutions in parallel to the solution generation. From Retrieval-Augmented Generation to DeepMind’s original AlphaCode, we see ML systems incorporate things like re-ranking, filtering, or validation. Confidence scoring can be similarly useful in enhancing the performance of systems that generate solutions, where we threshold out low confidence decisions.

Architecture: Three-Tier Confidence Composition

Our confidence scoring architecture operates at three hierarchical levels, each consuming outputs from the previous tier along with original inputs. Figure 2 uses a common building code, where bedrooms need a window, to illustrate this.

The extraction system is the core ML system that turns unstructured blueprints (images) into structured information (JSON) like windows and rooms. As seen in Figure 3, these then form the basis of the probabilities that:

1. The detected area in the file is a window (Value Confidence)
2. The room has a window (Check Confidence)
3. The application passes all regulations (Application Confidence)

Level 1: Value Extraction Confidence

State-of-the-art (SoTA) vision models provide a confidence score that is a combination of existence as well as class confidence. These outputs, illustrated in Figure 4, are used in conjunction with additional signals to estimate confidence that an object is a bedroom or a window:

• Object properties (e.g. thin, low-contrast lines may give less confidence)
• Application file properties (e.g. blurry files might give less confidence)
• LLM as a judge (e.g. ask for additional scoring based on a rubric)

In the long run, we want to lean on any production data flywheels and signals rather than hand-crafted features.

Level 2: Check-Level Confidence

In our copilot application, lower-confidence checks are flagged for human attention, as shown in Figure 5. This level of transparency helps our AI systems build trust with end users by clearly communicating when automated decisions require review.

Value Confidence forms the input basis of the Bayesian system, along with other application properties. Examples include:

• Data distribution (e.g. windows are usually not 100 meters wide)
• Application metadata (professional architect vs. homeowner submission)
• Check-specific Bayesian properties (e.g. a room only needs one window; every additional window detected increases the probability that the room is compliant)
• Some understanding of check thresholds and tolerances (e.g. if a safety railing needs to be at least 1.2 meters tall and it measures 1.40 meters, there is a meaningful buffer, and it is likely compliant) (see our post on blueprint information extraction)

Level 3: Application-Level Confidence

When all confidence scores for all regulation checks are high, we can mark the application as a candidate for straight-through processing, giving our partners the option to automate applications while routing uncertain cases to human reviewers (Figure 6).

These three levels combine into a confidence scoring system where uncertainty is explicitly represented and propagated across the full permitting workflow. Value-level confidence from document extraction feeds check-level confidence for individual regulations, which in turn aggregates into an application-level confidence signal that governs automation decisions.

Figure 7 shows a hierarchical probability graph. Signals from documents and other data sources combine to support transparent, end-to-end automation.

Evals, Evals, Evals

How do we know our confidence scores are any good? Do we generate confidence scores for the confidence scores? A useful system will generate high confidence scores when accuracy is high, and lower confidence scores when accuracy is lower. A natural starting point is a correlation metric such as Pearson or Spearman correlation. However, on certain values or regulations our ML system has very high accuracy across the board and there is little or no accuracy variance. In these cases correlation as a metric breaks down. Figure 8 visualizes this calibration goal, highlighting the difference between weak correlation and a well-aligned confidence–accuracy relationship.

If confidence scores are constrained to map directly to accuracy, then error or loss scoring like Expected Calibration Error or RMSE variants can be used. In practice we relax this constraint and use a precision-blended scoring system to measure correlation when there is variance and accuracy when there is not.

Production Deployment Insights

Part of delivering value on the application level involves targeted inference and engineering strategies beyond just ML evals. To process permits faster we aim to batch and parallelize as much of the computation as possible based on dependency flows. We also add caching for all levels of inference both for value extraction and confidence scoring. ML datasets rarely cover the full long tail of production edge cases at the outset: empty applications, corrupted files, conflicting information, overlapping text. Handling these edge cases thoughtfully is often the difference between a demo and a product people actually trust.

Future Directions

• “Closing the loop” with production activity: using human reviewer signals and comments to inform online learning and improve confidence scoring systems
• “Closing the loop” with confidence scoring: active learning guided by confidence scores to prioritize labeling of edge cases or detect distribution drift.
• Platform expansion: applying custom compliance frameworks to unstructured data across municipalities and regulated verticals. One system to verify everything.

Acknowledgements

The underlying work described here was a technical collaboration between James Huang and Justin Tinker.

References

1. The Frisc. “How Long It Really Takes to Get a Building Permit in San Francisco and Why.” (2023).
2. Geifman, Y., and El-Yaniv, R. “Selective Classification for Deep Neural Networks.” NeurIPS (2017).
3. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. “On Calibration of Modern Neural Networks.” ICML (2017).
4. Li, Y., et al. “Competition-Level Code Generation with AlphaCode.” Science (2022).
5. Brain Co. “Blueprint Information Extraction.” Brain Co. Blog (2025).