// AI RELIABILITY · FREE FRAMEWORK

The AI Observability Maturity Model

A free one-page framework for finding out where your AI system actually sits. From no visibility at all to a closed loop that catches quality drift before a customer does. Uptime and correctness are different metrics. Most teams' dashboards only track the first one.

// AI RELIABILITY · OBSERVABILITY FRAMEWORK

The AI Observability Maturity Model

Five levels, from no visibility into what your AI actually said, to a closed loop that catches quality drift before a customer does. Find your level below.

0 / 4Blind

AI features run in production with no structured record of what your AI actually said — only generic application logs that were never built for this.

What's in placeStandard app logs: request timing, HTTP status, error stack traces.
The gapNo record of the actual input/output pair. A bad answer leaves no trace to investigate.
1 / 4Logged, Not Watched

Full input/output pairs are captured somewhere, but nobody, and nothing automated, is actually reviewing them.

What's in placePrompts and responses are logged and retrievable after the fact, if someone thinks to look.
The gapLogging without review is an archive, not observability. Drift is invisible until a user reports it.
2 / 4Uptime Confused With Correctness

Real dashboards exist. Latency, error rate, cost, throughput. The team believes this counts as AI observability. It doesn't.

What's in placeTraditional APM fully wired up: alerts on errors, latency percentiles, uptime SLAs tracked.
The gapA 200 response and a correct response are different things. Nothing here can tell them apart.
3 / 4Manual Spot-Checks

A human reviews a sample of outputs on some cadence. It catches real problems — but only the ones that happen to land in the sample.

What's in placeScheduled human review, sampled transcripts, a real (if informal) quality bar being applied.
The gapDoesn't scale with volume, and the time between reviews is a fully unmonitored blind spot.
4 / 4Closed-Loop Semantic Monitoring

Every response, or a statistically real sample is scored automatically for groundedness and safety. Quality drift alerts the same way an error-rate spike would, and a bad output traces back to the exact prompt and model version that produced it.

THIS IS THE TARGETThe gap between 'shipped' and 'shipped correctly' closes to near zero — and closes automatically, not by luck.
Want to know exactly where your own system sits? The 50-Point AI Reliability Audit turns this model into 50 concrete, checkable items, or see Sentinel, a free reference implementation of Level 4.
// the five levels

Find your level in under a minute.

Each level maps what's actually in place against the specific gap it leaves. So this isn't just a framework to admire, it's a place to start.

0Blind — no structured record of what your AI actually said
1Logged, Not Watched — captured, but nobody's reviewing it
2Uptime Confused With Correctness — APM everywhere, quality nowhere
3Manual Spot-Checks — real review, but it doesn't scale
4Closed-Loop Semantic Monitoring — the target state
Michael Legemah

Michael Legemah is a Principal AI Engineer who has spent over a decade building production systems for AWS, the U.S. Army, and U.S. Space Force. The last several years focused specifically on agentic AI, RAG pipelines, and the evaluation infrastructure that keeps them honest. This framework comes from the same recurring gap he's diagnosed across real production systems: traditional APM tells you the request succeeded, never whether the answer was right.

One page. Five levels. Find out where you actually stand.