Series Auditing the algorithm. Part 6. Due process in automated government decisions: notice, appeals, human review, and what public agencies can learn from Canada
The human review protocol section names the real problem precisely: nominal human presence is not meaningful review. A reviewer who can see the output but not the reasoning trace isn't oversight — they're a liability shield.
What you've mapped is the institutional layer. There's a parallel problem one level down, at the model level, that your three artifacts assume has already been solved: the system has to be capable of explaining itself before any notice, pathway, or review protocol can function. If the model routes a high-consequence query to a cheap output path and never flags it, the human reviewer has nothing to work with. The audit trail is already broken before the institution touches it.
The model-level equivalent of your three artifacts: a routing architecture that treats irreversible-consequence domains as hard triggers (not soft signals), an output constraint that prohibits inferred-safety language absent grounded evidence, and a logged audit record that distinguishes forced escalations from standard processing. Without those, the due process stack you've described is sitting on a foundation that can silently fail.
Good series. The Robodebt anchor is the right one.
This is exactly the layer I left implicit — and you’re right that it needs to be explicit.
The three artifacts assume the model can surface enough of its reasoning to make notice and review possible. But if the routing architecture silently degrades high-consequence queries, or if the output path never flags irreversibility, the institutional stack is working with a broken input. The human reviewer isn’t reviewing a decision. They’re reviewing a sanitized summary of one.
Your framing of routing as a hard trigger rather than a soft signal is the right way to put it. Irreversibility shouldn’t be a weight in a scoring function — it should be a gate. And the distinction between forced escalations and standard processing in the audit log is something I hadn’t named precisely but belongs in the foundation of what I was describing.
What this points to is a sequencing problem that governance frameworks rarely acknowledge: you can’t bolt due process onto a model that wasn’t designed to surface the information due process requires. The institutional layer and the model layer have to be co-designed, or the stack is architecturally incomplete from day one.
None of this is written in stone. The point is to keep testing what actually works and to build the best possible version of these safeguards. And this is not only about AI: these are governance problems for any system that can affect rights. AI just amplifies them.
That’s probably the missing piece between Part 2 and whatever comes next in this series. Thank you for naming it so precisely.
Marcela — the three-artifact framework turns due process from a legal principle into an engineering checklist, and that's exactly what's been missing from this conversation. Most public-sector AI debate stays at the level of "we need accountability" without specifying what accountability looks like as a workflow. You specified it.
The human review protocol section is the sharpest part. "Nominal human presence is not meaningful review" — that distinction is doing enormous work. A person who can see the output but not the reasoning, who can approve but not override without institutional friction, who exists in the loop on paper but not in practice — that person isn't oversight. They're a liability shield wearing a name badge.
I build custom AI agents for private-sector businesses and the same principles apply at a smaller scale. Every agent I build has scoped permissions, an escalation protocol, and a kill switch — because an agent that can act but can't be reviewed, paused, or reversed isn't a tool. It's a liability. The difference between public and private is the stakes, not the architecture. Your three artifacts could be a universal design standard, not just a government one.
The Robodebt anchor is the right one. Not hypothetical harm — documented harm. Decisions at scale with no explanation, no meaningful appeal, no human who could reverse them in time. That's what happens when the system is designed for throughput instead of contestability. Glad someone's writing the blueprint for the alternative.
The human review protocol section names the real problem precisely: nominal human presence is not meaningful review. A reviewer who can see the output but not the reasoning trace isn't oversight — they're a liability shield.
What you've mapped is the institutional layer. There's a parallel problem one level down, at the model level, that your three artifacts assume has already been solved: the system has to be capable of explaining itself before any notice, pathway, or review protocol can function. If the model routes a high-consequence query to a cheap output path and never flags it, the human reviewer has nothing to work with. The audit trail is already broken before the institution touches it.
The model-level equivalent of your three artifacts: a routing architecture that treats irreversible-consequence domains as hard triggers (not soft signals), an output constraint that prohibits inferred-safety language absent grounded evidence, and a logged audit record that distinguishes forced escalations from standard processing. Without those, the due process stack you've described is sitting on a foundation that can silently fail.
Good series. The Robodebt anchor is the right one.
This is exactly the layer I left implicit — and you’re right that it needs to be explicit.
The three artifacts assume the model can surface enough of its reasoning to make notice and review possible. But if the routing architecture silently degrades high-consequence queries, or if the output path never flags irreversibility, the institutional stack is working with a broken input. The human reviewer isn’t reviewing a decision. They’re reviewing a sanitized summary of one.
Your framing of routing as a hard trigger rather than a soft signal is the right way to put it. Irreversibility shouldn’t be a weight in a scoring function — it should be a gate. And the distinction between forced escalations and standard processing in the audit log is something I hadn’t named precisely but belongs in the foundation of what I was describing.
What this points to is a sequencing problem that governance frameworks rarely acknowledge: you can’t bolt due process onto a model that wasn’t designed to surface the information due process requires. The institutional layer and the model layer have to be co-designed, or the stack is architecturally incomplete from day one.
None of this is written in stone. The point is to keep testing what actually works and to build the best possible version of these safeguards. And this is not only about AI: these are governance problems for any system that can affect rights. AI just amplifies them.
That’s probably the missing piece between Part 2 and whatever comes next in this series. Thank you for naming it so precisely.
Marcela — the three-artifact framework turns due process from a legal principle into an engineering checklist, and that's exactly what's been missing from this conversation. Most public-sector AI debate stays at the level of "we need accountability" without specifying what accountability looks like as a workflow. You specified it.
The human review protocol section is the sharpest part. "Nominal human presence is not meaningful review" — that distinction is doing enormous work. A person who can see the output but not the reasoning, who can approve but not override without institutional friction, who exists in the loop on paper but not in practice — that person isn't oversight. They're a liability shield wearing a name badge.
I build custom AI agents for private-sector businesses and the same principles apply at a smaller scale. Every agent I build has scoped permissions, an escalation protocol, and a kill switch — because an agent that can act but can't be reviewed, paused, or reversed isn't a tool. It's a liability. The difference between public and private is the stakes, not the architecture. Your three artifacts could be a universal design standard, not just a government one.
The Robodebt anchor is the right one. Not hypothetical harm — documented harm. Decisions at scale with no explanation, no meaningful appeal, no human who could reverse them in time. That's what happens when the system is designed for throughput instead of contestability. Glad someone's writing the blueprint for the alternative.
What stood out to me is how many systems treat “human oversight” as psychological reassurance rather than actual institutional power.
A human reviewer who cannot meaningfully challenge the system is often there to absorb blame, not protect people.