The Skill Is the Policy
A call for DevOps for legal
A few years ago, I sat in a conference room at a large technology company working through a privacy risk audit. Forty items. Each one manually scored by a room full of intelligent, well-intentioned people. We debated, we calibrated, we reached consensus. At the end, we had a number — something like 3.3 out of 5. Not 3.2. Not 3.4. We did this every two quarters, like clockwork.
That is the model most legal teams still run for AI governance. Pre-launch review. Score on a checklist. Ship. Forget. Surface the system again only when something breaks or a regulator asks.
And that is just for the AI governance teams advise their clients on. The AI inside the legal team — what lawyers are doing with AI in their own workflows — typically has no governance model at all. A friend of mine at an elite litigation firm told me their firm’s AI training is a fifteen-minute race-through that is essentially: exercise good judgment, check outputs for hallucination. Then everyone is left to their own devices. The firm does serious work. The training is not.
Checklist governance on the outside, trust-based guidance on the inside. Both were built for systems that did not move after they shipped. The exoskeleton is not one of those systems. As an agentic AI system reliant on non-deterministic output, it is not a product you stamp approve once and walk away from. It evolves just like a live codebase. The governance needs to evolve too. And most of the answers are already sitting in the engineering org’s DevOps practice. Legal just hasn’t picked them up.
The previous three pieces in this series — If a Tsunami Is Coming, Do You Row Faster?, Inside the Exoskeleton, and The Exoskeleton in Motion — described the four-layer system that comprises the exoskeleton. A deflection layer that swats away incomplete or irrelevant requests. A context layer that assembles the information you need. An attention-management layer that decides what work deserves your attention, when. A judgment layer for the work only a lawyer can do. This piece is about what happens after that system goes live.
Agentic AI Leaves Checklist Governance In the Dust
Checklist governance is the form risk management took for an era of systems that did not change after launch. The privacy review caught the issues you could anticipate. The contract terms locked the vendor in. The annual audit confirmed nothing material had shifted. The underlying assumptions:
1. The thing reviewed stays the same until a discrete, specific moment of change.
2. We will know when there is a problem.
3. Humans can review the system manually at a defined cadence.
For deterministic systems, those assumptions held — barely.
For agentic systems, all three break immediately.
The system reviewed at launch is not the system in production six months later. Models drift. Use cases expand beyond original scope. Skills get extended by the third team that adopts them. Dependencies age. The stamp of approval in March is governing a different system in October — and nobody has noticed because nobody is looking.
Failures in agentic systems are rarely loud. They are silent drift. An output that gets a little staler each week. A skill that handles nine cases correctly and degrades on the tenth without anyone noticing the new pattern. A dependency that the person who built it left to manage, until the person who built it left the company. By the time the next quarterly review surfaces a problem, the problem has been compounding for months.
The volume makes it worse. At the scale described in my Tsunami piece, no human is keeping up by reading audit logs. A colleague recycles and customizes a skill for a different team. Internal clients repurpose output in ways no one anticipated at launch.
And governance-as-document — a policy in a Confluence page, a checklist in a SharePoint folder, a risk register updated annually — has no mechanism for catching any of this. It was never designed to. The document is fine. The system has moved.
What the Engineering Org Already Solved
Engineering already has a practice for this, and it is called DevOps. Teams running production systems figured out years ago that governance could not live in a quarterly review, so they built something to live in motion. It is mature. It is battle-tested. And the answers map almost directly onto legal.
Three primitives carry most of the weight.
• Runtime observability. Logs, traces, alerts. The system tells you when it is drifting before a human has to go looking. Outputs are sampled. Patterns are surfaced. The on-call engineer sees the spike before the customer files the ticket.
• Dependency tracking. When you ship a change, the system surfaces what breaks downstream. You do not ship blind. The map of what depends on what is itself a piece of infrastructure, and it is queried automatically every time a change is proposed.
• Incident response. When production is on fire at 2am, there is a playbook. There is an on-call rotation. There is a postmortem after. The engineering org has institutionalized the question of what happens when the system breaks — not as policy theater, but as a rehearsed and ready break-glass process.
The point is not that legal should learn to code. The point is the practice exists and the answers map. Governance has a working analog in the room next door, and most legal teams haven’t walked over.
DevOps for Legal: Governance at Runtime
What does this look like in practice?
Steph is product counsel at a Series B SaaS — same Steph from the Blueprint piece, three-person legal team, every team member operating an exoskeleton. Today she is deploying a new skill. It will run agentic contract review for a category of vendor agreements that used to take her two hours each. She is not writing a policy document about contract review. She is making a series of design decisions that are, themselves, the policy.
She is not writing a policy document about contract review. She is making a series of design decisions that are, themselves, the policy.
The skill is the locus of intentionality. What playbook does it use. What is the order of deference when the playbook conflicts with itself. When does the system stop and loop in a human. What rules does it follow in edge cases. What does the output look like — what gets surfaced, what gets summarized, what gets flagged. Each of those decisions is calibrated to her organization’s culture, risk profile, and what the business can tolerate. They are not generic. They cannot be generic. The act of designing this skill carries the weight that drafting a policy document used to carry — and considerably more, because this policy will run thousands of times.
There is a quality bar before the skill goes live. The team has a test gate (which is itself a skill) — outputs validated against the playbook, against voice and policy expectations, against a small library of known-hard cases. A skill that fails the gate does not deploy. The gate is itself a piece of infrastructure that has to be designed, owned, and updated.
There is an open-source vetting practice. When a skill pulls in a public library, a model card, a reference dataset, somebody owns the question of whether that input is safe to depend on. Engineering has answers here — package vetting, license review, dependency scanning. Legal has the same problem and, in most teams, no equivalent practice yet.
There is runtime supervision. Subagents review outputs as the system runs. Hallucinations get flagged before they reach the human. Drift gets surfaced as a pattern, not as an after-the-fact discovery. The system catches itself when it goes off — not perfectly, but earlier than a human alone could. As part of the design process, Steph is building critical accountability mechanisms that reduce the likelihood of unknown drift and increase institutional trust for the tools the team relies on. These expectations are made clear to the team as a mandatory part of building any skill.
There is automated maintenance. Someone leaves the company — the skills they owned flag automatically and route to a new owner, instead of silently going stale in the corner. Someone wants to ship a change — the dependency map runs and shows what breaks if the change goes through as-is. The Friday-afternoon tasks no one wanted to do (“update template,” “review playbook deviations for this quarter”) are encoded into agents that just run. Steph approves what she wants to approve, looks at what she wants to look at, and the rest happens without her.
And there is an incident response plan. A hallucination reaches an internal client. A skill drifts and gives stale output for two weeks before anyone notices. An agent stops mid-conversation. Most legal teams have escalation paths for client matters and nothing for their own systems. Steph’s team has both, because in the agentic era, the systems are part of the practice.
The critical differences from the checklist model or the trust-based system:
1. Clear expectations for quality before tools ship.
2. Tools designed to check for quality, accuracy, and drift at every instance of runtime.
3. Maintenance and incidents planned for and largely automated.
Most critically, this needs to be a leadership-driven exercise and cannot happen bottoms-up.
This Is Not a Legal Ops Function
I know the overburdened senior lawyer with too much to do and too little time or resources. I have been one. The instinct, reading this, is to file it under “legal ops” — a workflow problem, an operational concern, somebody else’s job. That instinct is wrong, and the cost of being wrong about it is large and will grow larger.
Building and operating governance at runtime is a whole-team capability. The general counsel is the architect. She sets the principles, owns the risk model, makes the design calls about order of deference, what gets escalated, where humans stay in the loop. Those are not implementation choices. They are the firm’s stated theory of how it practices law in the agentic era, encoded into systems that run. This is where the practice of law is going.
Legal operations is the implementor. And, in many teams, the DevOps manager. Operations runs the harness — the test gates, the maintenance triggers, the dependency map, the incident response on-call. Legal operations as a function is going to look more like site reliability engineering than like contract administration, because the work has changed.
Every other team member is, in some sense, an engineer. A privacy lead designs the privacy skill. A commercial lead designs the contract review skill. A litigation lead designs the matter intake skill. They validate outputs in their domain, own runtime quality, escalate the patterns the system surfaces. The capability set is shifting under them whether they pick it up or not.
The objection that lands first — “we don’t have engineers on the legal team” — is exactly the point. Every lawyer is now an engineer. The teams that don’t pick up this capability set are running policy documents against systems that have silently moved three architectures past them. The gap between deployment pace and governance pace, in those teams, is not a workflow problem. It is structural.
The Skill Is the Policy
All of this — the skill design, the test gate, the dependency map, the subagent supervision, the incident response, the choices about deference and escalation and output shape — is the judgment layer. The actual practice of judgment, encoded into systems and operated by humans who designed them with intention.
It takes real thought. It takes real effort. It is what legal practice looks like now.
You cannot outsource it to a tool. An expensive tool will get you the user interface. It will get you the support contract. It will get you the dashboard with the green check marks. It will not get you the thinking — the choice of which playbook, in which order of deference, with which humans looped in for which decisions, calibrated to your culture and your risk profile and what your customers can tolerate. You can’t build it all, but must start building in order to know what’s worth building and what’s worth buying.
An expensive tool will get you the user interface. It will get you the support contract. It will get you the dashboard with the green check marks. It will not get you the thinking.
The skill is the policy. And the work of designing the skill is the work of practicing law in the agentic era.
Natalie Kim
Natalie is the founder of Inflection Advisory and works with organizations on AI strategy and governance. As VP Legal at Omnidian she led a full-arc enterprise AI adoption.
The Judgment Layer publishes on AI governance, board accountability, legal intelligence, and the judgment no algorithm is taking from you.