That process of planning, building, testing and improving software has a name: the software development lifecycle, or SDLC.
AI needs the same care, but there is more to check than the software itself. Imagine an online store changing the instructions and documents its AI chatbot uses to answer questions about returns. The chatbot might look exactly the same, yet start giving customers different—and possibly wrong—answers.
That change deserves the same basic questions as any app update: What changed? Does it work as intended? Who checked it? And what do we do if something goes wrong?
At Steav, that is the idea behind CID: help teams keep track of changes to their AI systems, test the results, and make informed decisions before customers depend on them.
The familiar lifecycle has more moving parts
Modern software delivery connects an intended change to evidence that it is suitable for release. Requirements become acceptance criteria. Code and dependencies become identifiable artifacts. Tests inform a release decision. Deployment is followed by monitoring, incident response and another iteration.
An AI system needs that same continuity.
| Software-delivery practice | What an LLM system adds |
|---|---|
| Version the implementation | Identify the model, data, prompts, retrieval and runtime settings that shape behavior. |
| Test the change | Evaluate task quality alongside API, integration, security and performance tests. |
| Approve a release | Attach the decision to the candidate and evidence actually reviewed. |
| Deploy and recover | Know which combination is serving and which compatible combination can be restored. |
| Learn from production | Turn observed failures into reviewed regression cases. |
This is an extension of established engineering practice. MLOps already applies CI/CD principles to data validation, model evaluation and training pipelines. NIST’s AI-specific Secure Software Development Framework profile likewise extends secure development practices to models and the systems that use them. Google’s MLOps architecture and NIST SP 800-218A provide useful foundations.
The practical challenge is connecting those controls across the actual release, including changes that never appear in a model-training job.
A passing test needs a defined claim
Software tests have never been perfect proof. They can miss requirements, behave inconsistently or exercise the wrong conditions. An LLM adds a learned component whose useful behavior often cannot be reduced to one exact expected string.
That changes how we design acceptance tests.
A support assistant might need to answer a policy question accurately, cite an accessible source, withhold restricted information and escalate when the evidence is insufficient. Valid answers can differ in wording. A response can also sound convincing while failing every requirement that matters.
Use deterministic checks for things that can be checked directly. Add task-specific evaluations and calibrated human judgment where needed. For an agent, inspect the resulting state: a message saying “the record was updated” is weaker evidence than checking the record. Repeated trials help expose behavior that a single successful attempt can hide. Anthropic’s agent-evaluation guidance describes these distinctions in detail.
Our implication is straightforward: the evaluation itself needs versioning. Cases, rubrics, model-based judges and test environments shape the result. A passing score should tell us which candidate passed which checks under which conditions.
It should not silently become a claim about every future request.
The model is only part of what ships
A useful AI release record needs more than a model name.
Depending on the application, it may need to identify the code and dependencies, exact weights and adapters, prompts, retrieval snapshot, embedding configuration, generation settings, serving runtime, tool permissions and applicable policy. Training changes also bring their data and preparation history.
Not every input changes with every release. What matters is being able to identify the combination that was tested and the combination that is running.
This is a release-design principle, not a claim that every component can already be pinned perfectly. External services may offer limited version control. Live data keeps changing. Some state cannot be rolled back. Those constraints belong in the release decision.
Provenance helps establish identity and history; it does not establish answer quality. SLSA’s provenance model offers a useful software-supply-chain precedent. The AI system still needs behavioral evidence.
The unit of release is the system whose behavior you tested.
Where CID applies the discipline
We are building CID around the handoffs between producing a candidate, assessing it, authorizing its use and operating it.
Its canonical model-development lifecycle has seven stages:
data_prep -> train -> aggregate -> eval -> benchmark -> validate -> deploy
Data preparation establishes the inputs. Training produces a candidate. Aggregation combines contributions where applicable. Evaluation assesses behavior. Benchmarking measures performance. Validation applies configured acceptance checks. Deployment registers the candidate and, when configured, delivers it to a serving target.
Keeping those responsibilities distinct matters. A completed training run is not an evaluation result. A strong benchmark is not release authorization. A registry entry is not evidence that a healthy service is running.
CID implements several controls that make those distinctions operational:
- Evidence-aware evaluation. Its production evaluation mode requires actual execution evidence and retention; structural dry-run results and formula-based estimates cannot stand in for serving-quality measurements. Teams still have to select meaningful cases, thresholds and evaluators. Production policy must be enabled; permissive dry-run behavior is a separate mode.
- Version-scoped approval. Model reviews and quality reports are checked for the version being promoted. With the required PostgreSQL backing, governance state persists and stale approval authority is rejected. Policy also supports explicit, audited administrator exceptions; it should not be described as an unconditional two-person rule.
- Registered artifact and runtime identity. In its supported registered single-file GGUF serving path, CID verifies stored artifact digests at cold start and enforces explicit versioned runtime policy. Changing that policy creates a new version rather than editing the old one in place. This is a scoped identity check, not a guarantee of immutable storage or identical output.
- Recoverable execution and observable delivery. Job checkpoints and execution leases support recovery after interruption. Serving integrations distinguish registration from rollout and target readiness, with bounded rollback on supported failure paths. External side effects and failures between systems still require reconciliation.
These are concrete controls, not a claim that CID makes an entire AI application reproducible in one transaction. A unified record joining model, prompt, retrieval, runtime, evidence and deployed state remains a broader product objective. Each deployment also needs operational verification against its actual infrastructure.
Nor should every application change pretend to be a training run. A prompt edit can reuse the same weights. A retrieval fix may require no aggregation. CID’s seven stages describe its canonical model lifecycle, not a claim that every custom workflow enforces all seven. The applicable release checks must follow the type and risk of the change.
A small settings change is still a release decision
Consider a different example: changing a model’s KV-cache precision and context allocation to reduce resource use.
The weights might stay the same. The serving configuration does not. Whether the trade improves performance without unacceptable regressions is something to measure on the intended workload.
An SDLC-informed process would record the objective, preserve the previous configuration, create an identifiable candidate, run relevant short- and long-context evaluations, and measure performance on the target backend. The review would consider that evidence before authorizing rollout.
The same logic applies to the support assistant’s prompt and knowledge-index update. “No training required” does not mean “no release review required.” It means the evidence should address the components that changed.
Keeping those changes small and attributable also makes failures easier to investigate. If several inputs changed together without a record, even deciding what to compare becomes an experiment.
Close the loop without removing the checks
Production monitoring is part of development, but a healthy endpoint does not establish useful answers. Latency, error rates and input-distribution changes are operational signals. They are not direct measurements of business correctness.
Likewise, a drift alert is a reason to investigate, not automatic permission to retrain. The right intervention might be better retrieval, a corrected prompt, a permission fix or no model change at all.
The loop we want is deliberate: observe a failure, curate a reproducible case, choose an intervention, compare it with the accepted baseline, and pass the appropriate release checks. Sensitive or untrusted production feedback needs review before it becomes training or evaluation data.
Continuous improvement is more useful when we can explain what improved and how we established it.
Make the release explainable
Before putting an AI change into production, a team should be able to answer five questions:
- What changed, including inputs outside the application code?
- What evidence supports this candidate for this use case?
- Who or what authorized the release, under which policy?
- What is actually serving now?
- How will we detect a regression and recover from it?
CID’s role is to help connect execution, policy and evidence around those questions, alongside the source control, identity, ticketing and infrastructure systems a team already uses.
The opportunity is not to remove uncertainty from AI. It is to make decisions about that uncertainty traceable, testable and accountable.
The SDLC still gives us the discipline. An AI release asks us to apply it to more of the system.