Data poisoning is the attack that lives in that gap. An adversary who can't touch your infrastructure, can't crack your auth, and can't exfiltrate your database can still own your model — by getting a few hundred crafted examples into the data it learns from. The model trains cleanly, passes a spot-check, and carries the attacker's payload inside its weights forever.
This post is about what that actually does to a model, what the modern defense stack looks like, and how we built CID to make poisoning somebody else's problem.
What poisoning actually does to a model
"Poisoning" gets used as one word for four different attacks, and they have very different consequences.
Availability attacks are the crude ones: degrade the model overall. Inject enough mislabeled or contradictory examples and accuracy sags across the board. These are the easiest to catch — any honest evaluation will notice — and the least interesting to a serious attacker.
Targeted attacks aim at specific inputs while leaving everything else intact. The model keeps passing every benchmark you throw at it, but misclassifies the one thing the attacker cares about — a particular fraud pattern sails through, a specific face is never recognized.
Backdoor attacks are targeted attacks with a trigger. The model behaves normally until it sees the trigger — a phrase, a pixel pattern, a formatting quirk — and then executes the attacker's chosen behavior. The nasty property: backdoors survive fine-tuning. Research on "sleeper agent" models showed triggers persisting through standard safety training, which means a poisoned base model can carry a live payload through your entire downstream pipeline.
Fine-tuning attacks deserve their own mention because this is where most teams are actually exposed. Nobody pre-trains anymore; everybody fine-tunes. And fine-tuning is more vulnerable to poisoning than pre-training, not less. A LoRA run might see a few thousand examples, each one weighted heavily. Ten crafted examples in a five-thousand-example set can measurably steer the adapter. If your fine-tuning data comes from user feedback, support tickets, or any channel an outsider can write to, that channel is an attack surface.
And if you're running RAG, the analog is knowledge-base injection: a poisoned document in the retrieval corpus doesn't change the weights at all — it just waits to be retrieved, then hijacks the context. Different mechanism, same supply chain.
The model trains cleanly, passes a spot-check, and carries the attacker's payload inside its weights forever.
The federated twist
In federated learning the attack surface moves, and in one way it gets worse. Participants don't submit data — they submit model updates. That means an attacker doesn't need to poison a dataset; they can compute malicious gradients directly, which is strictly more powerful. A single Byzantine client can push the shared model toward a backdoor, and because updates are aggregated blindly, classic data-side filters never get a chance to see the attack.
The good news is that the research community has spent a decade on exactly this problem, and the defenses are real, implemented, and deployable today.
The modern defense stack
The defenses that matter, in the order they should be applied:
Provenance and admission control. Know where every example came from, who uploaded it, and under what authority. Most poisoning succeeds because data admission is anonymous or unaudited. This is unglamorous work and it stops more attacks than any algorithm.
Data-side filtering. Deduplication, outlier and anomaly detection, label-consistency checks. A poisoned example that looks statistically weird compared to its neighbors gets quarantined before it ever reaches a trainer.
Influence bounding. Differential privacy and gradient clipping don't detect poison — they cap how much damage any single contributor can do. An attacker controlling one data source or one federated client gets a bounded slice of influence no matter how crafty their examples are.
Byzantine-robust aggregation. For federated settings: Krum, trimmed mean, coordinate-wise median. These algorithms compare each submitted update against its peers and discard or down-weight the ones that look structurally different, under a formal assumption about how many participants might be malicious.
Evaluation gates as poison detectors. This is the underused one. A curated golden dataset — small, human-verified, never trained on — is a tripwire. A model carrying a backdoor or a targeted bias fails behavioral evals that a clean model passes. If eval gates block promotion, poison gets caught at the door even when every data-side filter missed it.
Attestation and audit. Every stage of training should produce a signed, checkable record: what data, what code, what hyperparameters, what eval results. When something does slip through, attestation is the difference between "we can reconstruct exactly which batch did this" and "we're retraining from scratch and hoping."
None of these is sufficient alone. All of them together make poisoning expensive enough that attackers go find a softer target.
How CID handles it
We built CID around the assumption that training data is hostile until proven otherwise. The defenses above aren't aspirational bullets — they're wired into the pipeline.
Admission starts at data_prep. The first stage of every CID pipeline validates datasets before anything trains: modality-aware validation, an authenticated owner scope required for every data source, and no arbitrary filesystem paths — data enters through tracked upload references only. Anonymous, unaudited data can't get in, which closes the door most poisoning walks through.
Federated rounds are audited, not trusted. CID's FL security layer combines the defenses the literature converged on: Krum and trimmed-mean aggregation to isolate Byzantine updates, enforced secure aggregation so individual contributions are never inspected in the clear, per-client authentication with reputation scoring, and — the part we consider non-negotiable — built-in poisoning simulation. We attack our own aggregation with crafted malicious updates in the red-team suite (Byzantine and Sybil attack tests run in CI) so the defenses are measured against real attacks, not hoped against theoretical ones.
Influence is bounded by construction. Differential privacy budgets apply before aggregation, so no single client or data source can dominate a shared model regardless of what they submit.
Golden sets gate promotion. CID ships canonical golden evaluation sets, and every pipeline runs eval, benchmark, and validate as mandatory stages before deploy. A fine-tune that picked up a backdoor or a targeted bias fails behavioral evaluation against verified cases and simply does not get promoted. The gate is the poison detector of last resort, and it's always on — not a checklist item someone can skip when they're in a hurry.
Everything is attested. Each stage produces a signed record; governance gates require explicit human sign-off for promotion. If a poisoned artifact ever did reach production, the audit trail tells you exactly which data, which run, and which approvals produced it — and the unlearning machinery can remove its influence surgically instead of forcing a full retrain.
And we benchmark the defenses themselves. CID runs comparative poisoning-resistance benchmarks — the same federated task under FedAvg, Krum, trimmed-mean, and differential privacy, side by side — so the robustness numbers on a model's governance card are empirical, not marketing.
The takeaway
Data poisoning is a supply-chain attack, and it should be managed like one: provenance on the way in, bounded influence during training, robust aggregation when many parties contribute, verified evaluation before promotion, and a signed trail behind everything. You wouldn't deploy an unscanned container. Don't deploy an unaudited dataset — and don't deploy a model whose training pipeline couldn't tell you what it ate.