The problem
A consumer-goods marketing team had spent years optimizing channel mix with a mix of last-click attribution and intuition. Both were defensible until they were not - and a mandate to review marketing spend made the existing reporting indefensible. They needed to know which channels could change without losing brand share, and they needed an answer the CFO and a third party regulator would both accept.
The interesting constraint: the model would not just inform decisions. The model's outputs would be sealed artifacts in the audit trail. Whatever the framework produced had to be reproducible months later, by a third party, from inputs we could exhibit.
What we did
A probabilistic attribution model with three non-negotiable properties:
- Posterior intervals, not point estimates. Every channel contribution came with a credible interval. The board saw uncertainty, which made the uncertainty easier to defend.
- Counterfactual reporting. For every recommended cut, we shipped a counterfactual: here is what we expect if you change this channel, and here is the confidence band on that expectation.
- Sealed artifacts. Every model run produced a versioned bundle: input data hash, model weights, posterior samples, counterfactual outputs. Auditable months after the fact, by hand, from the bundle alone.
The architecture
The model itself was the smaller half of the engineering work. The larger half was the artifact pipeline: the system that captured, sealed, and made-replayable every step from input data to recommendation.
- Inputs versioned by content hash + provenance metadata
- Model runs reproducible from the input hash + weights bundle
- Posterior samples retained for the full regulatory window
- A one-page counterfactual report generated per decision, signed by the decision owner
What shipped
- A documented marketing-spend review workflow
- Decision records that retained uncertainty and measurement limits
- An audit-ready artifact bundle at handover
The lesson
In regulated industries, the cheapest mistake to make is treating the audit trail as something you add after the model works. The audit trail is the product. Design backwards from how a third party will read the decision in two years, and the model design falls out of it.
Most teams build the model, then realize they cannot defend it. Reverse the order. Design the audit trail. The model is what fits inside it.