Models and prompts that ship through a gate, not a hope.
MLOps engineers who turn notebooks and prompt experiments into governed release pipelines: versioned data, evaluation suites in CI, a registry with a promotion path, and drift monitoring once live.
- MLflow
- Ragas
- PyTorch
- Airflow
- DVC
- KServe
Machine-learning work often fails after the model is built, not before.
The notebook worked once; the training data was a CSV on someone’s laptop; the prompt was tuned by hand until the demo looked right. Six months later nobody can say which version is in production, why accuracy dropped, or whether last week’s change made things better or worse. MLOps is the engineering that stops that story from repeating — the same CI/CD, versioning and observability discipline we apply to ordinary services, extended to data, models and prompts.
Evaluation is the centre of it. For a classical model that means a held-out set and metrics that match the business decision; for an LLM feature it means a frozen set of real, labelled examples and a scorer that runs in CI. Either way the rule is the same: if it is not evaluated, it is not done. Models and prompts inside our own products ship with a baseline and a gate — the tier-prediction and next-best-offer models in Loyalty & Retention carry model versions and evaluation metrics a client team can inspect, and the AI features we build for clients are delivered the same way.
We do not build an MLOps platform for a company with one model. We build the smallest pipeline that makes releases reproducible and evaluations automatic, on tools your team already runs, and grow it when the second and third model arrive.
Six things we ship under the MLOps heading.
Evaluation suites for LLM features
Labelled example sets drawn from your real traffic, scorers for correctness, groundedness, refusal behaviour and format, and a CI job that fails the build when a prompt or model change lowers the score against the stored baseline.
Training and retraining pipelines
Airflow or Kubernetes-native pipelines that pull versioned data, train, evaluate and register a candidate on a schedule or a trigger — reproducible from a commit hash, not from memory.
Experiment tracking and registry
MLflow for parameters, metrics and artefacts on every run, and a registry where a model moves from candidate to staging to production only through a recorded, gated promotion.
Model serving
FastAPI or KServe endpoints with request validation, batching, timeouts, canary releases and rollback — the model is a versioned deployment like any other service on the cluster.
Monitoring and drift detection
Input distribution, score distribution, latency and cost per model version, with alerts when the live data stops looking like the training data — and a retraining trigger that responds.
Data and feature versioning
Datasets, features and labels versioned alongside code with DVC or lakehouse tables, so any prediction can be traced back to the exact data and model that produced it.
When to choose it.
Choose an MLOps investment when a model or an LLM feature makes decisions that matter — pricing, fraud, eligibility, customer-facing answers — and you need to prove what version did what. Choose it when more than one person changes prompts or models, when retraining is a recurring event rather than a one-off, or when a regulator, auditor or your own risk team will ask how a decision was reached. Choose evaluation-first delivery for every LLM feature, whatever its size; the eval set is cheaper than the incident.
Do not choose a full platform for a single batch model that runs monthly and a person reviews — a versioned script, a tracked run and a dashboard are enough. Do not adopt a feature store, a workflow orchestrator and a serving layer at once because a reference architecture shows them; adopt each when the pain it solves has actually appeared. And do not treat evaluation as a launch-week activity — it is a merge gate, or it is theatre.
An evaluation gate that decides whether a prompt change merges.
name: eval
on: { pull_request: { paths: ["prompts/**", "models/**", "evals/**"] } }
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: uv run evals/run.py --suite extraction-v3 --out result.json
# frozen, labelled set of real invoices; baseline stored with the suite
- run: >-
uv run evals/compare.py result.json --baseline evals/baseline.json
--min-accuracy 0.94 --max-cost-delta 0.10 --max-p95-ms 1800
- uses: actions/upload-artifact@v4
with: { name: eval-report, path: result.json }The path from a change to a model in production.
What the model pipelines we hand over look like.
- Reproducible from a commit. Code, data version, parameters and environment are all recorded on every run. Anyone can rebuild the model in production from its registry entry.
- Evaluation as a merge gate. A stored baseline, a scorer that runs in CI, and thresholds agreed with the business owner. A change that regresses does not merge — for prompts and models alike.
- Metrics that match the decision. Precision at the threshold you actually use, cost per false positive, groundedness on the questions customers actually ask — not a generic accuracy number that hides the failure that matters.
- Promotion is recorded. Candidate, staging, production: each move has a who, a when and an evaluation result attached. The registry is the audit trail.
- Deployed like a service. Containers built once, canary rollout, health checks, rollback in one command, and the same on-call runbook format as everything else on the cluster.
- Monitored for the model’s failure modes. Input drift, output drift, per-version latency and cost, refusal and human-review rates — with alerts and an owner, not just a dashboard.
- Cost is a first-class metric. Training spend per run and inference spend per thousand requests are tracked against a budget; an eval that wins on quality and loses on cost is a decision, not an accident.
- Personal data handled as data engineering. Training sets minimised and documented, PII masked in logs and traces, KVKK and GDPR obligations reviewed before the first extract and recorded with the dataset.
What usually ships alongside it.
- An evaluation suite per model or LLM feature: labelled set, scorers, stored baseline and the CI job that enforces it
- A training or tuning pipeline that is reproducible from a commit hash, with experiment tracking on every run
- A model registry with a documented promotion path and the release mechanics — canary, rollback — on Kubernetes
- A monitoring dashboard per model: drift, quality, latency, cost, human-review rate, with alert routing and an owner
- Runbooks for the incidents that actually happen: quality drop, provider outage, cost spike, retraining failure
Frequently asked questions
Neighbouring technologies and the service that uses them.
Need engineers who can make your models releasable?
Tell us what you run today — notebooks, hosted LLMs, a training pipeline — and where it hurts. We will propose a named team or place MLOps specialists into yours.