End-to-end fixture flow verified in CI

Your CI caught the regression. ModelGuard found why—and returned a validated fix.

A DataHub-grounded production ML agent that detects model regressions, traverses lineage, ranks evidence-backed root causes, generates a constrained repair, validates it in isolation, and writes the resolved incident back without duplicates.

Run it in one command

0.771 → 0.842F1 restored

H001 · 1.000leading cause

2 linesminimal repair

0 duplicatesrepeat publish

modelguard showcase

regression detected 1 / 6

$ modelguard evaluate
metric: f1_score
baseline: 0.842
candidate: 0.771
status: failed
exit_code: 1

DataHub MCP + SDK Deterministic gates Evidence registry Isolated validation Dry-run by default Apache-2.0

Six guarded phases

From silent regression to inherited operational knowledge.

Each phase has a typed artefact, explicit exit status, and a safety boundary. The reasoning layer cannot skip deterministic evidence or validation.

Detect

Fail the metric gate when F1 falls beyond policy.

0.071 regression

Context

Resolve schema, ownership, quality signals, lineage and deployments through DataHub.

3 upstream · 1 downstream

Diagnose

Generate competing hypotheses and rank them with supporting and counter-evidence.

H001 · high confidence

Repair

Generate only the smallest supported change inside an approved file boundary.

1 file · 2 lines

Validate

Apply the patch in a temporary workspace, compile, test and rerun the model gate.

3 tests · F1 0.842

Publish

Post one review-ready PR comment and one resolved DataHub incident lifecycle.

repeat action: noop

Why DataHub matters

The agent sees the whole ML path, not just the changed line.

Architecture showing raw customers flowing through customer features and training data to the churn model and production deployment, with ModelGuard using DataHub context throughout.

Lineageraw_customers → customer_features → training_dataset → churn-model-v3 → churn-api-prod

Schemamonthly_spend DOUBLE, account_age_months INTEGER, field-level relevance

Signals37 candidate infinities, 12 stable zero-age source rows, failed F1 quality event

Write-backone resolved incident so the next human or agent inherits the diagnosis and repair evidence

Evidence, not guesses

The winning hypothesis explains both the code change and the observed failure.

1feature_transformation

1.0000

2source_data_quality

0.6025

3schema_contract

0.2775

Eleven evidence records are linked by stable IDs. The direct transformation change also counts as counter-evidence against blaming the unchanged source rows alone.

E001

F1 changed from 0.842 to 0.771.

E002

The PR replaced safe division with direct division.

E006

customer_features is two lineage hops upstream.

E010

Candidate monthly_spend contains 37 infinities; baseline contains none.

E011

The same twelve zero-age source rows existed before the change.

A fix engineers can review

Two lines. One function. Independently validated.

The patch guard rejects protected paths, oversized changes, unsupported file types and unsafe tokens. The proposed change is applied only to a temporary copy, while the original workspace is hashed before and after validation.

Patch guardApproved

Targeted tests3 passed

Invalid values37 → 0

Source changedNo

validated_patch.diff+2 −0

def calculate_monthly_spend(
    total_spend: float,
    account_age_months: int,
) -> float:
+   if account_age_months <= 0:
+       return 0.0
    return total_spend / account_age_months

F10.771→0.842policy passed

Closed-loop operations

Publish once. Repeat safely.

GitHub

A single marked pull-request comment contains the ranked diagnosis, verified metric recovery and exact patch.

first runcreated

repeat runnoop

DataHub

A custom incident is raised and resolved against the affected dataset, preserving the operational history in the context graph.

first runraised_and_resolved

repeat runnoop

Stable delivery delivery-43a1891cc0d4

Hidden markers make publication idempotent across reruns.

RESOLVED

Judge it in under two minutes

One command replays the complete incident.

The showcase uses deterministic fixtures, needs no credentials and writes only to artifacts/. Live DataHub and GitHub modes remain explicit opt-ins.

python scripts/run_showcase.py

Judging guide Testing instructions Sample outputs

Submission scorecard

Meaningful DataHub usageRead + write-back

Technical execution6 gated phases

OriginalityDiagnose → repair → validate

Real-world usefulnessProduction ML CI

Submission qualityHosted + reproducible

Designed to earn trust

The agent is powerful where evidence is strong—and deliberately limited everywhere else.

Abstention

Weak or ambiguous evidence exits without a leading diagnosis.

Patch boundaries

Only a cited file, approved strategy and minimal line budget are allowed.

Isolation

Repairs run in a temporary workspace with no shell execution.

Human control

ModelGuard never applies or merges its own patch.

Dry-run default

External writes require explicit --apply and scoped credentials.

Audit trail

JSON, Markdown, diff and incident artefacts remain inspectable.

ModelGuard

Production ML protection, grounded in the context graph.

Detect the regression. Trace the cause. Validate the fix. Preserve the knowledge.

Explore the code Inspect outputs