Track C
Assure
PlannedHow do we know each output is good enough, at runtime, before it ships?
This page describes our research agenda. Status labels show what is planned, in progress or complete.
Why it matters
A cheap answer that is wrong is more expensive than a delayed or withheld one.
Orchestration manages where a request runs. Quality orchestration manages whether the result is good enough, as a control loop. Quality is treated as a managed service level.
Directions
- 1.Quality contracts: acceptable quality, error cost, abstention rule.
- 2.Runtime estimation: verifiers, self-consistency, cross-provider agreement, graph consistency.
- 3.Cascades with calibrated methods: cheap model first, escalate below contract.
- 4.Judge reliability.
- 5.Drift monitoring.
- 6.Closed learning loop.
Quality cascade
ConceptualCheap model, quality estimate, then ship, escalate or abstain. Conceptual. No numbers on arrows.
The diagram is also listed as a table below.
View as table
| Step | Outcome |
|---|---|
| Cheap model | Quality estimate |
| Pass | Ship |
| Below contract | Stronger model |
| Still below | Abstain |
End-to-end pipeline
ConceptualExtract, Ground, Route, Assure, Learn, with feedback from Learn to Route and Assure.
The diagram is also listed as a table below.
View as table
| Step | Feeds |
|---|---|
| Extract | Ground |
| Ground | Route |
| Route | Assure |
| Assure | Learn |
| Learn | Route, Assure |
Planned metrics
Metric names from the research agenda. Values stay Pending until results.json has a row.
Cost per correct answer
Pending
Abstention-adjusted accuracy
Pending
Calibration error
Pending
Escalation rate
Pending
Time to detect drift
Pending
The diagram is also listed as a table below.
View as table
| Metric | Value |
|---|---|
| Cost per correct answer | Pending |
| Abstention-adjusted accuracy | Pending |
| Calibration error | Pending |
| Escalation rate | Pending |
| Time to detect drift | Pending |
Connects to
Work with HOPN Lab
Researchers can apply through EHOPn. The lab agenda is public. Results stay Pending until measurements exist.