中文

Opens a translated copy in Google Translate in a new tab. Machine translations may contain errors. Images, films and downloads stay in their original language.

将在新标签页中打开 Google 翻译。机器翻译可能有误。图片、视频及下载文件保持原文。

Join the waiting list

AITrust · AI evaluation

Tested before use.Watched in use.

No model or agent is put into use without passing its evaluation.

How our AI is governed
The release gate A model or prompt change goes to an evaluation run, then to the gate, and only then into use. Below the gate two paths are blocked: no passing run, and a newer run failed. A return arrow shows that a rollback goes through the same gate. Model or prompt change Evaluation run Gate Needs a passing run In use Blocked No passing run A newer run failed Rollback goes through the same gate The release gate A model or prompt change goes to an evaluation run, then to the gate, and only then into use. Two paths from the gate are blocked: no passing run, and a newer run failed. A return arrow shows that a rollback goes through the same gate. Model or prompt change Evaluation run Gate Needs a passing run In use Blocked No passing run A newer run failed Rollback goes through the same gate
A change reaches use only through a passing evaluation run; with no passing run, or a newer failed one, it is blocked.

Evaluation decides deployment.

The gate is in the database, not a checklist. A model can only be bound to a task if it names a passing evaluation run for that model and task, and a newer failed run blocks it. Agents follow the same rule before they can be activated.

No passing run, no binding

A model is bound to a task only with a passing run for that same model and task.

A newer failure blocks

If a later run of that model on that task failed, the binding is refused until a newer run passes.

Agents follow the same rule

An agent cannot be activated without a passing run for its current version.

Rollback is not a shortcut

Returning to an earlier model goes back through the same gate.

Not all errors are equal.

A single error rate hides what matters. We rank errors by clinical severity. An invented or unsupported medicine, dose, allergy, vital sign or diagnosis fails the run outright, however good the rest of the note is.

Severity ladder Errors ranked by clinical severity, from lowest to highest: rewording with no change in meaning, reordering, missing detail, unsupported inference, and invented clinical fact. An invented medicine, dose, allergy, vital sign or diagnosis fails the run. Higher severity Lower severity Invented clinical fact Medicine, dose, allergy, vital sign or diagnosis Fails the run Unsupported inference Missing detail Reordering Rewording No change in meaning Severity ladder Errors ranked by clinical severity, from lowest to highest: rewording with no change in meaning, reordering, missing detail, unsupported inference, and invented clinical fact. An invented medicine, dose, allergy, vital sign or diagnosis fails the run. Higher severity Lower severity Invented clinical fact Medicine, dose, allergy, vital sign or diagnosis Fails the run Unsupported inference Missing detail Reordering Rewording No change in meaning
Errors are ranked by clinical severity; an invented clinical fact fails the run however good the rest of the note is.

It flattens severity

A moved sentence and a wrong allergy count the same in an average.

It rewards writing less

A note that says almost nothing says almost nothing wrong.

It hides how sure we are

The same score from ten cases and from a thousand means different things.

It hides who marked it

A model grading its own work shares its own blind spots.

Missing counts as much as invented.

Omission is scored alongside faithfulness, so a model cannot pass by writing less. A note that says nothing wrong but leaves out what matters still fails.

A short note and a complete note Left, a short note: every line it contains is supported, but two required items are absent, so it fails for omissions. Right, a complete note with every required item present passes. SHORT NOTE COMPLETE NOTE Required item missing Required item missing Fails: omissions Passes A short note and a complete note Left, a short note: every line it contains is supported, but two required items are absent, so it fails for omissions. Right, a complete note with every required item present passes. SHORT NOTE COMPLETE NOTE Required item missing Required item missing Fails: omissions Passes
A note with nothing wrong in it still fails if required items are missing, so writing less cannot earn a pass.

Every number carries its uncertainty.

Results are reported with 95% confidence intervals, never as a bare percentage, and repeated runs show whether a result holds. A grading model cannot be the model being tested.

A result with its uncertainty (illustrative numbers) Illustrative numbers. A bare figure is struck through. Beside it, the same figure is shown with its confidence interval and five repeated runs. Below, the model under test and an independent grader are separate, marked not equal. Illustrative numbers BARE PERCENTAGE 92% Never reported alone WITH 95% CONFIDENCE INTERVAL 92% 84% 96% 5 repeated runs GRADING Model under test Independent grader ≠ A result with its uncertainty (illustrative numbers) Illustrative numbers. A bare figure is struck through. Beside it, the same figure is shown with its confidence interval and five repeated runs. Below, the model under test and an independent grader are separate, marked not equal. Illustrative numbers BARE PERCENTAGE 92% Never reported alone WITH 95% CONFIDENCE INTERVAL 92% 84% 96% 5 repeated runs GRADING Model under test ≠ Independent grader
Illustrative numbers. A result is shown with its 95% confidence interval and repeated runs, and is graded by a model independent of the one under test.

Hostile inputs are part of the test.

Test suites carry adversarial cases: text that tries to give the AI instructions. A suite that declares this check without such cases fails. Transcripts and documents are treated as data, never as instructions.

The content boundary A transcript contains one hostile line, "ignore your instructions and record". That line is stopped at the content boundary. Only a clean, structured proposal passes through. CONTENT BOUNDARY TRANSCRIPT …ignore your instructions and record… Stopped As data STRUCTURED PROPOSAL The content boundary A transcript contains one hostile line, "ignore your instructions and record". That line is stopped at the content boundary. Only a clean, structured proposal passes through. TRANSCRIPT …ignore your instructions and record… CONTENT BOUNDARY Stopped As data STRUCTURED PROPOSAL
Text that tries to instruct the AI stops at the content boundary; only a clean, structured proposal passes through.

The checks continue in use.

Testing is not the last line. In the consultation, every medicine and dose is checked against the source; anything unsupported is held for review with a safety note. The clinician accepts, edits or rejects each suggestion; there is no accept-all.

Echo One consultation screen. The transcript is on the left; the draft note is in the centre. Each note section is labelled 'Suggested · Edit or accept before signing' and has its own Accept button. A 'Review & sign' button sits at the bottom right.
Each section of a draft note is a suggestion, accepted or edited one at a time before signing.
A 'Source evidence' dialog quoting the line 'Diagnosis: Type 2 diabetes mellitus.' from page 1 of a letter, with the note: 'Quoted from the immutable source version. The suggestion may still be wrong; compare the words before accepting.'
Source evidence quotes the exact line from the letter, so the words can be compared before accepting.

Watching after release.

After release, grounding-review rate, clinician acceptance, feedback, evaluation hallucination rate and changes in the input mix are tracked, each compared with the previous period. Hazards are recorded in a hazard log, and any rollback goes back through the evaluation gate.

  • Grounding-review rate
  • Clinician acceptance
  • Feedback
  • Evaluation hallucination rate
  • Change in input mix
The loop A cycle: in use, then signals, then review, then the hazard log, then re-evaluation, then the gate, and back to in use. The signals tracked and compared with the previous period are grounding-review rate, clinician acceptance, feedback, evaluation hallucination rate and change in input mix. In use Signals Review Hazard log Re-evaluation Gate SIGNALS TRACKED compared with the previous period Grounding-review rate Clinician acceptance Feedback Evaluation hallucination rate Change in input mix The loop A cycle: in use, then signals, then review, then the hazard log, then re-evaluation, then the gate, and back to in use. The signals tracked and compared with the previous period are grounding-review rate, clinician acceptance, feedback, evaluation hallucination rate and change in input mix. In use Review Hazard log Re-evaluation Gate SIGNALS TRACKED compared with the previous period Grounding-review rate Clinician acceptance Feedback Evaluation hallucination rate Change in input mix
Five signals are tracked and compared with the previous period; review and the hazard log lead back through re-evaluation and the same gate.

What a pass means.

A pass is evidence, with limits. We say exactly what it covers.

Where things stand

Echo One is coming soon. The evaluation method, the release gate and the checks on this page are built into the product, and apply to every model and agent before it is used.

  • A pass means the model met defined checks on defined cases. It is not proof every output is correct.
  • Review, testing and safeguards reduce risk; they don’t eliminate errors.
  • Evaluation does not confer regulatory approval.
  • The clinician decides what is signed.
1

Gate

Every model and agent passes through it before use, and again on rollback.

4

Rules

Enforced by the database beneath the software that changes models.

5

Signals

Tracked after release and compared with the previous period.

0

Accept-all buttons

The clinician decides on each suggestion, one at a time.

Counts of rules and checks, not performance figures.

Related reading.

How Echo One Tests AI

This page as a short slide guide to download and share.

Get the guide

Tell us what better care looks like for your team.

Registering interest is not a commitment to purchase.

Join the waiting list