domains / families / data / digital-health-clinical-asr-eval

digital-health-clinical-asr-eval

separate

Stage 3 of Clinical ASR Flywheel. Score a NeMo manifest, produce the five-section KER leaderboard (by-ipa_source diagnostic). Not for ASR auth (/riva-asr).

When to use

• Score a clinical-ASR manifest, produce a five-section KER leaderboard, and route the user via the post-eval decision tree. Methodology details (metric definitions, normalization, leaderboard order, special-case routing) live in Critical Wo…
• "Score my ASR manifest"
• "What's the KER on Parakeet TDT v2?"
• "Run the eval on cycle-N"
• "Compare two ASR models on the clinical benchmark"
• "Generate the leaderboard"
• "I have a manifest.jsonl, how do I score it?"
• "Why is KER 0.4 when WER is 0.07?"

When not to use

• **English-only by default.** Tokenization + normalization assume Latin script and en-US lexicon.
• **Strict-contiguous KER is conservative.** A near-miss like `cefa zolin` counts as a miss. That's intentional — pharmacy lookups fail on near-misses. Users wanting "soft" matching can switch to phoneme-level edit distance, which is a metho…
• **One model per eval run.** Comparing two models means running the eval twice and diffing the two `leaderboard_cycle<N>.md` files (or extending the recipe to write multi-model rows yourself).
• **Hosted-only paths assumed.** Self-hosted NIMs work but require `/riva-nim-setup` first.

Skill page

Source