Huzhou Dialect Speech Recognition

Whisper-medium fine-tuned on ~15,000 clips of Huzhou dialect (湖州话), a Wu Chinese dialect that standard speech recognition handles poorly. Output is written in Mandarin characters. Model on Hugging Face · Code on GitHub

News broadcasts · 1,705 held-out clips

26.7%vs98.8%

character error rate: fine-tuned vs baseline Whisper

Real speakers · 819 everyday sentences

55.3%vs93.6%

character error rate: fine-tuned vs baseline Whisper

Hand-picked Huzhou dialect clips the model never trained on, with English translations: news broadcasts, and real speakers reading everyday sentences.

Sample clips

Audio

Press Transcribe to compare the models.

Don't trust hand-picked examples? Roll a random clip (good and bad) from either held-out test set, or choose one by number. News broadcasts are from the same TV channel as the training data; real speakers are people reading everyday sentences, mostly recorded on phones, which is harder. Clips and transcripts are in examples/test_set/ and examples/real_speakers/.

Test set

Clip number

Test clip

Pick a clip, then press Transcribe.

Why ~15 seconds? A deliberate cost trade-off

This demo runs on a serverless AWS architecture built to cost close to nothing when no one is using it: the browser uploads audio straight to S3, which triggers a containerized Lambda function that runs the model and writes the result to DynamoDB, while API Gateway serves the page and job status. Everything is defined as code (AWS SAM).

Fitting the model: Lambda has no GPU and caps memory at 3 GB, so the 769M-parameter model was converted to 8-bit (CTranslate2, 3 GB → 742 MB). On held-out test clips its error rate stays within about 1 point of the full model (news: 24.6% vs 23.8%).
Tuning: profiling traced an early accuracy loss to the audio preprocessing rather than the 8-bit weights, and benchmarking chose x86 over ARM processors (about 30% faster per clip).
The result: about 13 seconds per clip on 2 CPU cores (up to a minute if the site has been idle, while Lambda starts up), at about $0.0007 per transcription. An always-on GPU server would answer in about a second but cost about $590 a month.

* Baseline outputs are precomputed, not run live. They were generated once in advance by running the unmodified openai/whisper-medium on every test clip, with the same settings (language: Chinese, task: transcribe, same loop guard). The fine-tuned model's output is computed live every time you press Transcribe. Error rates are character error rate (CER): edit distance divided by reference length, ignoring punctuation, which can exceed 100% when the output is longer than the reference. The scores in the header are for the full-precision model. All featured and random clips are held out: none were used in training. Real-speaker sentences come from everyday dialogue and narration, so word choices the model renders differently count as errors even when the meaning is kept.