Adalat AI, a company focused on improving judicial efficiency through audio transcription, encountered an issue where Indic language transcripts would abruptly stop after six seconds. This problem was traced to a token limit in the Whisper model, which caused the replacement character � to appear at the end of incomplete transcripts. The issue was identified during their work on multilingual courtroom systems in India, where they manage audio transcription for about 25% of courtrooms.
The root cause was determined to be the token limit in Whisper, which restricts the number of tokens the model can generate. Whisper is a transformer model that processes exactly 30 seconds of audio, producing up to 448 tokens. These tokens are divided between the start of a transcript, timestamp tokens, and the actual text. OpenAI's implementation reserves half of these tokens for previous text, limiting new tokens to 224. In contrast, Hugging Face Transformers allows up to 448 new tokens. The problem with Indic languages arises because each character in these scripts is three bytes, resulting in three tokens per character, which quickly exceeds the token limit.
The issue was further exacerbated by the fact that Indic scripts are underrepresented in the training data for Whisper's tokenizer, which is based on GPT-2's byte-level BPE tokenizer. This leads to fewer byte-level merges for Indic languages, making each character three tokens. As a result, the token limit is reached much faster for Indic languages than for English. The solution proposed by Adalat AI involves shortening the VAD window to six or seven seconds to keep transcripts below the token limit, preventing the appearance of the replacement character.
Source: huggingface
