whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence.

• Description
• Installation
• Usage
  • Python
  • Command line
  • Plotting word alignment
  • Example output
• Acknowlegment
• Citations

Description

Whisper is a set of multi-lingual robust speech recognition models, trained by OpenAI, that achieve state-of-the-art in many languages. Whisper models were trained to predict approximative timestamps on speech segments (most of the times with 1 sec accuracy), but cannot originally predict word timestamps. This repository proposes an implementation to predict word timestamps, and give more accurate estimation of speech segments, when transcribing with Whipser models. Besides, a confidence score is assigned to each word and each segment (both computed as "exp(mean(log probas))" on the probabilities of subword tokens).

The approach is based on approach Dynamic Time Warping (DTW) applied to cross-attention weights, as done by this notebook by Jong Wook Kim. There are some additions to this notebook:

• The start/end estimation is more accurate.
• Confidence scores are assigned to each word.
• If possible (without beam search...), there no additional inference steps are required to predict word timestamps (word alignment is done on the fly, after each speech segment is decoded).
• There is a special care about memory usage: whisper-timestamped is able to process long files, with little additional memory with respect to the regular use of Whisper model.

Notes on other approaches

An alternative relevant approach to recover word-level timestamps consists in using wav2vec models that predict characters, as successfully implemented in whisperX. But these approaches have several drawbacks, which does not have approaches based on cross-attention weights such as whisper_timestamped. These drawbacks are:

• The need to find one wav2vec model per language to support, which badly scales to the multi-lingual capabilities of Whisper.
• The need to handle (at least) one additional neural network (wav2vec model), which consumes memory.
• The need to normalize characters in whisper transcription to match the character set of wav2vec model. This involves awkward language-dependent conversions, like converting numbers to words ("2" -> "two"), symbols to words ("%" -> "percent", "€" -> "euro(s)")...
• The lack of robustness around speech disfluencies (fillers, hesitations, repeated words...) that are usually removed by Whisper.

An alternative approach, that does not require an additional model, is to look at the probabilities of timestamp tokens estimated by the Whisper model after each (sub)word token is predicted. It was implemented for instance in whisper.cpp and stable-ts. But this approach lacks of robustness, because Whisper models do not have been trained to output meaningful timestamps after each word. Whisper models tend to predict timestamps only after a certain number of words have been predicted (typically at the end of a sentence), and the probability distribution of timestamps outside this condition may be inaccurate. In practice, these methods can produce results that are totally out-of-sync on some periods of time (we observed that especially when there is jingle music). Also the timestamp precision of Whisper models tend to be rounded to 1 second (as in many video subtitles), which is too inaccurate for words, and reaching a better accuracy is tricky.

Installation

Requirements:

• python3 (version higher or equal to 3.7, at least 3.9 is recommended)
• ffmpeg (see instructions for installation on the whisper repository

You can install whisper-timestamped either by using pip:

pip3 install git+https://github.com/linto-ai/whisper-timestamped

or by cloning this repository and running installation:

git clone https://github.com/linto-ai/whisper-timestamped
cd whisper-timestamped/
python3 setup.py install

If you want to plot alignement between audio timestamps and words (as in this section), you also need matplotlib

pip3 install matplotlib

When using pip, the library can be updated to the latest version using

pip install --upgrade --no-deps --force-reinstall git+https://github.com/linto-ai/whisper-timestamped

Usage

Python

In python, you can use the function whisper_timestamped.transcribe() that is similar to the fonction whisper.transcribe()

import whisper_timestamped
help(whisper_timestamped.transcribe)

The main difference with whisper.transcribe() is that the output will include a key "words" for all segments, with the word start and end position. Note that word will include punctuation. See example below.

Besides, default decoding options are different, in order to favour efficient decoding (greedy decoding instead of beam search, and no temperature sampling fallback). To have same default as in whisper, use beam_size=5, best_of=5, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0).

There are also additional options related to word alignement.

In general, by importing whisper_timestamped instead of whisper in your python script, it should do the job, if you use transcribe(model, ...) instead of model.transcribe(...):

import whisper_timestamped as whisper

audio = whisper.load_audio("AUDIO.wav")

model = whisper.load_model("tiny", device="cpu")

result = whisper.transcribe(model, audio, language="fr")

import json
print(json.dumps(result, indent = 2, ensure_ascii = False))

Command line

You can also use whisper_timestamped on the command line, similarly to whisper. See help with:

whisper_timestamped --help

The main differences with whisper CLI are:

• Output files:
  • The output JSON contains word timestamps and confidence scores. See example below.
  • There is an additional CSV output format
  • For SRT, VTT, TSV formats, there will be additional files saved with word timestamps
• Some default options are different:
  • By default, no output folder is set: Use --output_dir . for Whisper default
  • By default, there is no verbose: Use --verbose True for Whisper default
  • By default, beam search decoding and temperature sampling fallback are disabled, to favour an efficient decoding. To set the same as Whisper default, you can use --accurate (which is an alias for --beam_size 5 --temperature_increment_on_fallback 0.2 --best_of 5).
• There are some additional specific options:
  • --compute_confidence to enable/disable the computation of confidence scores for each word.
  • --punctuations_with_words to decide whether punctuation marks should be included or not with preceding words.

An example command line to process several files with the tiny model and output results in the current folder as whisper would do by default:

whisper_timestamped audio1.flac audio2.mp3 audio3.wav --model tiny --output_dir .

Plot of word alignment

Note that you can use option plot_word_alignment of python function whisper_timestamped.transcribe(), or option --plot of whisper_timestamped CLI in order to see the word alignment for each segment.

• The upper plot represents the transformation of cross-attention weights that is used for the alignement with Dynamic Time Warping. The abscissa represents the time and the ordinate represents the predicted tokens; with special timestamp tokens at first and at last, and then (sub)words and punctuations in the middle.
• The lower plot is a MFCC representation of the input signal (features used by Whisper, based on Mel-frequency cepstrum).
• The vertical dotted red lines show where the word boundaries are found (with punctuation marks "glued" with the previous word).

Example output

Here is an example output of whisper_timestamped.transcribe(), that can be seen by using CLI

whisper_timestamped AUDIO_FILE.wav --model tiny --language fr

{
  "text": " Bonjour! Est-ce que vous allez bien?",
  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 0.5,
      "end": 1.2,
      "text": " Bonjour!",
      "tokens": [ 25431, 2298 ],
      "temperature": 0.0,
      "avg_logprob": -0.6674491882324218,
      "compression_ratio": 0.8181818181818182,
      "no_speech_prob": 0.10241222381591797,
      "confidence": 0.51,
      "words": [
        {
          "text": "Bonjour!",
          "start": 0.5,
          "end": 1.2,
          "confidence": 0.51
        }
      ]
    },
    {
      "id": 1,
      "seek": 200,
      "start": 2.02,
      "end": 4.48,
      "text": " Est-ce que vous allez bien?",
      "tokens": [ 50364, 4410, 12, 384, 631, 2630, 18146, 3610, 2506, 50464 ],
      "temperature": 0.0,
      "avg_logprob": -0.43492694334550336,
      "compression_ratio": 0.7714285714285715,
      "no_speech_prob": 0.06502953916788101,
      "confidence": 0.595,
      "words": [
        {
          "text": "Est-ce",
          "start": 2.02,
          "end": 3.78,
          "confidence": 0.441
        },
        {
          "text": "que",
          "start": 3.78,
          "end": 3.84,
          "confidence": 0.948
        },
        {
          "text": "vous",
          "start": 3.84,
          "end": 4.0,
          "confidence": 0.935
        },
        {
          "text": "allez",
          "start": 4.0,
          "end": 4.14,
          "confidence": 0.347
        },
        {
          "text": "bien?",
          "start": 4.14,
          "end": 4.48,
          "confidence": 0.998
        }
      ]
    }
  ],
  "language": "fr"
}

Acknowlegment

• whisper: Whisper speech recognition (License MIT).
• dtw-python: Dynamic Time Warping (License GPL v3).

Citations

If you use this in your research, just cite the repo,

@misc{lintoai2023whispertimestamped,
  title={whisper-timestamped},
  author={Louradour, J{\'e}r{\^o}me},
  journal={GitHub repository},
  year={2023},
  publisher={GitHub},
  howpublished = {\url{https://github.com/linto-ai/whisper-timestamped}}
}

as well as OpenAI Whisper paper,

@article{radford2022robust,
  title={Robust speech recognition via large-scale weak supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}

and this paper for Dynamic-Time-Warping

@article{JSSv031i07,
  title={Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package},
  author={Giorgino, Toni},
  journal={Journal of Statistical Software},
  year={2009},
  volume={31},
  number={7},
  doi={10.18637/jss.v031.i07}
}