Grammar not working

[loganlebanoff]

Running the sample wav file with a grammar gives no change in the output compared to without the grammar. I'm purposefully not giving any prompt because I want to see how it works without the help of a prompt.

Command:
./main -f samples/jfk.wav -m models/ggml-tiny.en.bin -t 8 --grammar ./grammars/chess.gbnf --grammar-penalty 100

Output:

whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-tiny.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head  = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 384
whisper_model_load: n_text_head   = 6
whisper_model_load: n_text_layer  = 4
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 1 (tiny)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =    77.11 MB
whisper_model_load: model size    =   77.11 MB
whisper_init_state: kv self size  =    8.26 MB
whisper_init_state: kv cross size =    9.22 MB
whisper_init_state: compute buffer (conv)   =   13.32 MB
whisper_init_state: compute buffer (encode) =   85.66 MB
whisper_init_state: compute buffer (cross)  =    4.01 MB
whisper_init_state: compute buffer (decode) =   96.02 MB
main: grammar:
root ::= init color [.]
init ::= [ ] [r] [e] [d] [,] [ ] [g] [r] [e] [e] [n] [,] [ ] [b] [l] [u] [e]
color ::= [,] [ ] color_4
prompt ::= init [.]
color_4 ::= [r] [e] [d] | [g] [r] [e] [e] [n] | [b] [l] [u] [e]


system_info: n_threads = 8 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 8 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:07.960]   And so my fellow Americans ask not what your country can do for you
[00:00:07.960 --> 00:00:10.760]   ask what you can do for your country.


whisper_print_timings:     load time =   288.52 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    12.49 ms
whisper_print_timings:   sample time =    44.04 ms /   139 runs (    0.32 ms per run)
whisper_print_timings:   encode time =   466.53 ms /     1 runs (  466.53 ms per run)
whisper_print_timings:   decode time =    10.08 ms /     2 runs (    5.04 ms per run)
whisper_print_timings:   batchd time =   128.82 ms /   133 runs (    0.97 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =   962.72 ms

It doesn't seem to make any difference if I increase the grammar-penalty or use a different grammar. It doesn't even work when I run it with audio that is in the grammar.

Command:
./main -f knight.wav -m models/ggml-tiny.en.bin -t 8 --grammar ./grammars/chess.gbnf --grammar-penalty 100

Output:

whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-tiny.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head  = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 384
whisper_model_load: n_text_head   = 6
whisper_model_load: n_text_layer  = 4
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 1 (tiny)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:      CPU total size =    77.11 MB
whisper_model_load: model size    =   77.11 MB
whisper_init_state: kv self size  =    8.26 MB
whisper_init_state: kv cross size =    9.22 MB
whisper_init_state: compute buffer (conv)   =   13.32 MB
whisper_init_state: compute buffer (encode) =   85.66 MB
whisper_init_state: compute buffer (cross)  =    4.01 MB
whisper_init_state: compute buffer (decode) =   96.02 MB
main: grammar:
root ::= init move root_3 root_4 [.]
init ::= [ ] [r] [o] [o] [k] [ ] [t] [o] [ ] [b] [4] [,] [ ] [f] [3]
move ::= [,] [ ] move_12 [a-h] [1-8]
root_3 ::= move |
root_4 ::= move |
prompt ::= init [.]
move_6 ::= move_7 [ ] move_11
move_7 ::= piece | pawn | king
piece ::= [b] [i] [s] [h] [o] [p] | [r] [o] [o] [k] | [k] [n] [i] [g] [h] [t] | [q] [u] [e] [e] [n]
pawn ::= [p] [a] [w] [n]
king ::= [k] [i] [n] [g]
move_11 ::= [t] [o] [ ] |
move_12 ::= move_6 |


system_info: n_threads = 8 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0

main: processing 'knight.wav' (44360 samples, 2.8 sec), 8 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:03.640]   night to E5.


whisper_print_timings:     load time =   267.92 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     6.95 ms
whisper_print_timings:   sample time =     9.25 ms /    34 runs (    0.27 ms per run)
whisper_print_timings:   encode time =   448.45 ms /     1 runs (  448.45 ms per run)
whisper_print_timings:   decode time =     8.05 ms /     1 runs (    8.05 ms per run)
whisper_print_timings:   batchd time =    25.38 ms /    29 runs (    0.88 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =   776.40 ms

It should give knight to e5 instead of night to E5.

[ggerganov]

Along with --grammar you have to also pass the name of the top-level grammar rule to use: --grammar-rule root for example

[loganlebanoff]

Thanks that worked!