Server crashes with index out of bounds

[samrat]

This promptlib program when run with phi2 seems to consistently crash the server with the following stacktrace.

program = pl.PromptProgram(aici)

# load a grammar file
c_grammar = open("../../../controllers/aici_abi/grammars/c.y", "r").read()

p = program + "Please write a C program that recursively calculates the Fibonnaci sequence to N digits.\n"
p += "#include<stdio.h>\nint main("
p = p.gen(max_tokens = 200, yacc=c_grammar, set_var="code", stop_at='\n\n')

result = program.run()

print(result['storage']['code'])

Stacktrace

thread '<unnamed>' panicked at controllers/aici_abi/src/toktree.rs:223:37:
index out of bounds: the len is 50295 but the index is 50839
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::panicking::panic_bounds_check
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:208:5
   3: <usize as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/slice/index.rs:255:10
   4: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/slice/index.rs:18:9
   5: <alloc::vec::Vec<T,A> as core::ops::index::Index<I>>::index
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2770:9
   6: aici_abi::toktree::TokTrie::token
             at /Users/samrat/code/aici/controllers/aici_abi/src/toktree.rs:223:37
   7: aici_abi::toktree::TokTrie::decode::{{closure}}
             at /Users/samrat/code/aici/controllers/aici_abi/src/toktree.rs:232:27
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:305:13
   9: core::option::Option<T>::map
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1072:29
  10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/map.rs:104:26
  11: <core::iter::adapters::fuse::Fuse<I> as core::iter::adapters::fuse::FuseImpl<I>>::next
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/fuse.rs:355:9
  12: <core::iter::adapters::fuse::Fuse<I> as core::iter::traits::iterator::Iterator>::next
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/fuse.rs:46:9
  13: <core::iter::adapters::flatten::FlattenCompat<I,U> as core::iter::traits::iterator::Iterator>::next
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/flatten.rs:601:29
  14: <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/flatten.rs:58:9
  15: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_from_iter_nested.rs:26:32
  16: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_from_iter.rs:33:9
  17: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2791:9
  18: core::iter::traits::iterator::Iterator::collect
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/traits/iterator.rs:2054:9
  19: aici_abi::toktree::TokTrie::decode
             at /Users/samrat/code/aici/controllers/aici_abi/src/toktree.rs:233:14
  20: rllm::seq::Sequence::gen_output
             at /Users/samrat/code/aici/rllm/rllm-base/src/seq.rs:210:25
  21: rllm::engine::RllmEngine<ME>::req_output::{{closure}}
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:621:28
  22: core::iter::adapters::map::map_fold::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/map.rs:85:28
  23: <core::slice::iter::IterMut<T> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/slice/iter/macros.rs:232:27
  24: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/map.rs:125:9
  25: core::iter::traits::iterator::Iterator::for_each
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/traits/iterator.rs:858:9
  26: alloc::vec::Vec<T,A>::extend_trusted
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2923:17
  27: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_extend.rs:26:9
  28: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
  29: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_from_iter.rs:33:9
  30: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2791:9
  31: core::iter::traits::iterator::Iterator::collect
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/traits/iterator.rs:2054:9
  32: rllm::engine::RllmEngine<ME>::req_output
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:622:18
  33: rllm::engine::RllmEngine<ME>::sample::{{closure}}
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:609:27
  34: core::iter::adapters::map::map_fold::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/map.rs:85:28
  35: <core::slice::iter::IterMut<T> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/slice/iter/macros.rs:232:27
  36: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/adapters/map.rs:125:9
  37: core::iter::traits::iterator::Iterator::for_each
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/iter/traits/iterator.rs:858:9
  38: alloc::vec::Vec<T,A>::extend_trusted
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2923:17
  39: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/spec_extend.rs:26:9
  40: <alloc::vec::Vec<T,A> as core::iter::traits::collect::Extend<T>>::extend
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/vec/mod.rs:2865:9
  41: rllm::engine::RllmEngine<ME>::sample
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:605:17
  42: rllm::engine::RllmEngine<ME>::run_model
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:644:48
  43: rllm::engine::RllmEngine<ME>::step_inner
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:879:55
  44: rllm::engine::RllmEngine<ME>::step
             at /Users/samrat/code/aici/rllm/rllm-base/src/engine.rs:846:44
  45: rllm::server::inference_loop
             at /Users/samrat/code/aici/rllm/rllm-base/src/server/mod.rs:370:23
  46: rllm::server::spawn_inference_loop::{{closure}}
             at /Users/samrat/code/aici/rllm/rllm-base/src/server/mod.rs:497:9

[mmoskal]

are you using rllm-llamacpp or rllm-cuda?

[samrat]

@mmoskal I'm using rllm-llamacpp.