Thanks and questions...

[beejaya]

@mayooear thanks for this nice project. Very helpful. Have a couple of items:

1. The current setup is missing a required package: sentence_transformers
2. Any thoughts on how to run this in GPU?

[saradhix]

The required package sentence-transformers is available. There is a hyphen instead of underscore in the package name.
Currently the model is the quantized version of the mpt-30b-chat model which is being loaded with Ctransformers. You can use the original huggingface llm with this code

import transformers
llm = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-30b-chat',
  trust_remote_code=True
)

You should replace lines 73 to 78 in question_answer_docs.py with the above code. It should work fine ( I haven't tried) with GPU if you have the GPU which can hold the 30b parameter model.