How to use gemma for multi-round conversations

[ranck626]

Thank a lot for your great work！
I deployed gemma-2b locally. I would like to understand how to have multiple rounds of dialog effectively.

I searched the internet and found that I could type in previous conversations to get answers for the next round. But I don't know exactly how it works inside Gemma. I hope to get your pointers or if you can recommend some existing tutorials.

I'm not a native English speaker and may have some grammatical problems. Thank you for your attention.

[KaneGreen]

The following code is copied from the Gemma's kaggle page

# Use the model

USER_CHAT_TEMPLATE = "<start_of_turn>user\n{prompt}<end_of_turn>\n"
MODEL_CHAT_TEMPLATE = "<start_of_turn>model\n{prompt}<end_of_turn>\n"

prompt = (
    USER_CHAT_TEMPLATE.format(
        prompt="What is a good place for travel in the US?"
    )
    + MODEL_CHAT_TEMPLATE.format(prompt="California.")
    + USER_CHAT_TEMPLATE.format(prompt="What can I do in California?")
    + "<start_of_turn>model\n"
)

model.generate(
    USER_CHAT_TEMPLATE.format(prompt=prompt),
    device=device,
    output_len=100,
)

It tells how to use <start_of_turn> and <end_of_turn> to control roles and turns.

This code can inspire us to write a program for multiple rounds of dialogue, but I still have some doubts:

1. What should I do if there are too many dialogue rounds and the number of tokens in the Gemma model is exceeded?
2. I am very confused: why to call USER_CHAT_TEMPLATE.format() again in the model.generate() function call? Such an operation will produce the following strange input:

   <start_of_turn>user
   <start_of_turn>user
   What is a good place for travel in the US?<end_of_turn>
   <start_of_turn>model
   California.<end_of_turn>
   <start_of_turn>user
   What can I do in California?<end_of_turn>
   <start_of_turn>model
   <end_of_turn>
   

[pkgoogle]

Hi @ranck626, Does the above response answer your question?