Adding Llama Guard notebooks #400
Conversation
…s to a file for later evaluation as well.
…ma model weights.
… Usign heuristic 1 - safe probability to avoid getting the probability from the model.
…ce script as well. Fixing readmes
There was a problem hiding this comment.
What is the new naming convention for the files? Is it lowercase or capitalcase?
| **Note on Llama Guard** | ||
| Use this command for testing with a quantized Llama model, modifying the values accordingly: | ||
|
|
||
| `python inference.py --model_name <path_to_regular_llama_model> --prompt_file <path_to_prompt_file> --quantization --enable_llamaguard_content_safety` |
There was a problem hiding this comment.
Looks like inference.py seems to be deleted? Where should this command be run from? Should we provide a cd instruction before python command?
There was a problem hiding this comment.
This is the inference.py script I'm referencing here, not the one in the llama_guard directory.
| @@ -2,14 +2,14 @@ | |||
| <!-- markdown-link-check-disable --> | |||
| Llama Guard is a language model that provides input and output guardrails for LLM deployments. For more details, please visit the main [repository](https://github.com/facebookresearch/PurpleLlama/tree/main/Llama-Guard). | |||
|
|
|||
| This folder contains an example file to run Llama Guard inference directly. | |||
| This folder contains example notebooks on running Llama Guard stand alone and validating Llama Guard performance against a reference dataset. The dataset is not provided, only the format in which it should be to use the scripts out of the box. Additionally, Llama Guard is being used as an optional safety checker when running the regular Llama [inference script](../../inference/local_inference/inference.py). | |||
There was a problem hiding this comment.
Should we callout this can be used to convert ToxicChat dataset using script to run validation on LlamaGuard?
There was a problem hiding this comment.
This is not intended to convert the toxic chat dataset yet, but it's the base for that in the future.
|
|
||
| ## Requirements | ||
| 1. Access to Llama guard model weights on Hugging Face. To get access, follow the steps described [here](https://github.com/facebookresearch/PurpleLlama/tree/main/Llama-Guard#download) | ||
| 2. Llama recipes package and it's dependencies [installed](https://github.com/albertodepaola/llama-recipes/blob/llama-guard-data-formatter-example/README.md#installation) | ||
| 3. A GPU with at least 21 GB of free RAM to load both 7B models quantized. | ||
| 3. A big enough GPU to load the models |
There was a problem hiding this comment.
Big enough is too vague. Can we be specific?
There was a problem hiding this comment.
@albertodepaola thanks for the PR, I wonder if there is chance we make a google colab link available for this notebook as well?
For quantization are we using bits&bytes here?
What does this PR do?
Adds notebooks to run Llama Guard from HF or local weights. Adds validation notebook to test Llama Guard performance on a custom dataset. The dataset is not provided, example datasets to come in future versions
Feature/Issue validation/testing
Tested running both notebooks and checking the results are as expected.
For Inference, the sample prompts are run through the downloaded HF model and the results printed.
For Validation, a sample dataset is run through the model and average presicion printed as well.
Before submitting
Pull Request section?
to it if that's the case.