Skip to content

Strong-AI-Lab/Logical-and-abstract-reasoning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Logical and Abstract Reasoning

Repository for the evaluation of Large Language Models on logical and abstract reasoning tasks

Installation

To install the repository, use the following command:

git clone https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning.git

To install the dependencies in a virtual environment, use the following:

cd Logical-and-abstract-reasoning
python -m venv env/
source env/bin/activate
pip install -r requirements.txt

You may need to install transformers from the repository:

pip install git+https://github.com/huggingface/transformers

Use

Evaluation

To evaluate a model in the repository, use the following command:

python run_evaluation config/model/<model_config.yaml> config/data/<data_config.yaml> --<kwarg_name> <kwarg>

You can choose the model to evaluate by changing the <model_config.yaml> file, and the dataset to evaluate the model on by changing the <data_config.yaml> file. You can add any additional arguments as <kwargs> (e.g. private API key for GPT models).

By default, all the results are saved in a csv file in the logs/ folder. You can re-compute the metrics from the evaluation run from this file by running the following:

python src/evaluate/evaluator.py logs/<results_file.csv>

Fine-tuning

To fine-tune a model on a given dataset, run the following:

python run_finetuning.py config/model/<model_config.yaml> config/data/<data_config.yaml> config/trainer/<trainer_config.yaml>

The configuration files work similarly as for evaluation. The <model_config.yaml> file contains additoinal configuration for training. The logs are saved in fine-tuning-output/ and the model weights are saved in fine-tuning-saves/.

Currently, only HuggingFace models can be fine-tuned.

LLaMA-based model instruction fine-tuning

We use the LLaMA-based model fine-tuning from the Stanford Alpaca training script. If you want to conduct a LLaMA-based model on instruction fine-tuning, you can do that by following this link.

Models

Inference Type Model Size Task Link Remark
Logical Reasoning on Reading Comprehension MERIt - Reading Comprehension paper
project
#3 on the ReClor leaderboard
LReasoner - Reading Comprehension paper
project
#6 on the ReClor leaderboard
AMR-LE - Reading Comprehension project #2 and #5 on the ReClor leaderboard
LLaMA - Reading Comprehension paper
code
Open source very large language model
LLaMA2 - Reading Comprehension paper
code
Open source very large language model
TinyLLaMA - Reading Comprehension paper
code
Open source very large language model
Alpaca - Reading Comprehension code Fine-tuned LLaMA
Vicuna - Reading Comprehension project
code
Fine-tuned LLaMA
ChatGPT - Reading Comprehension paper
project
Use api to do prompt tuning
GPT-4 - Reading Comprehension paper
project
Use api to do prompt tuning
Zephyr-7b-beta - Reading Comprehension code Fine-tuned Mistral-7b

Datasets & Benchmarks

Inference Type Dataset Size Task Link Remark
Logical Reasoning on Reading Comprehension ReClor - Reading Comprehension paper
project
Logical reasoning reading comprehension
LogiQA - Reading Comprehension paper
project
Logical reasoning reading comprehension
LogiQA V2 - Reading Comprehension project Logical reasoning reading comprehension
LogiQA Logical Reasoning Plus - Reading Comprehension project Logical reasoning reading comprehension for out-of-distribution evaluation
Abstract Reasoning ARC - Abstract Reasoning paper
code
Text version of a Visual Abstract Reasoning task
ACRE - Abstract Reasoning paper
code
Text version of a Visual Abstract Reasoning task
PVR - Abstract Reasoning paper Abstract Reasoning task
RAVEN - Abstract Reasoning paper
project
Text version of a Visual Abstract Reasoning task
Diagrammatic Logic - Abstract Reasoning code Extracted from OpenAI Evals
Logic - Abstract Reasoning code Extracted from OpenAI Evals
Logic Statements - Abstract Reasoning code Extracted from OpenAI Evals
Pattern Identification - Abstract Reasoning code Extracted from OpenAI Evals
String Patterns - Abstract Reasoning code Extracted from OpenAI Evals
List Functions - Abstract Reasoning code Extracted from Google BIG-bench

Acknowledgement

Our proposed new dataset logiqa-logical-reasoning-plus has been merged by OpenAI/Evals.