image_to_text

Image-to-Text

This example implements image-to-text with Redco.

Requirement

Install Redco

pip install redco==0.4.8

Usage

Download MSCOCO data

bash download_mscoco_dataset.sh

Training

python main.py \
  --data_dir=./mscoco_data \
  --model_name_or_path=nlpconnect/vit-gpt2-image-captioning \
  --per_device_batch_size 8 \
  --num_beams 4

See def main(...) in main.py for all the tunable arguments.

Customize image encoder and text decoder

See this HuggingFace example scrips to customize your model.