Skip to content

Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining, WACV 2024

License

Notifications You must be signed in to change notification settings

ugorsahin/Generative-Negative-Mining

Repository files navigation

Generative Negative Mining

Project Page | Model Checkpoint | Dataset

This is the official implementation for the paper Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining. We propose a framework that not only mines in both directions but also generates challenging negative samples in both modalities, i.e., images and texts. Leveraging these generative hard negative samples, we significantly enhance VLMs’ performance in tasks involving multimodal compositional reasoning.

Overview of the framework

Installation

Requirements: python>=3.8, cuda=11.3

git clone https://github.com/ugorsahin/Generative-Negative-Mining
cd Generative-Negative-Mining
pip install -r requirements.txt

We advise installing on a virtual environment to avoid library dependency crashes.

Data Generation

The semi-synthetic variations can be generated stage-by-stage or by using the pipeline script (if you don't have enough gpu memory)

To run the pipeline

  • Change into directory generation_pipeline by using cd generation_pipeline
  • Run the script as follows
python pipeline.py \
--tag2text-checkpoint=<path/to/tag2text_model> \
--gd-config=<path/to/gd_config> \
--gd-checkpoint=<path/to/gd_checkpoint> \
--sam-checkpoint=<path/to/sam_checkpoint> \
--sd-checkpoint=<path/to/sd_checkpoint> \
--output-dir=<path/to/output> \
--input-dir=<path/to/images> \
--root-dir=<path/to/root>

Train

To train the clip

  • Change into directory training by using cd training
  • Run the following code.
python train.py 
--epoch=<number_of_epochs> \
--mode='allinone|item_based|image_based' \
--save-path=<path/to/save_folder> \
--dataset=<path/to/variation_dataset> \
--image-root=<path/to/image_root> \
--coco-dataset=<path/to/coco_dataset> \
--coco-image_dir=<path/to/coco_images> \

Evaluation

To evaluate the model checkpoints

    • Change into directory evaluation by using cd evaluation
python eval_autogmentation.py \
--model-name=<tag-for-your-model> \
--snapshot_file=<specify if you want to evaluate one model>' \
--snapshot_folder=<specify if you want to evaluate all training>'
--evaluation-filepath=<path/to/evaluation_annotations> \
--evaluation-image-folder=<path/to/eval/images>
  • Either set snapshot-file or snapshot-folder

Resources

Training annotation file (.json)

Training images (.zip)

Pretrained model (pytorch)

Citation

If you find our work helpful in your research, please consider citing us

@misc{sahin2023enhancing,
      title={Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining}, 
      author={Ugur Sahin and Hang Li and Qadeer Khan and Daniel Cremers and Volker Tresp},
      year={2023},
      eprint={2311.03964},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      journal = {Winter Conference on Applications of Computer Vision},
}

Acknowledgments

About

Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining, WACV 2024

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published