Skip to content

A collection of MTurk templates designed to make complex tasks easier for human annotators.

Notifications You must be signed in to change notification settings

ahwang16/mturk-templates

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MTurk Templates for Complex Human Annotation Tasks

A collection of complex Amazon Mechanical Turk templates I created for my work. These templates are designed to make complex annotations tasks easier for human annotators to evaluate by breaking down each HIT into several subtasks. I will add templates and references here as I work on them.

Compare Recipes

This MTurk Task asks Turkers to compare two recipes by displaying the entirety of one and providing the other one step at a time. The ingredients list is shown to the far left to help ground the task in background information. Turkers answer questions about each step of the second recipe, followed by questions about both recipes in general. This template could be adapted for any task that involves comparing two long texts, like summarization, style transfer, machine translation, or question answering.

MTurk interface for the Compare Recipes task

Used in Large Language Models as Sous Chefs: Revising Recipes with GPT-3 (Hwang et al., 2023).

Use and Citation

Feel free to fork this repository, submit pull requests, or use these templates for your work. If you use any of these templates for your work or inspiration, please cite this repository:

@software{Hwang_MTurk_Templates_for_2023,
  author = {Hwang, Alyssa},
  month = jun,
  title = {{MTurk Templates for Complex Human Evaluation Tasks}},
  url = {https://github.com/ahwang16/mturk-templates},
  version = {1.0.0},
  year = {2023}
}

About

A collection of MTurk templates designed to make complex tasks easier for human annotators.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages