MRPC FAQ

Instructions for MRPC can be found here!

Thanks for doing our HITs! With your help, we think we’ll be able to build some pretty exciting technologies to help computers better understand human language.

Why is there a training phase?

Sometimes this task can be tricky and we want you to get a sense of what the task is before you work on the main project. We already have labels for these examples, and what we’re doing here is gathering data to understand how well people perform on the task when they have some instructuions and a little bit of training.

Can I immediately start working on the main project after completing training?

Unfortunately no, there is no automatic way for us to add you to our qualified list of workers. We go through the submitted HITs on the training task at least once a day and add worker IDs to the qualified list. Once your name is on the list, you will be notified thruugh Hyrbid and will be able to start working on the main project.

Should the same label be more common than not same?

Ideally, yes. We already have labels for all of these pairs, and we know that there are more same labeled pairs. So if you find yourself assigning slightly more same labels, don’t be alarmed. If your responses are balanced or skew more towards not same, reconsider how you are evaluating the prompts.

Will you reject my work?

No. Unless it’s clear to us that you are assigning labels across many HITs without even considering the prompts, we won’t reject any of your work.

Where do these pairs of sentences come from?

Each sentence was taken from a news article and collected into pairs as part of a previous data collection effort.

When should I fill out the ‘problems’ field?

You should fill this out if you can’t complete the HIT, and not otherwise. This could be if HIT interface is partially broken (an empty page, for example). If there is a typo in a sentence, but you think you know what it means anyway, please don’t report it. Never put anything in this field if there isn’t a problem.

Who are you?

We are busy graduate students with the Bowman Group, a subgroup of the ML² group at New York University Center for Data Science. We are also affiliated with the NYU Departments of Data Science, Computer Science, and Linguistics.

I have more questions!

Email us through Crystal (crystal.phoenixsystems@gmail.com)! Or email through Hybrid!