“A man’s got to know his limitations,” as Clint Eastwood famously said as Dirty Harry in Magnum Force. Well, as Panos Ipeirotis reports on his blog, Turkers (the people who are paid to perform tasks using Amazon’s Mechanical Turk service), actually do know their limitations.
Read the full post for details, but here is the punch line:
Turkers can self-report accurately the difficulty of labeling an example correctly! Since example difficulty and labeling quality are strongly interconnected, this also means that they are good at estimating their own quality!
It’s great to see Mechanical Turk used in the service of productive research, and not just as a way to shill on the cheap.
The Noisy Channel 