I’m not convinced that automation would decrease quality on a per-submission basis, but it certainly introduces thorny dependency issues. But I think that’s an acceptable price to pay in order to potentially expand the marketplace by several orders of magnitude.

Davidc, perhaps I’m naive to take Mechanical Turk at face value. Though even spam creation is ripe for automation. Anyway, I do like Hanson’s attitude (this link worked better for me). But my goal is here is the advancement of commerce rather than science. That the latter would benefit is merely a benign side effect.

I think you have a good idea though. How about a online prize site. You would need measurable tests on each task and give the reward to the best algorithm. Robin Hanson has a good paper on the value of prizes for science here
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67.9839&rep=rep1&type=pdf

I imagine Amazon’s reasoning is to prevent fraud (e.g., someone builds a script that automatically submits thousands of bad results, hoping the HITs will be approved). Certainly the requestors could build ways to not approve such results, however, I imagine Amazon is requestor-focused, especially in ameliorating their concerns for quality. Some requestors I know approve all HITs as policy and just avoid any fraudulent workers in future tasks.

Humans also occasionally submit bad results, but I think the overall level of bad results may increase if automated submissions were facilitated.

There should definitely be MTurk like platforms that are optimized for developers (ideally also on the Amazon system), for those interested in finding out ways to automate tasks. That way requestors could pick and choose.