Monday, July 11, 2011

Problem with Mechancial Turk Masters

This is a general posting for problems regarding mTurk masters.  This post is directed towards potential requesters, amazon, and workers.  Please note the first issue while reading this.

1.  mTurk masters are unknown.  Meaning that there is no documentation on what a worker needs to go about to get this qualification.  There is no list of who the current masters are, how they became masters, or any evidence that new masters are getting the qualification and existing masters are losing the qualification for not keeping up with whatever it is that they did to get the qualification.

2.  It is doubtful masters earned the qualification.  This very point makes current qualification holders super defensive because they were probably brainwashed into thinking they are super special.  If you read about masters amazon promotes them as doing thousands of hits from multiple requesters.  There are workers, myself included, who aim for 1,000 hits a day.  My approval rating is 99.3% (most of my work comes from majority rules requester) and over 120k approved hits.  I do work for fair paying requesters who pay fast and are responsive.  There are workers with much better stats then mine with approval ratings upwards of 99.9%, been turking longer, and more approved.  They don't have it. 

3.  Masters were most likely given this qualification based on some random hits several months ago.  This part, like everything else, simply unknown.  Current masters don't know how they were chosen but they want to believe they are special.  Non masters simply point out their stats are better than current masters and they aren't masters.  As of this writing, it was hinted that amazon might extend the qual to more workers, but there is no evidence that any new worker ever received this qual yet.  Also, the only evidence is that current masters were the ones chosen for a beta started months ago.  Nobody knows who was chosen.  It was suggested that working for specific requesters granted the qual.  However, shouldn't all photo mod requesters be considered for their workers?  And some current masters admitted they didn't do work for the requesters that was listed as being watched.  This suggests there was some random alpha testing done and the people selected to beta did some alpha hits that are unknown. 

4.  If legit, you are getting old workers.  As stated, there is no evidence that more workers were given this qualification, nor is there evidence existing workers were removed from the system.  There is no hierarchy to show who the best of the best is so workers who want the qual can aim for it.  So, if we assume that all the current masters actually earned it, they were the "best of the best" about 4 months ago.  Wouldn't you want the best of the best today?  Chances are there are probably enough scamming workers who got the qual because they'll do any sort of hits while sharing an account.  This, of course, isn't saying all master workers are scammers.  But as there are no checks and balances you really don't know what you are getting.

5.  Masters aren't paid more.  Someone suggested that during beta amazon made reference that the increased cost to requesters would be shared with workers.  It sounds like this isn't the case. Why should it be?  If you want to pay workers more you should pay workers more.

6.  Masters won't work for peanuts.  It sounds like there are some tasks that have higher requirements because the requester is paying more.  If your hits are subjective, uses majority rules and pays bad people won't touch your hits.  This is amazons fault.  After all, if you are paying amazon more you'll want to make sure your work is being done properly.  But you aren't paying for any additional benefits from amazon and, in fact, you are preventing workers that want to do your hits from doing them because they weren't given this imaginary qualification.

7.  Masters get a human response when suspended.  Why?  Shouldn't this be given to all workers?  Amazon should hand check workers accounts before suspending them but they won't even do that for master accounts they suspend.  Non masters will only ever get form letters from amazon that doesn't say anything.  As a worker I'd pay for this kind of service for workers.  My account shouldn't be at risk because of scamming requesters and it's not fair that a human being won't review my account if I'm not a master.

8.  What is the logic of giving any work to masters?  I'm reminded of a Canadian show called "Endgame".  It's about a chess world champion (btw that doesn't know how to castle) who is hired to help solve crimes.  This guy is supposed to be really good at chess but what makes him qualified to solve crimes?  What makes someone super-duper skilled at counting people in pictures qualified to write articles or do transcription? 

9.  Masters don't have to type captchas.  The biggest benefit for masters is having a human being respond to suspensions.  Next is not having to type in captchas.  When I do 1,000 hits a day I'm having to prove every 6-8 minutes I'm a human being.  How is this fair?  I don't mind because if it helps against spamming workers I don't mind.  But is it really?  

*** Conclusions

Amazon:

1.  All workers deserve to have someone respond to them when suspended.  Stop with the friday suspensions so that you don't have to deal with workers over the weekend.  If it costs more money to hire people for this then charge requesters a percentage more to cover the costs.

2.  Captchas suck.  Figure out a better way to stop scammers. 

3.  Make these qualifications something people can shoot for.  If I were to hire a piano tutor for my child I'd want some proof they are actually good besides someone said so.  My mother hired a cleaning lady that came recommended by someone.  That cleaning lady ended up stealing everything from stuff in the garage, money from my brothers funeral, to drugs to fight my moms cancer.  While different, master turkers should actually be the best of the best and not hide behind some imaginary algorithm.

4.  Don't allow requesters to just give any old work to master turkers.  That defeats the purpose.

Requesters:

1.  If you want to pay more to workers, pay them more.

2.  The added cost to amazon doesn't give you anything special.

3.  Your best bet is figure out how to set up qualifications, set them up, and give better scores to your better requesters.  You should want to find workers who do your work as best as they can and not pay more because amazon claims something.

Workers:

Most likely you aren't special.  You did some hit months ago and amazon brainwashed you into thinking you are special.  You are not.  They just needed these "master turkers" to sell a lie to requesters in hopes that they'll pay amazon a premium.  As a chess player I know exactly where I stand in world rankings.  If I wanted to become a grandmaster I know who I have to beat and beat them.  This should be a very simple solution and actually means something if amazon did this right.  I could tell amazon how to do it but they wouldn't listen because they don't care to find out who their best workers are.  All they are interested in is labeling people to claim they are the best so they can profit off the lies.

All this said, I'm not saying anyone with a master turk qual are bad workers, scammers, or anything.  And if it ever becomes available how people earned these qualifications I'll eat my hat. 

3 comments:

  1. Thanks for posting this information. Trying to find information regarding how to receive a Masters level status seems nearly impossible. Though it does not answer my question totally, it gives me more insight to the issue. Thanks again!

    ReplyDelete
  2. I've just delved into the world of Mechanical Turk and I agree that this whole "Masters" set-up is fishy. I am willing to go the extra mile for a requester. People like me just need the chance.

    ReplyDelete
  3. I would love to understand how to do 1,000 hits a day without dying. How is that possible. You must be the fastest typer and thinker around. If I get 100 a day I feel like I've done something. If you have time could you explain that a little bit please.

    ReplyDelete