Mechanical Turk had a good run, but not surprised it's shutting down. I'm sure the platform was getting flooded with people doing task arbitrage and using lots of AI anyway.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
johnsmith1840 4 hours ago [-]
I think yes but on different categories. First one I imagine is robotics control and support.
"This robot is having trouble folding a tshirt help it out for 1$"
Unless they go the waymo route of highly trusted people but I think mass deployed robots are a bit safer than a car for this.
madrox 3 hours ago [-]
This is what I meant by full stack AI companies. I don't think you could get humans into the loop fast enough if they didn't have some idea of the type of task involved. I don't want people to be asked to fold a tshirt one moment and do a difficult traffic merge the next.
its-summertime 2 hours ago [-]
There is training systems and validation of skills in mturk iirc: for tshirt folding, you'd be given fake setups to be able to get used to controlling the robot, if you can't do it, you won't ever get assignments to do it. For traffic overrides, you'd be tested on having correct knowledge, and once again given supervised tasks to show you can actually be trusted (and there would be safety systems, elevating tasks that can't be performed at your level to people who can, etc)
The worker needs to do a bunch to opt into any given work group, which makes the (lack of) payments extremely unreasonable on top of everything else
ntauthority 3 hours ago [-]
warioware shows it can be fun though but i'd indeed not like to see that applied to safety-critical tasks
mikestorrent 3 hours ago [-]
That's a remarkable idea. It could be heavily gamified, it could train models, and it might actually be mentally stimulating since you'd be facing different situations all the time.
Except, I'm a grown adult and I can't fold a t-shirt properly
madrox 3 hours ago [-]
It could require listing your credentials to get you the proper tasks: doctor, lawyer, or tshirt folder
Yokohiii 38 minutes ago [-]
It is weird because the last time I've heard about MTurk was about developing countries being rather reliant on it for doing AI grunt work. If am not totally wrong this must mean that the data work has moved to other services.
SV_BubbleTime 6 minutes ago [-]
> task arbitrage and using lots of AI anyway.
Oh, I remember UpWork.
timcobb 4 hours ago [-]
How could it come around again?
Legend2440 3 hours ago [-]
There are like three dozen companies selling similar services for generating AI training data.
madrox 4 hours ago [-]
I'm unsure. If it does, it will be work that is too expensive or inaccurate or regulatory for current AI methods. For example, you want a doctor to sign off on some AI output on a diagnosis.
However, I'm not sure a single platform will be how it emerges
x0xMaximus 4 hours ago [-]
As AMT's largest requester for the past 10 years, this news was relayed to requesters at the same time as respondents. It's also worth noting that our lead contact, the Sr Program Manager at AWS leading AMT, transitioned to Amazon Bedrock and SageMaker Model Evaluations a ~2-3 years ago.. Leaving behind essential zero team managing the project after they migrated over the stored value accounts to native AWS billing.
jan_Inkepa 4 hours ago [-]
our of curiosity, can you say what your use of it was?
x0xMaximus 4 hours ago [-]
Initially the annotation of biomedical literature (https://pubmed.ncbi.nlm.nih.gov/25592589/) but left academia for commercial use where we transitioned to arbitrage the market research and ephemeral task markets. It was essentially a research panel with a underdeveloped UI for tasks.. so abstract away AMT's tooling so it can be used by any buyer in the ResTech, Political Polling and Data Annotation space
My favorite part of AMT is always going to be figuring out that if we only paid in $0.07 intervals, their commission algorithm would round down to the nearest whole cent when their 20% commission resulted in a fractional cent on the unit transaction level, not the monthly invoice level. Was ultimately worth it to have implemented https://git.generalresearch.com/panels/amt-jb/tree/jb/flow/a...
swyx 2 hours ago [-]
> My favorite part of AMT is always going to be figuring out that if we only paid in $0.07 intervals, their commission algorithm would round down to the nearest whole cent when their 20% commission resulted in a fractional cent on the unit transaction level, not the monthly invoice level.
can someone ELI5 because what in the office space
ramses0 2 hours ago [-]
Go watch Superman III for more details...
owebmaster 2 hours ago [-]
20% of 0.07 was billed as 1 cent, not 1.4 nor rounded up to 2.
x0xMaximus 2 hours ago [-]
1
testbjjl 2 hours ago [-]
It was cheaper and looser automated billing practices than current AI “solutions”?
Buyer A is presented with the following rate card:
< 10min targeting gen pop: $3
10 <= 15min targeting gen pop: $5
Of course they'll bid saying their 14 min survey only takes 9 min to complete. Buyers try to cheat pricing strategies of exchanges just as much as respondents try to cheat buyers on survey platforms. Both parties can't be trusted and have adverse incentives.
muragekibicho 42 minutes ago [-]
Perhaps they're targeting a super specific niche. Like somebody who sees WXET Ephemereal and yelps in triumph "that's my quant!".
sswaner 57 minutes ago [-]
“ General Research combines coordinated operations to identify and neutralize foreign actors using technological advancements, international covert operations, and networking analysis as an Internet Service Provider.”
What is so hard to understand????
rajamaka 3 hours ago [-]
I think it's a satirical website aimed at poking fun at corporate-linkedin-speak
x0xMaximus 2 hours ago [-]
Wish we didn’t even need one
tourist2d 3 hours ago [-]
It smells like they don't know what they're doing. Had a lol at the mission page accidentally implying their customers are unethical though:
> Our Customers
> Ripe with fraud, labor exploitation, political polling manipulation; our customers demand the best tools and security ...
DrewADesign 2 hours ago [-]
At least they’re ripe! I hate unripe customers, rife with astringency, sharp vegetal notes, and an unpleasant, unyielding texture.
m3rc 3 hours ago [-]
What's up with the part of the website where you are talking about "neutralized foreign actors", was that also using contracted Mechanical Turk labor?
It should be a bipartisan issue that a Swedish company is paying a UAE Residential Proxy company [1] to build tools that allow people from anywhere in the world to take US political polls that are used by both parties to collect election data.
Just for starters, you can't even think about soliciting online work from a panel without a robust residential proxy detection methods. We had to build our own:
Polling data isn't some guaranteed right or something. If you ask the Internet questions and then sell the collated answers, it's your job to authenticate the responses or your polls will be wildly wrong and people will stop buying them.
Which is already the case and nobody but a bunch of campaign contractors who can't justify their do nothing jobs anymore cares.
schoen 2 hours ago [-]
I just remembered that in the story "Simulacron-3", there is ostensibly a law requiring people to respond to opinion polls. It is illegal to refuse.
Someone using an IP that a geoip database says is US, doesn't mean that person is a US voter. The existence of proxies on the internet is a feature.
x0xMaximus 2 hours ago [-]
Obviously, a problem not even fully addressed by L2’s datasets. Sure, but the use of proxies to masquerade the identity of users being sold to researcher buyers that paid for a different service is fraud
charcircuit 39 minutes ago [-]
It makes me question why the surveys don't ask for the users country. That way they cd can figure it out without trying to guess it based off of IP.
x0xMaximus 16 minutes ago [-]
Users/respondents lie, and many buyers/researchers are neo-luddites; most C level still come from the telemarketing days. A fun example: mobile targeting is terrible when off wifi because all survey platforms uniq identify users based off their IPv4, so any T-Mobile LTE users in the same city going through the same CGNAT often share profiling data that ends up conflicting, which ends up meaning they don't get sent into the best survey(s). The issue is even worse in heavy IPv6 countries like India+France.
3 hours ago [-]
BobbyTables2 1 hours ago [-]
That’s top-tier “OfficeSpace” thinking there…
testbjjl 2 hours ago [-]
Sounds like your program manager was smart and valuable to the company. Do you have any plans to replace the service/data provided by MTurk?
shortformblog 3 hours ago [-]
So, I have a story to share about Mechanical Turk that you might find interesting. I’ve shared it a couple of times before on Twitter and Bluesky, but I’ll share it again.
Long story short: Mechanical Turk saved my bacon.
Back in 2005, I was working a job at a small-town newspaper in a town I’d never lived before. I didn’t know anyone beyond the staff (as I was working layout rather than as a reporter), and I had gotten interested in the idea of doing Mechanical Turk for a few extra bucks. I thought there might be a formative scene of interested people doing this, so I started working on a blog for it. I briefly collaborated on it with another guy who put it on forum software because he didn’t know how to use, like Drupal.
That blog was called Turking.com, and it seemed like it was going well for a bit. We even got a mention on the AWS website. But after about two or three months, it was clear the initial excitement around the idea (and the initial work) had died down. (The work picked up later, but in clearly different ways. I don’t think folks were really “excited” about it after that point.)
The site had started to die out in part because my iBook suffered a catastrophic GPU failure and me, being a broke small-town newspaper employee, did not have the money to replace it. Plus, the community just hadn’t emerged like we had expected.
But then I got an email out of the blue: Someone wanted to buy my domain, which I owned outright, but they didn’t want to say who. I got contacted by a broker, and the exchange took place over escrow.
(The domain most assuredly was bought by Amazon and is managed by MarkMonitor.)
I didn’t get a ton of money from the deal, but I did get enough to pay for a new laptop. I had some regrets about selling it (in part because I originally built the site with someone else), but I was in a bit of a dire financial situation which that proved to be the starting point for getting myself out of.
That domain purchase was notably more than I ever made from clicking and classifying random pictures, that said.
idiotsecant 3 hours ago [-]
So did the other person just get hosed when you did your 'ipo'?
shortformblog 3 hours ago [-]
I explained the situation after the fact, and they were understanding. But certainly I admit that if I could do it again I probably would have clued them in sooner. The site had slowed down by the point this happened, so I’d describe it as a little more of a fire sale.
I learned a lot from that situation that I took to future sites.
conception 4 hours ago [-]
It’s kind of crazy they’re shutting this down just when this Service probably has the most possibilities ever. You have an agent with Multiple people doing actual physical tasks in the real world seems like something that could be really powerful.
paxys 3 hours ago [-]
The concept behind it isn't going away. There are plenty of companies that hire people in bulk to do manual data labeling, transcription, RLHF, moderation and lots more. It's just that they are now catering to large AI companies, not regular people looking to get some repetitive work done (since that can now mostly be done by AI).
robot_jesus 4 hours ago [-]
Right? Not to mention the training/tuning possibilities.
Mercor has a market value of $20B basically doing the same thing but desperately trying to find workers.
I did this a long time ago and it actually paid for a few meals.
They are terribly monotonous tasks.
I can see why it's shutting down if its still the same thing.
Lllms could probably do everything there without rotting out minds for basically pennies.
Forgeties79 3 hours ago [-]
Instead we pay to rot our minds lol
shermantanktop 2 hours ago [-]
The tagline for MTurk used to be “artificial artificial intelligence.”
It appears now we can get along with just a single “artificial.”
mturk 3 hours ago [-]
I'll still be around on Oct 1.
rickcarlino 1 hours ago [-]
Wishing you the best of luck on the road ahead ;-)
rcr-anti 3 hours ago [-]
Had some absolutely bizarre results from their attempt to integrate mturk with Bedrock's 'ground truth' thing as of a few months ago. Threw simple mnist digits recognition at it, see what the quality, timing, and cost was. Figured mnist digits was at this point trivial. Spent $10 and the accuracy was marginally better than guessing, completely unusable results. Was completely baffled, people were publishing peer reviewed research based on exclusively mturk results.
transitorykris 1 hours ago [-]
The only time I ever used it (as a turk?) was when Jim Gray went missing at sea and satellite imagery of vast regions of the pacific were fed through Mechanical Turk. Seems a trivial problem now, but was not then, and needed humans to take a look.
mips_avatar 3 hours ago [-]
Surprised Amazon managed to fumble mechanical turk at the same time Mercor/Scale and all these other companies started hiring humans to do data labeling tasks.
cautiouscat 4 hours ago [-]
I was literally wondering this morning if mturk was still around! I assumed a lot of the stuff could just be done with LLMs and OCI now.
wanderingstan 3 hours ago [-]
I used it to have people transcribe my dad’s handwritten letters and journals. It was touching to get notes from a “Turk” saying how much she enjoyed his travels and following the cast of characters in his life!
wileydragonfly 3 hours ago [-]
I made a couple thousand dollars off it squeezing in tasks here and there between meetings. Amazon Payments were challenging to redeem as I recall. I did more or less write the bulk of someone’s doctoral thesis. Kept giving me a dollar to summarize the findings of various psychology papers. The most memorable was the one where people were put in a room and someone sprayed “liquid ass” on the wall. The subjects given no explanation experienced higher levels of anxiety than those that were told there was a sewage leak being repaired. One of the strangest dollars I ever made. My PS4 and game collection was spectacular.
willmeyers 3 hours ago [-]
I wouldn't be surprised if Fiverr and Upwork start pulling back too. There's just so much AI bots now that have flooded the task marketplace.
aboardRat4 4 hours ago [-]
End of an era.
tibbon 3 hours ago [-]
Humans were once useful!
2 hours ago [-]
qarl2 4 hours ago [-]
I wonder why.
xyst 3 hours ago [-]
bezos has determined LLM has replaced "on demand 24/7 workforce"
PrimeAli 27 minutes ago [-]
[flagged]
retr0rocket 2 hours ago [-]
[dead]
OkWellBye 4 hours ago [-]
[flagged]
Rendered at 04:16:15 GMT+0000 (Coordinated Universal Time) with Vercel.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
"This robot is having trouble folding a tshirt help it out for 1$"
Unless they go the waymo route of highly trusted people but I think mass deployed robots are a bit safer than a car for this.
The worker needs to do a bunch to opt into any given work group, which makes the (lack of) payments extremely unreasonable on top of everything else
Except, I'm a grown adult and I can't fold a t-shirt properly
Oh, I remember UpWork.
However, I'm not sure a single platform will be how it emerges
My favorite part of AMT is always going to be figuring out that if we only paid in $0.07 intervals, their commission algorithm would round down to the nearest whole cent when their 20% commission resulted in a fractional cent on the unit transaction level, not the monthly invoice level. Was ultimately worth it to have implemented https://git.generalresearch.com/panels/amt-jb/tree/jb/flow/a...
can someone ELI5 because what in the office space
https://www.instagram.com/p/DZacltqHMjT/?img_index=8&igsi=bT...
> Ripe with fraud, labor exploitation, political polling manipulation
I am not sure if they want to convey what I think I am reading or not.
[0] https://generalresearch.com/mission/#:~:text=Our%20Customers
< 10min targeting gen pop: $3 10 <= 15min targeting gen pop: $5
Of course they'll bid saying their 14 min survey only takes 9 min to complete. Buyers try to cheat pricing strategies of exchanges just as much as respondents try to cheat buyers on survey platforms. Both parties can't be trusted and have adverse incentives.
What is so hard to understand????
> Our Customers
> Ripe with fraud, labor exploitation, political polling manipulation; our customers demand the best tools and security ...
It should be a bipartisan issue that a Swedish company is paying a UAE Residential Proxy company [1] to build tools that allow people from anywhere in the world to take US political polls that are used by both parties to collect election data.
Just for starters, you can't even think about soliciting online work from a panel without a robust residential proxy detection methods. We had to build our own:
`wget -N 'https://grip.net/files/grip-proxy-30d.mmdb'`
[1] https://www.youtube.com/watch?v=eOmeQcwSK3o flagged by Nokia Deepfield and CTRL for it's involvement in botnets
Which is already the case and nobody but a bunch of campaign contractors who can't justify their do nothing jobs anymore cares.
https://en.wikipedia.org/wiki/Simulacron-3
Spoiler: gur jbeyq va juvpu gur ynj fhccbfrqyl rkvfgf vf n fvzhyngvba, perngrq ol be sbe cbyyfgref!
Long story short: Mechanical Turk saved my bacon.
Back in 2005, I was working a job at a small-town newspaper in a town I’d never lived before. I didn’t know anyone beyond the staff (as I was working layout rather than as a reporter), and I had gotten interested in the idea of doing Mechanical Turk for a few extra bucks. I thought there might be a formative scene of interested people doing this, so I started working on a blog for it. I briefly collaborated on it with another guy who put it on forum software because he didn’t know how to use, like Drupal.
That blog was called Turking.com, and it seemed like it was going well for a bit. We even got a mention on the AWS website. But after about two or three months, it was clear the initial excitement around the idea (and the initial work) had died down. (The work picked up later, but in clearly different ways. I don’t think folks were really “excited” about it after that point.)
If you want to get an idea of it, there was one capture on the Wayback Machine: https://web.archive.org/web/20051124231722/http://www.turkin...
The site had started to die out in part because my iBook suffered a catastrophic GPU failure and me, being a broke small-town newspaper employee, did not have the money to replace it. Plus, the community just hadn’t emerged like we had expected.
But then I got an email out of the blue: Someone wanted to buy my domain, which I owned outright, but they didn’t want to say who. I got contacted by a broker, and the exchange took place over escrow.
(The domain most assuredly was bought by Amazon and is managed by MarkMonitor.)
I didn’t get a ton of money from the deal, but I did get enough to pay for a new laptop. I had some regrets about selling it (in part because I originally built the site with someone else), but I was in a bit of a dire financial situation which that proved to be the starting point for getting myself out of.
That domain purchase was notably more than I ever made from clicking and classifying random pictures, that said.
I learned a lot from that situation that I took to future sites.
Mercor has a market value of $20B basically doing the same thing but desperately trying to find workers.
They are terribly monotonous tasks.
I can see why it's shutting down if its still the same thing.
Lllms could probably do everything there without rotting out minds for basically pennies.
It appears now we can get along with just a single “artificial.”