Interview with Jesse Maddox, CEO and Co-Founder of Trip Lingo
Today I spoke with Jesse Maddox, CEO and co-founder of Trip Lingo, an app that provides practical colloquialisms in a variety of languages, maximizing people’s immersion in their travels by minimizing the language barrier.
Jesse and his team in Atlanta use Mechanical Turk, as needed to fully realize app-enhancing ideas. Jesse explains, “there are a lot of cool things you can do with mTurk if you try to be creative.
For 25 cents a HIT, Turkers thought up mnemonic devices for 1000 different French phrases. When Jesse decided to include film clips in which actors speak a foreign language, he asked Turkers to brainstorm films and find the clips. He tries to share the mission behind the HITs with the Turkers so they understand how they are adding value to his company. Jesse has been satisfied with the results.
Jesse found out about mTurk in the news and has used it off and on for a few years. He has tried oDesk, but is partial to Elance when he needs workers with a specific skillset.
Trip Lingo puts work through mTurk template builder, but Jesse notes that the system is a bit complicated; he spent a decent amount of time training his intern to use it. Jesse thinks that the difficulty of using mTurk might steer people from crowdsourcing’s practicality.
Ultimately, Jesse and his team don’t use mTurk for anything that’s “super mission critical”—for example, translations for Trip Lingo—but explains that “if you define something in the right way and set it up correctly, you can get good results.” As a periodic user of mTurk, Jesse has been satisfied with the results. He believes in crowdsourcing and considers it an underutilized tool.
Name: Jesse Maddox, Co-Founder and CEO of Trip Lingo
Crowdsourcing Marketplace: Amazon’s Mechanical Turk (mTurk)
# of HITs/Month: as needed
Price per HIT: ~25 cents
Interview with Michael Bauer, CEO of Brilliant Arc
Last Friday, I spoke with Michael Bauer, one of the cofounders of MapQuest and current CEO of Brilliant Arc, classification / categorization software. Michael uses Amazon’s mTurk for his crowdsourcing needs; Amazon is a name he trusts.
What kind of tasks does Michael send through mTurk? Say an interior design firm needs additional information about a series of images of houses. Brilliant Arc has a “taxonomy software system” that allows the firm to input information on how they want to classify each image — kitchens, bathrooms, bedrooms — and then input what features they want recognized in each room — is there a stove, french doors, or a wine rack?
As Michael so aptly stated, “our clients hire us — and through us Mechanical Turkers — to do this job because computer systems aren’t good at this kind of work. People are.”
He and his company have been mostly satisfied with mTurk. Work is completed almost as soon as he assigns it. It’s cheap. In addition, he is heavily invested in their API not only because it was time consuming to build, but also because as time has progressed, his system has learned which Turkers to trust.
Brilliant Arc spends a lot of time managing Turkers. His staff reviews turk-work in house, which controls quality before delivering to clients, but also keeps an eye out for bad workers. Michael notes that some Turkers try to game the system.
Originally, Michael would ‘block’ anyone who submitted wrong answers, but Turkers got angry about this way of ensuring quality and badmouthed Brilliant Arc on their worker forum, TurkerNation. Turkers take the opinions of other Turkers on this forum seriously. If a company gets a bad reputation as a work requester, no one will want to work for it.
Now, rather than block bad workers, Michael’s team works in a positive reward system, bonusing good workers and auto-approving their HITs. He emphasizes how much he appreciated the feedback from Turkers following the TurkerNation uproar, “like anyone just starting out on anything, we made some mistakes that failed to appreciate how passionately the core workers take their work…we hope we’ll continue to grow and make them appreciate working with us going forward.”
Even though Brilliant Arc has made mTurk work, Michael says his company would pay at least 2x more and accept a longer turnaround time on crowdsourcing that requires less management. Michael hasn’t demanded more of his crowdsourcing marketplace because he doesn’t know of a better available option. Ideally, he wouldn’t have to review the HITs in house, nor would he have to worry that he was compromising his entire workforce by blocking bad workers from his system.
He’s considering posting an additional task on mTurk. Some of the photos given to him by his clients are not good enough quality for the workers to identify at all, so he might need a first round that simply asks workers if the photo is good enough quality to tag. Perhaps it would be easier if there was an option on the image tagging photos along the lines of “cannot tag” which would send the photo back to Michael. He hasn’t done this yet, assumedly because it will take time and effort to write that into the API.
Name: Michael Bauer, CEO of Brilliant Arc
Crowdsourcing Marketplace: Amazon’s Mechanical Turk (mTurk)
Type of HIT: Image Classification
# of HITs/Month: over 10,000
Price per HIT: ~5 cents
Ghost in the Machine
Hi, we’re Office Llama, a blog devoted to understanding crowdsourcing from the perspectives of workers, clients, and fellow students of this work revolution.
Here’s some background on crowdsourcing:
Since Henry Ford’s invention of the assembly line, work has been chopped into tiny pieces and doled out to workers, who, in a mechanized fashion, repeat a specialized task over and over. Unskilled workers are cheaper and require less oversight and training.
Crowdsourcing is the online assembly line. Big projects that require human intelligence are broken down into small, specific tasks and completed for a few cents or more each. These tasks are accepted or rejected by the system in place by those requesting the work, which determines whether workers receive their pay.
Here are some examples of common tasks assigned to crowdsource workers:
- choose the best category for this term
- page classification
- are these two pictures of the same kind of place?
- categorize products
- write a title for a short excerpt of text
- find contact info
- copy text from a business card (or anything)
- how relevant are these results?
- identify if people are looking at the camera
I will elaborate on these tasks as I speak with and learn about the companies who use crowdsourcing.
Return of the Human Computers
Summer 1937. Recovery from Depression had stalled and American government officials had stimulus money to spend but, with winter looming, there were few construction projects to fund. So the officials created office posts instead. One project was assigned to a floor of a dusty old New York industrial building, not far from Times Square. It would eventually house 300 computers—humans, not machines.
The computers crunched through the calculations necessary to create mathematical tables, then an indispensable reference tool for many scientists. The calculations were complex and the computers, drawn largely from the ranks of New York’s poor, possessed only basic numeracy. So the mathematicians in charge of the project worked out how to break each calculation down into simple operations, the outcomes of which could be combined to give a final result.
Until recently, that is. Over the past few years, human computing has been reborn. The new generation of human computers carry out different tasks, but they mirror their predecessors in many other ways. They are being drafted in to perform tasks that computers cannot. They are employed in large numbers and are organised into streamlined workflows. And, as was the case in the age before electronic computers, their output is combined to generate results that could not easily be produced in any other way.It was a technique that had been employed for decades across America and Europe. The field of human computing even had its own journal and trade-union representation. Computing offices calculated ballistics trajectories, processed census statistics and charted the course of comets. They would continue to do so until the 1960s, when electronic computers became cheap enough to consign the profession to history.
In one proof-of-principle experiment, published earlier this year, human computers were used to create encyclopedia entries. Like performing mathematical calculations, this is a skilled job, but one that can be broken down into simpler parts, such as initial research, writing and editing. Aniket Kittur and colleagues at Carnegie Mellon University in Pittsburgh, Pennsylvania created software, known as CrowdForge, that manages the process. It hands out tasks to online workers, which it contacts via Mechanical Turk, an outsourcing website run by Amazon. The workers send their work back to CrowdForge, which combines their output to produce surprisingly readable results.
Several American start-ups are operating similar workflows. CastingWords breaks audio files down into five-minute segments and farms each out to a transcriber. Each transcription is automatically bounced back to other workers for checking and, once deemed good enough, an (electronic) computer combines the segments and returns the finished product to the customer. At CloudCrowd a similar system is used to co-ordinate teams of human translators. Others are combining human and artificial intelligences. An app called oMoby, produced by IQ Engines, can identify objects in images snapped by iPhone users. First it applies object-recognition software, which may not be able to cope if the lighting is poor or the image was captured from an unusual angle. When that happens, the image is sent to a human analyst. Either way, the user gets an answer in half a minute or so.
Much more is to come. In old-fashioned computing offices, workflows were co-ordinated by senior staff, often mathematicians, who had worked out how to deconstruct the complex calculations the computers were tackling. Now silicon foremen such as San Francisco-based Humanoid oversee human computers. These algorithms, which co-ordinate workers by plugging into Mechanical Turk and other online piecework platforms, are relatively new and are likely to get considerably more sophisticated. Researchers are, for example, creating software to make it easier to assign tasks to workers—or, to put it another way, to program humans.
Eric Horvitz, a researcher at Microsoft’s research labs in Redmond, Washington, has considered how such software could be put to use. He imagines a future in which algorithms co-ordinate an army of human workers, physical sensors and conventional computers. In the event of a child going missing, for example, an algorithm might assign some volunteers to search duties and ask others to examine CCTV footage for sightings. The system would also trawl local news reports for similar cases. These elements would be combined to create a cyborg detective.
This sounds terribly futuristic, and rather different to the pen-and-paper human computation of the 19th century. But David Alan Grier, a historian of computing at George Washington University in Washington, DC, thinks that the architects of the new systems could learn a lot by studying the old ones. He points out that Charles Babbage, the designer of an early mechanical computer, gave much thought to reducing the errors that human computers made. Babbage realised that duplicating tasks and comparing the results was not enough, because different workers tended to make the same mistakes. A better solution was to find different ways to perform the same calculation. If two methods produce the same answer, the result is much less likely to be flawed, Babbage reasoned.
There are many more such useful tips in the historical record, says Dr Grier. Human-computing pioneers also wrote a lot about how best to break a complex calculation into sub-tasks that are completely independent of each other, for example. “There are all sorts of hints in the old literature about what’s useful,” he says. He is often invited to human-computing conferences at which he likes to chide researchers for overlooking such lessons from this forgotten but intriguing early chapter of computer history.
(Source: economist.com)
