Crowd Sourcing Could Be a Dangerous Way to Measure

standout from crowd photo
Image by Keoni Cabral

Hmm it was just last week that I expressed some concerns over Facebook’s plan to start flagging false information using a simple crowd sourced method; If enough people flag it, it must be false, right?

Then I discovered something even scarier to think about from this story:
How a Lone Hacker Shredded the Myth of Crowdsourcing

Now the story is about a 2011 DARPA contest, and a team that tried to use crowd sourcing as a way to solve the puzzles set before them, and the competitor who subsequently killed their chances by messing up the work the other users in the crowd had accomplished. It’s interesting, but if you want to just get to my point, it’s in the final paragraphs of the story:

“Myself and others in the social sciences community tend to think of such massive acts of sabotage as anomalies, but are they?” wondered Cebrian. To settle the question, Cebrian analyzed his (and other) crowdsourcing contests with the help of Victor Naroditskiy, a game theory expert at the University of Southampton. The results shocked him. “The expected outcome is for everyone to attack, regardless of how difficult an attack is,” says Cebrian. “It is actually rational for the crowd to be malicious, especially in a competition environment. And I can’t think of any engineering or game theoretic or economic incentive to stop it.”

“adversarial attacks are effective against all machine learning algorithms, and coordinated attacks are particularly effective.”

What should we take away from this? Well let’s think about how Facebook decides what to show in your newsfeed, or how Twitter decides what to show you in it’s new “What you missed” feature? It’s all based on algorithms. The competition for attention is huge on social media, and continues to get more and more competitive. We’ve seen attempts at gaining that attention hijacked repeatedly. Look no further than Coca Cola’s Super Bowl #MakeItHappy campaign. At first it was met with the typical snark:

If you’ve been around social media at all, you know to expect the snark, right? But then it got worse. The campaign, a machine algorithm, was actually gamed by Gawker, leading to it’s shutdown. Why? For kicks and giggles mostly, but also to steal some of the attention away from the effort. If you’re working for Coke, how do you protect against that? Well, you really can’t in any realistic sense. It’s the price of doing business in social media.

Social media metrics are all based on one thing, and one thing only. What the crowd has to say. The more people who retweet, like, comment, etc. the more the algorithms will recognize your content as “good”, and the more it will be displayed to users. But once something starts to garner attention, there’s an incentive to attack it, in order to force it to lose the share of attention that it currently has. Thus popular hashtags start to pop up in completely unrelated statuses, automated systems get bombarded with attempts to alter the outgoing messages in negative ways, honest attempts at sharing stories or answering questions get hijacked by groups with political agendas, and so on.

Look, that’s the way social media works, and there is no easy way to prevent it. The beauty of using social is that we can reach a large audience with authentic messages and ideas, because people will share them with their own networks. The downside is that this open communication tool is available to those who would use it to attack people, brands, or anyone in order to draw attention away from one group and onto themselves. Systems that rely heavily on crowd sourcing as measurement are especially vulnerable to this behavior. So what should we do about it?

Well, first thing is to always take the algorithm with a grain of salt. Just because the “crowd” says something is quality content, doesn’t necessarily mean it is. It could be content that is being gamed with a coordinated effort to make it appear popular.

In that same theme, also be careful with who you follow. If you are depending on the crowd to bubble up good content for you, don’t follow people who are bubbling up content that doesn’t fit your description of quality content. They’re screwing up the algorithm for you.

Lastly, be on the lookout for those who would try and game, or attack, crowd sourced algorithms. Google has been doing a good job of finding, and punishing, sites trying to game it’s algorithm with fake links, stolen content, etc. but that hasn’t stopped people from trying. Don’t be surprised when someone tries to hijack your hashtag, or a popular comment thread, for their own purposes.

The truth is, once you put something out there in an attempt to attract engagement, you’ve lost control of it. Hopefully, people will engage with you, but some of those people may be “engaging” for nefarious reasons. It’s a never ending battle for the small attention span of most social media users, so don’t be surprised if it gets a little cutthroat.

Similar Posts

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.