Welcome to our first bias deep-dive at Wordnerds. Any data scientist worth their “Kiss Me I’m a Data Scientist” novelty baseball cap knows that bias is an enormous challenge in big data, Artificial Intelligence, and beyond.
So this week, we’re going to talk about how Wordnerds has minimised the inherent bias in Artificial Intelligence, caused by an inherent bias in human society.
Wow, that last sentence requires some unpacking. It’s making three colossal assumptions, which we really need to talk about in turn. Those assumptions are:
- There is bias in human society.
- That bias is reflected in AI.
- Something can be done to fix the inherent bias in AI
In the coming weeks we’re publishing three articles taking each of these challenges in turn. And where better to start on our quest to understand bias than the gloriously resurgent, but occasionally maligned bastion of Britishness, Strictly Come Dancing?
How could we judge whether there is Strictly bias?
The tricky thing about trying to establish whether there’s bias in Strictly is deciding whether someone leaving the show is fair or not. Most of the articles that talk about Strictly bias focus on either which contestants leave strictly, or which contestants win.
But how do you judge whether somebody deserved their Strictly fate? Maybe they were just the worst, or the best, dancer?
As a dancer with all the grace and panache of an elephant trapped in a toilet cubicle, I’ve only got one way of understanding how good a dancer is: the scores they were given by the Strictly judges.
So we compared the scores given by the judges, to the votes given by the audience. Now the votes aren’t revealed by the BBC, so all we have to go on is who ended up in the bottom two. So, we’ve based how celebrities were treated by the Strictly audience on the number of times they were:
- Saved. The judges placed the dancer in the bottom two, and the audience took them out.
- Dropped. The judges placed the dancer outside the bottom two, and the audience put them in.
We’ve counted up the times each couple was saved or dropped. If you’re saved, you get a point, if you’re dropped, you lose a point. If you’re joint second bottom with another dancer, that counts as half a save or half a drop.
Got that? Good. This is our way of trying to establish bias in Strictly. Now let’s talk about all the problems with it.
Limitations of this way of doing things:
- In order to be saved a lot, you have to be put in the bottom two by the judges. In other words, you’re more likely to be saved a lot if you’re a bad dancer. So you expect worse dancers to do better.
- This way assumes that the judges are completely unbiased. So we don’t take into account strange decisions by the judges. If we did, that would bring in another set of subjective opinions and therefore biases.
- There have been a total of 264 couples on Strictly prior to this year. This isn’t an enormous sample size, especially when we’re breaking down by different demographic groups.
But with those caveats firmly in mind, let’s take a look at who gets saved by the Great British Public, and who they’re happy to drop.
Most saved Celebrities in strictly history
Most saved celebrities. “Age” is celebrity age at the start of their series
It gets political at the top of the table, with professional politicians and decidedly amateur dancers Ed Balls and Ann Widecombe being saved eight times in their respective series. Last year’s loveable bandwagon Dan Walker barely scrapes into the all-time top 10.
- As you might expect, the top of the saved chart trends older. The average age of all strictly celebs is a shade over 40. The average age of the top ten most saved contestants is over 48.
- Interesting to see an exact split between men and women in the top ten saved.
- What’s really stark, though, is how the most saved strictly contestants are split by race. In that, they aren’t. Over 1 in 5 Strictly contestants have been BME, but they make up 0% of the most saved contestants. The most saved people of colour were Alex Scott and Martin Offiah, both saved twice. 24 white contestants are higher in the table than them.
Most Dropped Celebrities in Strictly History
Most dropped celebrities. “Age” is celebrity age at the start of their series
- The bottom ten contestants average slightly below the average age overall, at exactly 38
- Women outnumber men eight to six in the slightly elongated (due to five contestants being tied on 2.5) top ten.
- Again though, the most telling stat is around race. 21% of the contestants in Strictly History have been people of colour, but they make up 42% of the all-time most dropped contestants.
Overall Stats - Saves and Drops by demographic group
We then compared all the different demographic groups we’ve discussed so far, also combining race and gender, to get a sense of the demographic qualities that the audience seemed to get behind. Based on what you’ve read so far, you’re not going to be shocked by what comes next:
- White males are by far the most saved demographic group. 40% were saved at some point in a series, with less than a quarter dropped. Going into Strictly as a white male gives you, on average, half a save.
- While men are more likely to be saved than women, BME men have a lower overall score than BME women. Only 16% of BME men were ever saved, and well over half were dropped at some point in their strictly career.
- No BME contestant has ever been saved more than twice. It’s happened to nearly 15% of white men.
So what does this mean? That Strictly is entirely watched by bigots? Obviously not, and that’s the point. Even if there is a very, very slight subconscious bias in a system, if you run that system over and over again (say, through millions of strictly votes), the bias can end up making massive, material differences.
Which brings us to AI. Natural language processing is built upon, is trained by throwing millions upon millions of sentences through a neural network. Those sentences were written by people. Next time, we’ll look at how the bias in society can create bias in AI.
