From 5% to 48%: learnings from another industry’s progress in diversity
Unconscious biases have a surprisingly high impact on how we evaluate candidates for a job position. Here are a few best practices for making the recruiting process more impartial and transparent.
con·fir·ma·tion bi·as: the tendency to interpret new evidence as confirmation of one’s existing beliefs or theories.
Few highly skilled industries can say that 48% of their top talent are women, but there is one notable exception. At top tech companies women only make up only about 15% of technical staff, while Black and Hispanic employees together account for less than 10%. This is a sore reality that hurts companies, their customers and their bottom line.
Improving the pipeline and diversity training won’t fix the diversity problem at tech companies. Instead, we should look to a better historical guide for a better solution.
That’s where the orchestra comes in.
Orchestras were historically male dominated. Since then they renovated how they hire to be more diverse. Gender diversity now has a more equitable distribution, with 48% of women. So the question arises: what does research about the changes in the orchestra tell us about the underlying causes and how can we can properly tailor a solution?
Root cause analysis
Confirmation Bias is the core cause for lack of diversity. It starts with how candidates are screened and persists through the entire hiring process. The simplest explication for Confirmation Bias is “if a candidate is similar to me, they must be good”.
Confirmation Bias is rarely done on purpose. It’s an Unconscious Bias. It can take the form of criticizing a candidate’s schooling, how they deconstruct problems or using their past work as a proxy for ability.
When a good candidate works out, we conclude it’s because the hiring process worked well. This reinforces our biases in a self-defeating feedback loop and has is laden with little to no quantifiable metrics.
The key questions to ask are “Did you ever once stop to evaluate the counterfactuals? Of all the people excluded, how would they have worked out if they were given the chance to succeed?”
We have literally answered those questions in my team, and the result is that they worked out amazingly well.
The fix
How do we remove biases if by their nature we aren’t consciously aware of them?
The orchestra solved this in the 80’s by excluding out all extraneous factors except merit. They exclude schooling, past work experience; everything except for asking “can they do the job?”.
The process doesn’t start with a resume.
It starts by asking candidates to take the same audition.
But that doesn’t go far enough. To remove unconscious race and gender bias they ask the candidates to perform behind a screen so that the judges don’t have the possibility of introducing false signals into their assessment. They even went so far as to carpet the stage, since women walking in heels on a wooden stage could easily be picked out and reintroduced bias.
With all external factors removed, a panel would rank and have a quantifiable consensus of who does the job best. Hiring on only merit gives everyone equal footing and let’s the best of us shine.
Quantifiable hiring
How can we apply the orchestra’s blind performance auditions to the tech industry? We have a working model in my team that I’d like to share.
What we’ve done in our engineering group is to first remove Confirmation Bias from our testing. Things like evaluating big-O and algorithms only confirm candidates have taken a sophomore level CS course — but when building fast, useful software they rarely come into play.
Whiteboarding is also a poor signal of ability. Its intent was to assess they have a logical thought processes, but not everyone performs well in this environment. Also, whiteboarding is not a requirement in order to write good code.
Here’s the first table flip: we do not screen candidates on resumes.
Recruiters shouldn’t be gatekeepers. Instead, our screening process is an audition. In our case it is a custom exam. The exam provides better signals about a person’s merit. It takes our engineers about as long to assess as it does to read a resume. Our exam sets out to model what daily work is like at our company. We create the exam by setting key goals. Ours are: 1. can they write logical code and 2. can they learn new things quickly.
Prior to the exam, prep the candidate. Give them the proper expectation and let them prepare. When they feel ready, they can come back and take the audition. The exam is a small code base where we ask the candidate to build a few small set of features.
In order to gauge if the candidate can learn, we include some proprietary tool they haven’t used to see how they adapt to it. We purposely do not include stupid gotchas, like asking a web developer to find the distance between two points. It’s not rocket science, but how many times does a web developer do canvas drawing, for example? Everyone has gaps (and strengths) in their knowledge. Our process is build to encourage and discover those differences.
The last key feature of our exam is that it is a fixed 2 hour test.
Junior developers won’t finish on time, and that’s fine. Senior engineers will shine brighter and we’ll see what choices they make in the time given. Having a short time limit also removes additional age or environment biases. People taking care of kids might not be able to devote an entire weekend to take our test. We also run it remotely so their current distance from our office is not a factor on their ability to do the job. We have a proctor available over chat to answer any questions because speaking up when you’re stuck is also a great skill for candidates.
Having a short time limit also removes additional age or environment biases. People taking care of kids might not be able to devote an entire weekend to take our test.
Once the exam is complete, we have a panel of our engineers score them on predefined criteria. This scoring is also great because it gives us a baseline for candidates. If and when they fall out, we can look for statically variances as an early warning sign to improve our process.
When a candidate passes the audition, we offer a 30-minute video chat with the candidate to explain their code to us. We are not looking to see that their thought process matches our own. We only want to see that they can explain why they made their choices and how deep and wide their knowledge is.
The last stage of hiring is a more traditional on-site interview with a few people. The big difference with our interviews is that we already know they are perfectly qualified to do the job. This puts more power with the candidate because they already made a great first impression.
The other legs of the stool
There are two other topics worth touching on quickly that feed into creating a good blind audition process: inclusion and screening.
The blind performance audition IS the screening process — not reading resumes.
That’s a touchy topic for recruiters. In this process their role effectually pivots from a gate keeper to an evangelist. They should be reaching out to organizations that help underrepresented minorities and persuading people on why our company is a great place for them to bring their skills. Inclusion is also massively important in retaining diverse, happy, and productive employees. Before retooling hiring, start by addressing potential inclusion problems by having an external group of experts audit your company.
Fixing the screening, evaluation and inclusion are the vectors that will make rapid and noticeable improvements in diversity.
A more diverse workforce leads to higher sales and restores companies to being better stewards of their local communities. Our process was around Tech, but a similar blind performance audition can be applied to design as well. I’ll leave you with some cliff notes to help your company to take advantage of amazing and diverse people in your community.
The short list for inclusion & diversity in your recruiting process
- Seek outside experts to assess and improve your office’s inclusion before anything else.
- Flip recruiting from reading resumes to evangelizing and outreach.
- Create a blind performance audition that focuses on only the core skills while eliminating all other factors that breed bias.
- Continually quantify and normalize how the hiring panel rates the exam.
- Use analytics from the blind performance audition to follow dropouts and candidates through the process to look for disconnects and reinforce the system.
Research Appendix
- Twitter diversity report
- Facebook diversity report
- Apple diversity report
- Harvard blind auctions study
- Orchestra diversity reports
See you in the next story.
