Which type of pre-onsite technical interview is the best for recruiting engineers?
Timed coding challenges, phone screens, and take home assignments; which do developers actually prefer and why?
Like many engineers, I’ve had a career full of horrible interviewing experiences, both as a candidate and an interviewer. As a candidate, I’ve been told to write a distributed system implemented using mutexes on a whiteboard. I wrote it in pseudo code to a silent and judgmental interviewer, and then was told I failed the interview because I didn’t write the name for the Java mutex package correctly. Java was not listed on my resumé and I had never written it professionally.
As an interviewer, I interviewed a person whose skillset only included writing HTML and CSS, but who was convinced they were deserving of a senior engineering title. I gave them a hard pass, but I was overruled. Then I was told I had to be their onboarding partner and mentor for the next few months.
Many years of facepalm-worthy interview experiences would turn most people off from wanting to ever interview on either side of the table ever again. Instead, it awoke the part of my brain that loves solving user problems. By all measures, interview experience problems are user experience problems.
At my last full-time position, I got heavily involved with technical recruiting in hopes to improve the experience for everyone. I worked with my colleagues to design and administer interview slates, and created my own inner philosophy of how technical interviewing should be done. Now I’m on a quest to codify good practices, and what better way to start a redesign of the technical interview experience than with the first step in the user experience design process: User research.
Thus, I started by running a few polls on LinkedIn and Reddit, asking web developers and SWEs¹ what their preferred pre-onsite interviewing experiences were.
Pre-onsite Experience Preferences
Quantitative poll data
Generally, when it came to the pre-onsite experience, most people preferred a casual chat about their past experiences over technical interviews. Some companies find those sufficient to move on, but most would like to see some demonstrable technical skills before leading the candidate on to the coveted, expensive 5+ hour day experience. So, I asked people what technical interview type they preferred:
In case there was a bias toward people who like projects on Reddit, I also ran the same poll on LinkedIn:
Seems like an easy conclusion, doesn’t it?
So, should we stop here and conclude that we should all be solely doing take-home projects? No! 64% of people preferring take-home projects from this data still leaves a pretty sizable 36% who do not prefer it. In business-speak, that’s like saying you’re leaving over a third of your potential user base on the table. Let’s drill down.
Qualitative comments and chats
To further explore why people felt the way they did, I sifted through comments, started DMs, and ran some Zooms with poll responders. After all, to get a more well-rounded approach of what our “user” base wants and is looking for, we need to get some meaty qualitative data to explain our numbers.
Despite take-home assessments being the top choice, the people who were the most vocal and willing to share their thoughts with me were the ones that were emphatically against take-home projects. It seems we have a majority who prefer them, but a very impassioned base who hates them.
The main complaint about take-home projects was the concern of how long they can take. Most engineers had a strong sense of how much their time was worth and wished they could be adequately compensated for doing an assignment:
But why do engineers feel that they should be paid for take home projects, but not for other interview types that are almost equally as long? When asked this question directly, the common pattern that emerged was that engineers were scared that the company would not hire them but would still take their submitted project code and use it in a product without providing adequate compensation. Engineers reason that if there is a possibility they are doing actual work that benefits the company, they should get paid for it.²
This fear and opinion is what is causing companies to now pay their candidates for completing take-home assignments. Just a quick search on Glassdoor will show you comments of people receiving payment from completing their take-home interviews. Many people now refuse to do these types of interviews unless they are paid, and the impassioned against having them at all is highly reflected online.
Not to say everyone who preferred technical phone screens felt this way. Some genuinely enjoyed the 1:1 interaction they get with their interviewer, and found those experiences to be a lot more collaborative:
Interestingly, many people who preferred take home projects found the other two options not collaborative at all. Many found that phone screens can be too intimidating and judgmental. The general theme for those who preferred take home projects really enjoyed the flexibility that style of interview afforded them:
Emerging patterns/personas based on interview preference
After sifting through all the data acquired, and after talking to other interviewers and hiring managers, I’ve discovered some patterns of the types of people that prefer one option over another. You might even say that personas were born from this process.
Most people preferred take-home projects because they felt it gave them an opportunity to better demonstrate their skills. This was a more preferable option to people who…
- …recently graduated from a bootcamp/are self-taught and are confident in their practical product building skills
- …have developed domain expertise through their career that is not commonly tested (e.g. performance, accessibility, QA automation)
- …approach problems with a breadth-first perspective and may be slower to solve singularly focused problems
- …get anxious in timed testing scenarios similar to how students get anxious when taking standardized tests
- …do not naturally think aloud and feel pressured/nervous when others observe how they code
- …prefer to explore and work on their own time and share their thoughts and considerations later in a prepared manner
Those who prefer technical phone screens…
- 🌟 …recently graduated from a formal Computer Science (or equivalent) program and are confident in their ability to code contained problems quickly
- 🌟 …are highly paid engineers who value their time and feel that these formats would exemplify their knowledge, logic, and efficiency
- 🌟 …feel they have strong expertise in a specific programming language that would shine in a timed scenario
- 🌟 …flourish under pressure and were always good test takers
- …naturally speak their thought processes aloud and enjoy collaborating with the interviewer on reaching the final solution
Those who prefer online coding challenges shared the 🌟 points above and also…
- …desire efficiency in their interviewing, preferring to do only a few coding challenges in order to be able to interview onsite with a myriad of companies
- …enjoy challenging themselves with riddles or brain teasers
Conclusions
So what’s the conclusion? Is every choice a wrong choice? Is there no way we can ever create a good pre-onsite interview experience?
Not exactly. The ideal interview, in my opinion, is the one that gives the candidate the best opportunity to demonstrate their knowledge, and the ones that give the interviewer the best insight into that knowledge. Thus, the best experience for both is to offer multiple options. Some people polled even explicitly stated they preferred being given a choice:
Yes, offering multiple choices costs more for the company, but it will create a better experience and signal-to-noise ratio. Many companies are already offering this “choose your own adventure” format, including Patreon, Eaze, and Mailchimp.
In my experience, most tech companies give technical interviews like they would administer a standardized test. Their mentality screams “We are the place YOU want to work at, therefore YOU will bend over backwards to our whim”. We should change the way we look at interviewing; we should view our interviews like another company product and our candidates like a user of that product. In my opinion, the mentality should be closer to “We need to understand the different types of people that would apply to this job, and we need to remove barriers so it’s easier for both parties to get what they want”.
As one hiring manager polled put it:
As an interviewer, I want to know that every interviewee is advertising their best self.
As an industry, we talk endlessly about how we’d like to increase diversity and inclusion in the workplace, yet our interviews still don’t reflect this desire. We already know that the way to create better, more diverse teams is by expanding the first steps in the recruiting funnel. Let’s start doing that by treating humans like humans, not like test-taking robots.
My hypothesis is that if we start letting candidates choose from a more inclusive set of interviewing methods, our technical teams will naturally become more balanced and diverse. Who wants to test this theory with me?
¹ Apologies to my mobile, infra, and other engineering cousins; I focused on frontend and backend web dev because I am most familiar with those disciplines. A few of you answered my polls and for that I’m grateful! I’d love to hear from more of you if you have thoughts!
² I tried to find some evidence of this ever happening in reality, but couldn’t find any, so I’m not sure from where this fear manifested. If you know of a story, please comment and share!
What do you think? Is all of this extremely obvious? Would you like to see me do the same type of research on the on-site experience? I’d love to hear your comments and opinions.
