Fixing the Interview Process

I think most people looking for a job in 2024, especially those in the tech industry, feel like the application and the interview processes are both broken. And they are right, both of them objectively work worse than they did previously. And while the issue of fielding an avalanche-worth of applications is a little more difficult to define good all-purpose solutions for, I believe that the interview process could be made significantly better with one rather small change.

Always go through the full interview process with every applicant you interview.

That's it. I can guarantee that the quality of incoming employees will go up significantly with this one change.

Current "Pass-the-Baton" Methodology

I've noticed a growing trend in interviewing over the past few years that I like to call the "pass-the-baton" methodology. You start with an initial call to see if your background and skills are a good fit for the role at a high-level, usually this is a brief and non-technical discussion. I consider it a "pre-interview", as this deals more with applicant screening than actual interviewing.

After that you get to the interview process, which is a sequence of interviews where you need to "pass" each interview to continue to the next one. In my analogy, you are the baton and each interviewer passes you to the next one down a set sequence of interviewers. If you make it all the way to the end, typically you "win".

Anyone that deals with fault-tolerant system design should be able to easily identify the major problem with this approach: every interviewer is effectively a single point of failure. If any interviewer "fails" during this process, it cuts the candidate off from all remaining interviewers.

This might not necessarily be an issue if you could guarantee that every interviewer within a company held the exact same opinions and held the same views about the qualifications needed to perform a particular job. But that is never the case. Even if every interviewer were closely aligned about what qualifications should be considered, everyone will have at least slightly differing opinions on what it means to meet or exceed those qualifications.

But in reality, most interviewers just don't have the proper training or inherent talent for interviewing effectively and so end up "failing" in their goal to accurately assess the viability of a candidate. Even at companies where I had the highest of praise for their interviewing process, not all interviewers were equal, and that's nothing to be discouraged or ashamed about. It just means we need fail-safes in the process to ensure that one person doesn't hold a disproportionate amount of power in the interview process, especially those that aren't going to be managing, or directly working with, the candidate if they were hired.

The best fail-safe is, in my opinion, democratizing the interview process. You need more interviewers for every single candidate. You need as many interviewers as you can reasonably spare. Then you can have all of them discuss feedback before rendering a final decision.

And this is not a new concept. In fact, this used to be the standard way of interviewing, at least for me it was. But we've left it behind for what I can only assume is a misguided desire to streamline the process. Companies seemingly decided that they were comfortable with reducing the overall quality of candidates in an attempt to reduce the perceived resource consumption of the interviewers.

And in the end, there's no benefit to the interviewers. If one interviewer has a high failure rate it just means that they, and any interviewer that typically goes before them, have to deal with more interviews than those who typically come after them.

To demonstrate the numbers, let's assume you have four candidates go through five interviewers each. That's roughly 20 total interview, although each individual interviewer only does four interviews. If the second interviewer has a higher rate of rejection for candidates than other interviewers, that could mean you have to interview twice as many candidates to get someone to pass them. That means eight candidates, where seven of them go through the first two interviewers and only one of them goes through all five interviewers. That totals 21 interviews: 18 for the first two interviewers and then one for each of the remaining three interviewers.

And that's a generous example where we assume that only one person has a high rejection rate. If you had more "wildcard" interviewers in that process, it would take even longer, and the distribution of interviews would skew heavily toward those typically involved earlier in the sequence.

But enough theoretical, let's get to some real-life examples.

The Group Interview

The first job I ever applied for was a part-time job for a company that primarily handled USA military contracts. I was a student at college and they liked to hire a few of us part-time every year to give college graduates some real-life work experience they could put on a resume.

They did a single phone call with me and then asked me to show up for an in-person interview with some of the team leads. The idea was that I would interview with the leads from multiple teams so that I could be assigned to whichever team they thought I would fit best.

I like to call this a group interview because you engage with a relatively large number of interviewers at one time. Some companies might even do multiple group interviews. I personally consider any interview session containing more than two interviewers to be a group interview; this is not counting anyone that is just listening in to the interview and does not get "voting rights".

For this specific interview, I was trapped in a conference room with 8 or 9 people for roughly one hour. It was overwhelming for my first-ever job interview but, on the good side, I was done with the entire interview process after that hour. And a couple days later they got back to me saying that three of the team leads wanted me and it took a while for them to choose who would get to have me.

This has become incredibly rare, which is understandable. This company allotted plenty of time for team leads to do interviews, it was just a part of their work responsibilities and they weren't expected to "make up for lost time" with their other work tasks. Most companies see interviewing as a necessity that interrupts normal work operation. This means that it can be difficult to schedule large groups for the same time.

The On-Site

My second ever job interview, which was for my first full-time role, was with Sony Online Entertainment (long before they became Daybreak Game Company). The pay was atrocious, but I figured getting a Sony-subsidiary on my resume would more than make up for it, and I was young enough that I didn't need a huge salary for myself at that time.

My two would-be teammates did not like how inexperienced I was. They thought I interviewed decently over the initial phone call but was lacking in the on-site. It's as though they were incapable of understanding that interviewing for your first full-time job might make someone extremely nervous. They also would have preferred someone with a little more experience.

But my would-be boss and one other manager within the company both interviewed me and said that it was my potential that was really good. Plus, I was more than willing to do the boring and tedious jobs and work on-call rotation with very little pay, something that the other engineers weren't really factoring into the decision. People were not lining up for this role back then.

And so I was hired. And after about a year on the team, the three of us go out for lunch together and they tell me that they were against hiring me initially. But after having worked with me for a year, they were both glad that I had joined them because I was a reliable and enjoyable colleague who had learned very quickly and noticeably improved the workload distribution for the team.

Imagine if I instead spoke to only one of the engineers and never got to speak with my would-be manager? I would not have landed the job and, honestly, the position probably would have remained open for much longer than they would have liked. And even if they could have held out for a better candidate, there's also an equal likelihood they would have hired a worse one.

Democratize the Process

When I worked at a company that did monthly mystery box subscriptions, I was frequently part of the hiring process. And we always did an initial phone interview followed by one day of in-person interviews. This was common 10+ years ago; although more exhausting than a single large group interview, it yields similar results.

It was not uncommon for one or two people out of six or seven interviewers to at least be "leaning no" for a candidate who we put forward an offer to. That's because anyone not sold on a candidate was provided an opportunity to discuss why it wasn't a yes, and these discussions were always productive.

And it wasn't just convincing people why we should hire someone. There were instances where most people said "yes" initially and then flipped to "no" after hearing the reasoning of others, especially if they were just "leaning yes" previously. So it was equally as good as preventing incompatible candidates from getting accepted as it was preventing compatible candidates from being rejected.

In short, the process was:

• Interviews for the whole day.
• Next day (or sometimes end-of-day), all interviewers convene to discuss their conclusions.
• If it's a unanimous "yes", immediate offer. If it's a unanimous "no", immediate rejection. If it's mixed, discuss.

It was still possible to have the interview process cut short, like if a huge red flag was raised during one of the interviews, but usually all interviewers took their turn and we democratized the selection of candidates that way. Having six or seven opinions of varying quality allowed us to average that feedback quality to something more reasonable.

Out of the 12 people I helped hire at that company over a six-month period - including VP and CTO positions - only 1 of them ended up being a poor fit for the job. That's a 92% success rate; insanely good. And I would say that at least half of those candidates did not receive a unanimous "yes" when the interviewers initially convened to discuss their feedback.

As a bonus to the candidate, they typically waited less than a week to hear a definitive answer. Obviously a rejection was handled quickest, usually less than 24 hours after the on-site, but sometimes if we had more candidates pass than we had open positions for, it might take another day or so to rank them and send out offers.

The Virtual On-Site

Even pre-pandemic, a growing trend for companies was to do "virtual on-sites", where you would be at home on a video call and then interviewers would pop in and out over the course of anywhere between 3 and 6 hours. Certainly the pandemic made it wildly more common for a short time, but then the competing "pass-the-baton" methodology seems to have won in terms of adoption and popularity.

That's not to say that virtual on-sites are no longer a thing. I know that some large companies like Amazon still do them, and honestly I think that's great. There's still usually a bit of pass-the-baton there, where you might have one or two more technical interviews before the virtual on-site, and honestly they should just cut those. If you're going to have people spend half the day or longer just doing a series of non-stop interviews, that's definitely sufficient for making a decision on a candidate. No need to complicate things further.

And I would argue that this is the methodology that most employers should be using right now: the virtual on-site. You get the scheduling flexibility from having people participate remotely, you get the democratization from having many voices discussing their feedback, and you get quick turnaround times on candidate judgements because all feedback is acquired at the same time.

Interviewing is Hard

This might be hard to hear, but most interviewers are not good at interviewing. It's hard. If you don't have the proper training or inherent talent for it, it's just not as straight-forward as people might think. And many times, someone learns how to interview just from seeing other people interview and from their own experiences as a candidate. But if those interviewers they are emulating weren't great at it, then the "next generation" of interviewers won't be great at it either.

I once worked for this very small mobile games development company; there were less than 20 employees when I joined. One of the people hired shortly after me was incredibly smart academically, and a solid software developer as well, although he absolutely carried an inflated ego. Still, he was overall pleasant to work with and produced solid code.

Toward the end of my time with the company they were looking to hire a couple more engineers. This colleague of mine decided they wanted to throw in a "hard" programming question about consistent hash rings. We didn't use or maintain anything related to consistent hash rings. But he figured that it would help weed out anyone below a subjective level of skill that he didn't want to "waste" time interviewing.

It's also worth mentioning that this was around the time that companies like Google were using objectively bad questions during interviews because they felt like it helped identify "quick thinkers" or whatever. Stuff along the lines of "how many beans are in this jar?" Questions that have absolutely no bearing on someone's ability to do their job (unless guessing beans is somehow your job).

I told him that it made no sense to quiz someone about a technology that we didn't use and which had no relevance to the daily tasks they would perform at the company. But I wasn't part of the interviewing process at that point and so he did whatever he wanted to, probably because of the aforementioned ego.

Someone eventually did really well on his hard question. And within three months of their hiring, the company was looking to let him go because he was absolutely trash at doing regular programming tasks. In short, he was smart academically but inept practically. And that's just part of the danger of letting someone really good at their job interview people when they have no inherent aptitude or formal experience or training with interviewing methodologies and techniques.

I could honestly go on and on about interviewers that end up reducing the quality of potential hires because they just aren't very good at interviewing. People whose personal opinions about a role don't align with the responsibilities or the description of a role; people who choose interview questions that have no bearing on the content of work you would perform for a role; and people who simply have their own prejudices and won't render fair judgement in a subset of candidates.

There are just so many bad interviewers. And while it's usually not their fault that they don't interview candidates well, it's obvious to me that you can easily reduce the impact of bad interviewers by simply adding more interviewers to the process. The average quality of interviewers should theoretically reach a happy medium.

To Recap

Up to this point I have shown examples of how:

1. People who don't specifically have training on how to interview can deny opportunities to people that are actually a great fit for the role.
2. People who don't specifically have training on how to interview can select candidates that are actually a terrible fit for the role.
3. Democratizing the interview process yielded faster and more effective hiring.
4. Group interviews specifically resulted in significantly shorter interview processes with more placement opportunities for candidates.

I understand that group interviews can be rough to scale at the current volume of applicants. Larger companies could do this more effectively if they wanted to, but smaller companies would probably instead focus on the "virtual on-site" methodology where you have one or two days of back-to-back interviews with smaller groups of interviewers.