Navigating the Complexities of Technical Interviews: A Data-Driven Approach

I've been conducting a lot of interviews recently, and I wanted to share some learnings and insights I've gathered along the way. Interviewing, especially in the tech industry, can often feel like a "noisy prediction" problem that's NP-hard. This post is highly opinionated, but I hope it provides value to those involved in the hiring process or candidates preparing for technical interviews.

Interviewing as a Prediction Problem

At its core, interviewing is a predictor of job performance. To predict well, you need to define a set of X (independent variables) and Y (dependent variables) and gather as many data points as possible within the constraints of time and money. The more data points you have, the better your chance of building a good model.

Interviewing is not just about "filling a gap" but about finding the person who will help solve specific problems and contribute significantly to your team or organization.

Why Initiate a Hiring Process?

There are several common scenarios where initiating a hiring process makes sense:

1. Hiring for Expertise: You have a problem that is extremely difficult to solve without someone with a specific skill set.

2. Hiring for Continued Advice: To gain high confidence and ongoing insights in a certain area.

3. Hiring for Maintenance: To maintain or build something when you lack the capacity. It's important to ensure your problem space is broad and well-modularized, and to recognize that hires in this area often want to do more than just maintain existing projects.

An interesting fact: Studies have shown that cognitive ability (IQ) is a strong predictor of job performance for software engineering roles. This is why algorithmic questions have traditionally been effective in interviews, although their relevance has diminished as they've become more predictable and candidates have learned to game the system.

In addition to the broader trend of AI improving overall engineering productivity, I believe the primary bottleneck for developers is having high-productivity feedback loops across local development, product, and other areas.

On Increasing Y-Parameters

In my recent hiring efforts, particularly for data engineering roles, it became clear that we needed to hire due to a skill gap and workload demands for upcoming projects.

When aligning with business objectives and goals, we started with a clear thesis on how our startup would differentiate itself and win in the market. From there, we planned projects and focused on increasing the amount of data for our Y-variables (desired outcomes).

To increase the number of Y-variable predictors, we considered factors such as:

• Projects
• Personality type
• Startup experience

Experienced startup operators are invaluable assets. For the role I was hiring for, I focused on the following attributes:

• Communication
• Technical skills in data engineering
• Proactiveness
• Potential for growth
• Teamwork
• Experience with cloud platforms (e.g., AWS)
• Startup experience

On Increasing X (Data Points)

To gather more data points on potential candidates, I took several steps:

• Collaborated with recruiters
• Attended industry events and announced we were hiring
• Leveraged my professional network to find referrals for good data engineers

Working with recruiters provided a significant number of candidates, though the quality varied. As an organization grows, you naturally accumulate more data points, making job performance prediction easier.

Building the Interviewing Model

Once you have promising candidates, how do you stack-rank them against each other?

I approached each stage of the interview as a separate information-gathering step. Each interview component was designed to be orthogonal, gathering diverse data points without redundancy.

After identifying 7–8 key criteria to screen for, I structured the interview process as follows:

1. Quick-Fire Questions: To assess technical skills and communication. These questions help determine if a candidate truly understands fundamental concepts.

2. Difficult Systems Design Question: Using an existing system as a reference, this evaluates proactiveness, experience with cloud platforms, startup experience, and teamwork.

3. In-Person Coding Assignment: A half-day coding task remains one of the strongest predictors of performance, screening for general coding ability and problem-solving skills.

During the interviews, I was careful not to spend excessive time talking about myself or the company, apart from an initial overview of why it's an excellent place to work. The focus was on learning as much as possible about the candidate in the time allotted.

This structured interview process allows for quick decision-making. If a candidate doesn't successfully pass stages 1 or 2, we can efficiently move on. More importantly, it gives us the conviction to move fast when we find the right candidate who meets most of our requirements. This agility is crucial in a competitive hiring landscape.

Quick-Fire Questions as an Interview Format

Quick-fire questions are an interesting tool. They provide immediate insight into whether an individual knows what they're doing with a given set of questions. For software engineers, this might involve reading code snippets and identifying errors or inefficiencies.

With tools like ChatGPT, creating these questions can be done efficiently, and the reliability of the questions is improved with the right prompts.

This method is also a clear-cut way to determine if candidates have hands-on experience with coding tasks.

The Value of In-Person Coding

In-person coding assessments are incredibly valuable. They remain one of the strongest indicators of future performance. While we aim to respect candidates' time, these sessions provide deep insights into a candidate's problem-solving approach and coding proficiency.

Conclusion

Interviewing is a complex but manageable process when approached methodically. By defining clear criteria, increasing the number of data points, and structuring the interview process to gather diverse information, we can improve our ability to predict job performance and make better hiring decisions.

I hope these insights are helpful to others navigating the intricacies of technical hiring. As always, I welcome feedback and discussions.

Regards,

Jacky W