Hiring is hard. Assessing technical proficiency, planning for the needs of individual teams, and feeling out culture fit company-wide means that there will always be human decision making involved when filling roles. We assembled our engineering hiring committee — senior developers Mark, Colin, Graeme, and Steven — for an open discussion on what goes on behind the scenes when you apply for a position at Sortable.
What are the key takeaways from our discussion?
We’re not perfect at hiring. We recognize that biases are impossible to eliminate completely. We try to minimize the effect of biases by assessing all candidates through the same process. This includes competency rubrics and our code challenge, two tools that give a clear scoring system. The committee-based approach means that input presented by each member are rooted in technical facts.
Our coding challenge prioritizes which candidates we look at. It provides us a way to gauge the candidate’s problem solving, architecture, and planning skills. And it tells us much more about the quality of work than a resume can. The specific score isn’t the most crucial factor.
New grads looking to enter the market can help themselves get to the top of the candidate list by including a succinct resume, and completing the challenge to provide a work sample. Completing the coding challenge levels the playing field in a way that would not be possible with just a resume review.
The edited transcript of our discussion is below.
I. Why we’re here
What’s the experience that differentiates a developer workload here versus working for a large company down the street?
Colin: When you work at a company where software is not the core business, it can feel like you’re treated like second class citizens in both the tools you have access to, to do your job, and the amount of pushback that you get from management to be compliant with some sort of schedule. We’re more sensitive to the challenges that happen in engineering and accommodating deviations when bumps come up on the road. We work on cooler stuff because we’re not working on a line of business applications.
Mark: More variety of technologies — relevant ones. You get to design and build software yourself, make sure it’s working in production and deal with it when it breaks.
Graeme: What I tell candidates is that the biggest part about working here is that nobody knows what the right solution is, we only know the direction we want to go in. When you start picking up projects, we’ll say “We’re going to do this”. You’ll say “Great, how do we do this?” And they’ll say “I don’t know.”
We don’t know what the fully formed version of this looks like because we’ve never built something like this before. That kind of ownership is scary but empowering. It’s an accelerated learning process. It makes you a better developer very quickly. It also feels like you’re setting the foundations for the company. There’s a lot of power put in the hands of the developers here.
Mark: You get to that point over time. We don’t want to scare people. There’s opportunities to take on smaller tasks that are well defined first.
Steven: We don’t throw people into the deep end either. Everyone is very open to questions and you get support from your neighbours and managers.
II. The hiring process
What’s the candidate process right now?
Mark: You decide to do our coding challenge, apply to a posting with the completed challenge, at which point the developers look at it. We run your challenge against the rubric for code evaluation. We have run hundreds of these and have a good idea of what we’re looking for. Code-wise, if it hits most of the points that we’re after, then we schedule you in for an interview.
You then go through two interviews. The first is a 90-minute interview with two senior developers, which includes resume discussion and code review. We propose problems with the code and ask questions. The second interview is a design interview with two different senior developers. We talk about a high level design problem, and while the question is the same for all candidates, the direction it takes is based on where the individual takes it.
What do you look for in a candidate for a development role?
Steven: We have a rubric for consistency and we like lists. Clarity is what we’re looking for. We bring these back to things we care about instead of a bad experience or a feeling. The rubric was developed by consensus and through a process that started with competencies.
Colin: We were spending a lot of time debating bullshit nitpicky things and one aspect of the rubric that is great for the process was that it helped identify core relevant things. Rather than us spinning our wheels on things that don’t matter, we can make a decision and move along.
I thought those who scored in the top 20% get called in. Is that true?
Mark: We used to care a lot about the score candidates achieved on the coding challenge and never talked to anyone who wasn’t above the 80th percentile. The score is still a factor now, but we care much more about how the code is constructed, and the general approach to the problem. You have to have a certain level of reasonableness. The score can’t be a total disaster though — where the code looks great and performs really badly. Also, it should execute in a reasonable amount of time.
Speaking of decision making factors, how much credibility do you give gut feel?
Colin: It provides the areas to probe when you’re asking questions in the interview.
Graeme: Hiring is hard. We can get better at it than we are, and we strive to. We examine our processes and try to be better. We’re never going to be one hundred percent happy with our process and chase out every issue. And not being one hundred percent happy is not how engineers like to think about things. We like working on things to the point that they are perfect and optimized to the point that it’s always working.
How do you guard against bias? For education, gender, or experience?
Mark: All these things we’ve talked about is an attempt to get that gut feel out of the process because that’s where bias plays.
Colin: Part of that is the committee approach. If someone is on the fence about who to interview, then they can kick it over to someone else to spot for other motivations for why we should bring them in for an interview. After the interview, during our debriefs, everyone has their notes and does an assessment of the candidate and makes their decision on the hiring privately before we share that with the group.
The motivation for making the decision privately is that we want people to form their own opinions. So if I’m influenced by some bias, someone else might pick up some data points that could serve to dissuade the bias. They’re not influenced by me speaking first. They’ve already made their decision, they can then defend their viewpoint and provide a counterpoint to my decision.
Steven: For gender and race, this is one big advantage of the code — it’s just the code. Several of us have made a habit of reading and evaluating the code before looking at the resume, or even the candidate’s name. For education and experience, it’s more complicated. We’re willing to hire a wide range of experiences. Again, the challenge is nice for letting people get in the door and say “Look, I can code. I can do the challenge without a formal education.”
Mark: We all have biases. We’re actively trying to overcome them in our hiring process. We really only just got serious about it in the beginning of this year with the hiring committee. We’re not perfect, but we’re making iterations to our process toward our goal of being really really good at hiring. People can also apply without the challenge. In which case we’re totally biased. We have nothing else to evaluate them on.
IV. Candidate prep work
What’s the best way a candidate can prepare to present the best version of themselves?
Mark: Do the challenge. And do it really well.
Graeme: This is a tricky question personally because what I look for in candidates you can’t prepare for — I like to see an organized thought process. Someone who thinks about a lot of the possibilities before taking action. I like to see someone to asks a lot of questions and makes sure they understand things before moving forward.
Conversely, what are some of the worst behaviours you’ve seen from candidates?
Colin: A code review is a work test. It’s a thing we do everyday in the office. If someone has done the challenge, we’ll walk through it and poke at it and ask them why did they do it this way (there’s always multiple ways of doing it). We want to hear about the thought process and the tradeoffs they made. If they don’t have an answer, or if they have a defensive or aggressive posture to “Have you thought about it this way?” That’s a flag. Because everyday, we’re getting feedback on each other’s code and you have to collaborate.
Steven: I have pet peeves but I try to discount them because they’re stupid and unrelated. It’s not important to the decision making. I’ll go fume in a corner while Colin asks them follow up questions.
Mark: Or the room goes silent for fifteen minutes.
What can applicants include in their application that would make your lives easier?
Colin: A resume that is not several hundred pages long.
Mark: Oh my god, yeah.
Graeme: I review the candidate’s code and the number of years of experience because it helps set the bar for how where I expect them to answer. So someone with ten years of experience should give a better thought-out answer than someone with zero years experience.
Mark: If you participate in side projects or open source stuff, then include them. Link to your GitHub or wherever it is, because we will go and look at those. Including a link to your GitHub when you have nothing there though is… slightly annoying.
What advice would you give for new grads?
Mark: We get a ton of new grad applications and applicants in general. You’re competing against a whole bunch of people and we’re going to give those who complete the coding challenge the most attention, because they’ve spent some time on us already. It’s prioritized.
Steven: More directly, work samples have high validity for assessing the candidate, says the research. This mitigates a huge amount of risk for us.
Colin: They should have something that they’re excited for, or looking forward to, in their job. One thing we look for is intellectual curiosity. It can be a school project, some paper they read, or even a previous work term placement. If they can latch onto that and show some enthusiasm, it’s a huge leg up to showcase their eagerness to learn and contribute.
Graeme: General interview advice: saying something is almost always better than saying nothing. There’s a great temptation for technical people to sit quietly and think about a problem. But that does not give the interviewer any insight into your thinking process. Say it aloud.
Even if you feel awkward, explain the problem to the interviewer as you understand it. The design portion of our interview is a conversation, a back and forth. If I can understand where you’re coming from, I can point you in the direction I want you to go in. If you say nothing, there is nothing to discuss.
Colin: I just want to say: if you think you have a half-baked idea, or only a half formed solution during the interview process — and yet we’re still interviewing you — we think you’re pretty smart and we want to hear what you have to say.
New grads should know they’re in a very good market. They should go get multiple offers, see the options. Everyone on the team is very talented and could go work at any other place in town. And this for me is a reason for working here. I enjoy the work that I do, the projects I work on, and the team.
Mark: Apply to more than one place.
Steven: But if you’re an experienced person reading this, and your job sucks… then come on down!
Any final comments? Around hiring, around working here, around Sortable’s goal to take over the world?
Steven: Sortable has plans to take over the world? That should be communicated.
Mark: That’s going to change all of our priorities.
Colin: This will be heavily edited, I assume.
We’re hiring at Sortable. Learn more about our culture and open roles.