Table of Contents
The Story
Nine months ago, Andrew moved from Chicago to San Francisco to do a startup. At his last job in Chicago, he worked on a highly successful project around skills management and skills development, and after the end of the project he decided there was clearly a great product waiting to be built. The product would help HR managers at technology companies more purposefully develop the skills of their employees.
He built a product he thought was a winner, released it, and no one used it (for all intents and purposes). On the advice of some fellow founders, he decided to back up and reboot the startup with a more disciplined, systematic approach. For starters, he decided to start with a problem/job he believed might be worth solving/improving, as opposed to a particular solution. The problem he decided to focus on was screening technical skills sets for companies that hire a lot of engineers.
His next step was to make sure he understood the people that have this job, HR managers and the functional managers they work with when they’re recruiting new employees. He wanted to learn how they go about recruiting and screening new candidates and how well their current alternatives work for them. He completed these and synthesized his results is a set of personas and problem scenarios.
The Hypothesis
Based on the current process HR managers and functional (hiring) managers have in place for screening candidates and their relative satisfaction with how things are working, Andrew believes it’s worth testing some solution ideas. In general, he thinks this might be a web application that allows HR managers to set up lightweight technical quizzes based on the job descriptions they get from hiring managers. In terms of a testable hypothesis, he’s framing it as follows:
If Enable Quiz (working title for his company) offers companies that hire engineers lightweight technical quizzes that screen job candidates for engineering positions, then these companies would trial, use, adopt, and pay for such a service.
The Right Experiment?
The basic idea with the Lean Startup approach, which is one method Andrew’s applying, is that you should break your ideas into testable hypotheses and test them as quickly as possible with an MVP (Minimum Viable Product). The tricky/interesting thing about MVP’s is that anything that resembles what you’d think of as a real product is probably a terrible MVP. Or, put in a more glass-half-full kind of way, the best MVP’s don’t require building an actual product. A famous example of MVP-based testing is running a Google AdWords campaign for your product and just seeing if anyone clicks. Or creating a signup page for interested customers to hear about your launch when (and if) it happens, driving traffic to said page, and seeing how many people are interested enough to sign up.
What MVP options might make sense for testing Andrew’s hypothesis?
Considering specifics, what might at least one MVP concept look like for each of the following archetypes: 1) concierge 2) wizard of oz 3) sales? (see Exhibit B for the archetypes)
For the above, what would be the pivotal metric and what’s a reasonable threshold for that metric?
Other ideas for testing the idea?
Exhibit A- Personas & Problem Scenarios
A persona is a humanized view of who your customer is and what makes them tick (in your area of interest). Problem Scenarios (or jobs-to-be-done) are fundamental problems, jobs, habits, desires that exist for your personas. They’re independent of a particular solution. Both of these can (and should) be researched through customer discovery interviews where the interviewer asks a series of open-ended but increasingly specific questions. For more depth on these topics see:
Personas Tutorial
Problem Scenarios Tutorial
Customer Discovery Tutorial (see Persona & Problem Hypothesis)
The personas, problem scenarios, and interview guide for Enable Quiz appear below.
Helen the HR Manager
Helen’s been on the job in HR for 10+ years. She knows the routine but she’s never gotten particularly comfortable with the technical topics her managers cover. She’d like to know more, but a lot of her day is taken up with fairly standard issues like benefits management, vetting employee situations, and setting up infrastructure for new hires. She feels a little stuck in the admin area- facilities, accounting, etc. She’s like to engage more in the other functional areas of the business. Helen’s job in terms of hiring is to help the functional manager write the job description, identify channels for recruitment, and perform initial candidate screening.
She doesn’t have deep knowledge about software development, so she’s getting the functional manager to give them the detail they need on job descriptions to screen for specific qualifications. That helps but it’s kind of arbitrary. The system for getting the list is unstructured and highly imperfect. From her point of view, a more systematic ready-to-go system for screening candidates and passing validated screening information to the functional manager would be great. Bad hires are Helen’s least favorite part of the job- it’s a really unfortunate thing for the hire as well as the company. Unfortunately, she has little visibility into whether a hire will work out.
She does the screening and passes the recruit for the functional managers for further screening. Having a more systematic view of why hires don’t work would be highly desirable but she hasn’t gotten a lot of traction with the functional managers on doing those post-mortems.
| Thinks | Helen thinks the hiring process should be so much better- more systematic, fewer bad hires. Professional development is something they’ve identified that they want to do better, but the functional managers aren’t engaged enough to get the whole thing started. |
| Sees | Helen is at the tail end of every bad hire and sees the damage it does to the employee and company, alike. Helen sees that online learning has rocketed forward in the last few years. If someone wants to learn a specific skill, there’s a number of high quality options online, many of them free. They just need a way to help employees organize select into these courses. |
| Feels | Helen feels like crap whenever they have to let someone go. She hates it. The employee hates it. The manager hates it. It’s incredibly destructive and de-motivating for everyone involved. Helen would love to be more involved, more included in functional skills evaluation and improvement. She’s love to have a success story to talk about. Most HR departments don’t do a whole lot in this area. |
| Does | Helen is responsible for recruiting 6-12 new positions per year, screening 25-50 candidates for each position. |
| Problem Scenarios/ Jobs-to-be-Done |
Current Alternatives | Value Propositions |
| Helen sources and screens engineering candidates for open positions, sending only qualified candidates to the hiring manager. Many of the skill requirements are outside her background. | She calls references to get a general sense of their performance on the job. | We’ll offer her a new capability for meaningful screening of technical candidates, increasing % of successful hires and lowering Frank the Functional Manager’s workload on recruiting. |
| Helen writes job descriptions with Frank the Functional (hiring) Manager. In a perfect world, they’d continually improve these based on hiring outcomes and employee job satisfaction. | Right now she just gets lists from the Functional Manager and basically passes them through (to job postings, etc.). | We’ll offer a best practice of menu of possible skills linked to popular job descriptions. They can then pair each of these with quiz content to assess candidates’ familiarity with target skills. |
Frank the Functional Manager
Frank’s been a technical manager for the last 8 years. He started out as an individual contributor in development, then running a technical operations group and after that a product development team. He’s got the team down to a predictable rhythm but he still spends most of his time interfacing with other departments that need things from his group. He’s learned that recruiting is one of his most important success factors, but still spends far less time on it than he’d like. chronically under invest their time in recruiting.
In terms of hiring, his job is to write up a job description for HR, coaching them on how to screen candidates. Screening candidates, even just the resumes, takes up a lot of his time and it often ends up meaning that he delays interviewing candidates for a week or two longer than he’d like. Something that would give him a structured, comparable criteria for looking at new hires would be helpful. Bad hires happen. Some are surprises, some just don’t fit in with the team. Frank admits the surprises shouldn’t happen- he and his team should have the ability to screen new hires to make sure they have the right skill sets. Anything that could prevent that would be of interest to Frank. Frank would like to get a more systematic view of where his team is at on key skill sets.
He has a general sense but he has too many people to know where everyone is on everything. Plus, there are new technologies out there where Frank isn’t that well up to speed. There may be some hidden gems and opportunities out there. Anything that would help Frank get a sense of where everyone is would be welcome.
| Thinks | Frank knows he should spend more time on quality recruiting and professional development. He plans to- it’s on his list. It’s a stitch in time saves nine thing- the time he spends on managing around various skills deficits would be much better invested in tighter up front evaluations and a more systematic approach to skills to development. |
| Sees | Frank sees big gaps in what he was hoping certain staff would be able to do and what they actually do. He works around it, fixes it when he can, it’s part of his job. |
| Feels | When Frank makes someone a stronger contributor than they were when they started with him, he feels good. Frank’s never gotten a lot from HR other than rules and paperork he has to do- he doesn’t seem them as a strategic asset for helping him do what he wants to do.Frank likes programs, systems- if he can see something that way he’s much more inclined to get bought in. |
| Does | Frank gets busy and drops the ball. A little coaxing from HR is OK if he sees the value. Too much and he gets annoyed and he’ll be much less likely to use the system. He is responsive to direct contact from knowledgeable coaches (from Enable Quiz). |
| Problem Scenarios/ Jobs-to-be-Done |
Current Alternatives | Value Propositions |
| Frank’s in a growth area and hiring talent is one of his most important responsibilities. | He spends time with Helen as much as possible to source and screen talent, but he thinks he’s chronically under-investing his time and energy. | We can help Frank easily provide a focused, testable job description and screening criteria to reduce the amount of time he has to spend and improve his outcomes. |
| Culture and personality fit is one of the most important factors in determining a candidate’s overall satisfaction with a position on Frank’s team. Frank needs to assess this. | With all the time he has to spend screening candidates for the basics on skills, Frank doesn’t have enough time to do a good job here. | We’ll free Frank up to spend more time on this by a) reducing the amount of candidates he interviews and b) giving him a strong first-order idea of where the candidates are on key skills. |
Notes Interview Guides
The two sections that follow describe the interview guides for the HR manager and the functional managers they collaborate with when making a new hire. The functional manager is the person who will manage the new hire and created the job description for the team member they need. The HR manager collaborates with them by sourcing candidates, doing initial screening, and helping with the final decision.
Interview guides are not the same thing as questionnaires. The goal here isn’t to create statistically valid or even specifically comparable data. The idea is to get at what’s truly important to the subject, why, and how they actually behave. As an interviewer, you have to apply judgement toward getting the right information from your subject and managing the time and energy both you and the subject have available. Sometimes this means asking the same basic question a few different ways over the course of the interview, and sometimes it means skipping whole sections of your interview guide in order to focus on a thread of conversation that particularly useful.
HR Manager- Interview Guide
SCREENING QUESTION
How many open positions have you filled in the last three months? [Should be >1 if extenuating circumstances but probably >5]
PERSONA HYPOTHESIS
| Question Format | Example Questions (Enable Quiz) |
| Tell me about [yourself in the role of the persona]? | Tell me about being an HR manager? How did you choose that line of work? Why? What do you most, least like about the job? What are the hardest, easiest parts of the job? I’ve heard [x]- does that apply to you? |
| Tell me about [your area of interest]? | Do you do screen new candidates? If not, who? Can you tell me about the last time? Who else was involved? What was it like? |
| Tell me your thoughts about [area]? | How do things work now vs. how would they ideally work? |
| What do you see in [area]? | Where do you learn what’s new? What others do? Who do you think is doing it right? |
| How do you feel about [area]? | What motivates you? What parts of it are most rewarding? Why? What about least motivating, least rewarding? |
| What do you do in [area]? | Would you show me your interview guide? Example notes? What the vetting process was like on the last few candidates? |
PROBLEM HYPOTHESIS
| Question Format | Example Questions (Enable Quiz) |
| What are the top [5] hardest things about [area of interest]? | What are the top 5 most difficult things about making good tech hires? Why? |
| How do you currently [operate in area of interest- if you don’t have that yet]? OR Here’s what I got on [x]- is that right? | How do you currently screen for technical skill sets?Who does what?How does that work? |
| What’s [difficult, annoying] about [area of interest]? | What’s difficult about screening technical candidates?How do you validate they have the right skill set? How are the actual outcomes? Examples? |
| What are the top 5 things you want to do better this year in [general area of interest]? | What are the top 5 things you want to do better in technical recruiting and hiring? |
| Why is/isn’t [your specific area of interest on that list]? | Why is/isn’t screening for technical candidates on that list? |
Functional Manager- Interview Guide
SCREENING QUESTION
As a manager, how many new hires have you made in the last six months? [Should be >1]
PERSONA HYPOTHESIS
| Question Format | Example Questions (Enable Quiz) |
| Tell me about [yourself in the role of the persona]? | What’s it like managing an engineering team? How did you choose to go into management? Why? What do you most, least like about the job? What are the hardest, easiest parts of the job? I’ve heard [x]- does that apply to you? |
| Tell me about [your area of interest]? | How do you decide when you need someone new on the team? How do you decide what you need? What’s the process typically like? What’s it like writing the job description? How long does it take? How do you decide which candidates to interview? Can you tell me about the last time? Who else was involved? What was it like? |
| Tell me your thoughts about [area]? | How do things work now vs. how would they ideally work? How did you make your last decision on a candidate? |
| What do you see in [area]? | Where do you learn what’s new? What others do in the area of acquiring the right talent? Who do you think is doing it right? |
| How do you feel about [area]? | What’s it like bringing on new talent? What parts of that process are most rewarding? What parts are least rewarding? What would it be like in your perfect world? |
| What do you do in [area]? | How many new candidates did you hire last year? Other than just increasing the size of your dev. pipeline, were there particular new skills you were looking for in those hires? |
PROBLEM HYPOTHESIS
| Question Format | Example Questions (Enable Quiz) |
| What are the top [5] hardest things about [area of interest]? | What are the top 5 most difficult things about making good hires? Why? |
| How do you currently [operate in area of interest- if you don’t have that yet]? OR Here’s what I got on [x]- is that right? | How do you currently screen for technical skill sets? Who does what? How does that work? |
| What’s [difficult, annoying] about [area of interest]? | What’s difficult about screening technical candidates? How do you validate they have the right skill set? How are the actual outcomes? Examples? |
| What are the top 5 things you want to do better this year in [general area of interest]? | What are the top 5 things you want to do better in technical recruiting and hiring? |
| Why is/isn’t [your specific area of interest on that list]? | Why is/isn’t screening for technical candidates on that list? |
Exhibit B- MVP Archetypes
The basic idea with Lean Startup is to use the scientific method to test your ideas. This means starting with ideas that are testable, designing experiments to test them, and then creating the time and focus to run those experiments to a conclusion. A high-functioning program in Lean Startup means that the team is making major decisions based on the results of experiments (as opposed to intuition or best guesses or just doing whatever the most senior person decides). In Lean Startup lingo, that means driving to a pivot or persevere moment where you either decide that you’ve got a negative result and should try something else or a positive result that you should explore further or scale.
For more on Lean Startup, check out the Lean Startup Tutorial.
A lot of the actual practice of Lean Startup revolves around designing experiments (including MVP’s) that are a good fit with your hypothesis and current understanding and then running those experiments to a definitive conclusion (and often iterating through that a few times). The sections below describe a few common patterns or archetypes for these experiments.
For these, let’s use the example of a startup that’s exploring the idea of a talking bicycle compass- it might be a device or just an app on a smartphone. They’ve gone out an interviewed serious cyclists that cycle at least twice per month. They found that the job of finding interesting new routes is complicated by the difficulty of navigating while on a bike. They want to test this hypothesis:
If we offer avid cyclists a way to choose new routes and receive verbal guidance while riding, they’ll buy our [app or device], use it at least once a month, and recommend it to their friend on social media.
The Concierge MVP
The basic idea with a concierge MVP is that you hand create the experience you want the customer/user to have, and see what happens. I mention this one first because it delivers the high quality, deepest observations, and that’s useful when you’re in the early stages of a new concept.
In our talking bicycle compass example, a concierge MVP might be to attach a smartphone to an interested subject’s bicycle, turn on GPS tracking, call them and give them verbal guidance to help them follow a route they’ve selected. A scrappy entrepreneur could, for example, solicit subjects in front of a bicycle store located near a popular cycling location (we have several such venues in the Bay Area where I live).
Lean Startup is very focused on observability and metrics (as is its foundation, science). While the Concierge MVP is more about qualitative observations, there are some specifics you can focus on, even recording them into discrete variables in some cases. This is mostly about observing customer participation and focus/involvement in various steps or branches in the process. For example, with the bicycle example, you might be interested in how they select a route with you or whether they already know their route. You might be specifically interested in whether they follow the route they originally selected. You might also be interested in the social aspect- whether they’re the navigator for a group of friends.
This MVP probably has the lowest value in terms of a definitive result on demand for your proposition, the Sales MVP (see below) being the best. That said, as long as you create the right kind of separation, there’s no reason you can’t add another test to the end of your concierge process. For example, at the end of your concierge MVP you could give cyclists a flyer to sign up for a free beta of your upcoming product and see if they opt in once they leave your presence.
The Wizard of Oz MVP
The basic idea with a Wizard of Oz (WoO) MVP is to test a target user experience without real working software- there’s a (wo)man behind the curtain making things happen, hence the mention of Oz’s wizard. For example, the WoO MVP is very popular in robotics where programming and testing a given function takes a lot of investment.
A WoO test for our cycling example might employ a chatbot and then a filtered voice for navigation- all with real humans actually operating them. A user would download the app and then use the chatbot to evaluate and decide on a route (or just supply it). The nice thing about a WoO is that you can see how a user would actually use a given interface without detailing it out- for example, does the user type responses to the chatbot or draw on a map and post the image back to the chatbot? The balance of the experience would be similar to the Concierge MVP except that now we’re trying to make it seem like the voice responses are programmed (filtering and changing the real person’s voice) instead of just presenting them as coming from a human. Our user is likely to respond somewhat differently in this case- or maybe not, but that’s what testing is about.
Metrics-wise, we might be interested both in the usability of the ‘fake’ software (are they able to/willing to complete a given task) as well as their affinity to interact with it.
The Sales MVP
The basic idea with a Sales MVP (or Smoke Test MVP, as it’s sometimes called) is to see if you can sell something before you actually have it. For example, I would characterize a Kickstarter campaign as a Sales MVP. Running a series of Google AdWords campaigns for your hypothetical product and measuring CTR (click-through rate) is a Lean Startup classic. At the next step in your funnel, bringing visitors to a landing page with a proposition and a call-to-action for an email signup is another classic. In a B2B context, asking for a deposit or funding for a specific development is something that can serve as a Sales MVP.
All of the B2C (business to consumer) options are, of course, possibilities in the bicycle example. The team could test a few pairings of Google AdWord ads and landing page designs/propositions and look at sign-up. Metrics-wise, they’d just be looking at performance along their funnel- click through rates, conversions to email sign-up for visitors to the landing page, etc.
Why not just go straight to the Sales MVP if it’s so good at validating hypotheses? The main reason is that it doesn’t offer much depth of observation. Frequently, teams will observe very low CTR’s and then not be sure where to go next. The right place to go is probably an MVP with more observational depth- the concierge or the Wizard of Oz, for example.
Summary of MVP Archetypes
| MVP Archetype | Metrics | Observational Depth | Hypothesis Validation |
| Concierge | Mostly qualitative- sometimes discrete observations on subject participation/interest in different steps/branches of the experience | High | Low |
| Wizard of Oz | Subject progress through interface (like a usability test) and affinity for experience | Medium | Medium |
| Sales | General growth metrics- click through rate, sign up, open and response rate on emails | Low | High |
