Table of Contents
This is a case study for use in a class like Software Design. It deals with moving from validated learning on customer/user needs to testable user stories.
A Season of Change & Disruption
“HVAC in a Hurry” (HinH) is now itself in a hurry. After decades of happy existence as a prosperous regional supplier of heating & air conditioning services for commercial properties, its industry is consolidating. The CEO, Mary Condor, feels they need to either scale, combine with another firm, or gradually be out-competed into insolvency. Scaling is her first choice.
She’s brought in Frangelico DeWitt to lead a change management and digital transformation program. His charter was to take the best of what HinH has learned and to use it to improve existing operations and facilitate rapid expansion into new regions (possibly through a franchising model). While resources are scarcer than he would like, DeWitt feels up to the challenge (with a first name like Frangelico, you have to be resilient).
Learning What to Build
It’s Friday afternoon, and the team is closing out a one-week design sprint to understand the company’s key roles/personas and the relative importance of their various tasks/jobs-to-be-done. Based on what they learned in the interviews, the team thinks the most valuable early wins will involve improving the interface between dispatch and the HVAC field technicians. The dispatchers work at headquarters and field calls and emails from customers as well as technicians about HVAC repairs. Technicians are dispatched to customer sites to execute jobs (installation, maintenance, and repair). Basically, the idea is to take what individuals at the the company have learned works well and to standardize and automate it (where possible) with updated software.
The team has tuned their current charter to focus on three design challenges:
-
- How might we improve the troubleshooting and problem resolution dispatch is able to do over the phone to reduce site visits and improve customer retention?
- How might we improve the preparation of technicians when they go on site for a job to reduce time on site and the number of visits per job?
- How might we improve the self-service capability of technicians on site to reduce downtime on site, the number of visits per job, and customer retention?
Solving the Right Problem with the Right Solution
To close out their current sprint, the team plans to focus on developing user stories for both of the first two design challenges (#1 & #2 above), since they’re so closely related. In terms of the specific problems and propositions where they think there’s a likely win with some new software, they’ve sketched out the following focal points:
| Job-to-be-Done + Current Alternative | Proposition | Outcome Metrics |
| JTBD) Resolving Customer Problems Remotely
Alt.) Dispatch does this. The level of know-how and success varies a lot between dispatchers. |
A) If we create a best-practice troubleshooting tool for common problems, dispatchers will use it and it will improve resolutions over the phone.
B) If we create a post-mortem and review infrastructure, dispatchers and tech’s will engage on improving and expanding the troubleshooting tools. |
– Increase in % Issues Resolved Remotely
– Increase in Customer Retention (Av. Contract Length) |
| JTBD) Finding Available Technicians with the Right Skills
Alt.) Dispatch has a means to see availability with the current system. |
(There might be a win around more sophisticated filtering and selection based on skills and experience with a particular customer, but it doesn’t look like as easy or obvious a win as the other items we’re working. It’s on the product backlog for now.) | – Increase in Customer Satisfaction per Job
– Increase in Customer Retention (Av. Contract Length) |
| JTBD) Preparing Tech and Customer for Job On Site
Alt.) Right now, it’s just free-form notes and the tech’s feel they’re encountering the same kind of problems over and over- missing information about accessing the site and the nature of the problem so they can bring the right parts and equipment. |
A) If we create a set of adaptive checklists based on the information on hand in the account and the type of job, dispatchers will use it and tech’s will be better prepared for jobs. | – Decrease in Time on Site
– Decrease in Site Visits per Job – Increase in Customer Retention (Av. Contract Length) |
From Validated Proposition to Testable Solution
To close their design sprint, the team wants to draft a set of user stories and rough prototypes to make sure they’ve thought through the specifics of how they might deliver the propositions they’ve identified. The basic idea is just to make sure they’ve thought through a practical, realistic user experience that would deliver on the propositions.
To make sure they’d really thought through the delivery of their propositions, the team started with storyboards.
Storyboard 1: Handling Customer Calls
This storyboard is a take on the process the team has learned about where the dispatch is taking an initial call and figuring out what to do next. This take is mostly focused on repairs (calls for installation and maintenance differ some). Her goal is to either resolve the issue on the call or make a specific conclusion about whether they need a site visit and what needs to be done.
| Panel | Notes |
| A | Here Danielle the Dispatcher take an inbound call from a customer who has a time-sensitive HVAC issue. Danielle needs to bring up their account for background and to make sure they’re a current customer. |
| B | Danielle the Dispatcher steps through a few questions, isn’t able to resolve the issue. |
| C | Danielle has an idea about the fix and thinks it might be something where of the technicians could suggest a remote fix. She asks for a snapshot to send to a technician who’s been working at the site. |
| D | Carl the customer takes the requested photo and sends it. |
| E | Danielle asks Tess if she can take a look at the issue. Tess the Technician gets the photo and a question via text from Danielle. |
| F | Tess the Technician responds with a possible fix but thinks the fix will probably require a visit. |
| G | Danielle the Dispatcher gets back in touch with the customer, who tries the fix. The fix doesn’t work and Danielle determines that they’ll need to schedule a site visit. |
Storyboard 2: Scheduling & Prepping a Site Visit
In this storyboard, Danielle (or Daniel) the Dispatcher has concluded that they need a site visit to complete a repair. Her goals is to schedule the site visit with the best available tech, making sure both the tech and the customer are ready to do what needs to be done to complete the repair.
| Panel | Notes |
| A | Daniel has made the determination that HinH needs to do a site visit to make the repair and needs to arrange this with the customer. |
| B | Danielle needs to schedule a time that works for the customer and where there is an appropriate technician available. |
| C | Danielle wants to make sure the address and contact information is current and to update the technician if there are any particulars for this visit (ex: alternate contact). |
| D | Danielle wants to make sure all the right contacts at the customer know when the visit is happening, the scope of the visit, and can easily update it if they require changes. |
| E | Danielle wants to add a new alternate contact to the account for future reference. |
| F | After 48 hours, Carl the customer needs to move the visit to the next day. |
Towards a Testable Design
As the team sits down to draft stories, they have a few key priorities. First, they want to be detail-oriented- not about the specifics of the implementation, but about the experience a user would have that delivers their target propositions. This is a key feature of user stories: they are great at working out user experience but avoid the mistake of over-specifying the implementation details prematurely. They’re a focal point for discussing implementation; not a specification.
Second, they want to make sure they think about the ‘messy’ parts of the experience- times when things don’t go as planned. They know from experience that glossing over these situations is often what causes users to abandon or underutilize a system. This doesn’t mean they plan to cover every theoretical corner case- just that they won’t ignore the inconvenient details they’ve learned about in their interviews.
Third, they want to make sure their user stories are testable. Probably the best way to make sure you have a coherent user story is to ask yourself ‘Could I mock up an interface for this, prompt a user to try to execute the action in the story, and then specifically observe whether or not they achieved the target goal/reward?’. For example, if you were making a mobile app for salespeople to log their customer appointments, you might ask them to show you they’d add a note for XYZ customer that says (something) and then ask them if they think it was saved successfully.
Working from the storyboards, the team plans to do the following:
- Draft an ‘epic user story’ to summarize the user experience described by the storyboard.
What initiates or triggers this story?
When is it done and successful from the perspective of the user?
Note: see appendices on the format and content of user stories - Detail individual ‘child user stories’ within the epic to detail the component interactions in more detail.
Note: These have the same format as the epic (three clauses)- they’re just at a lower level of detail.
How does the story unpack? How does the user go from ‘a’ to ‘b’ to ‘c’? Is it just one happy path or are there plausible error/problem conditions they might need to work through?
Do you think the storyboard is missing important details? It’s normal and healthy to swivel between the storyboard and the user story as you develop the details.
Do you see any details they might have missed? - Layer in testability (and possibly revise!)
How would you prompt an actual user (test subject) to see how they approach the story with a given interface (prototype or working software)?
Once it’s released, how would you observe the user population’s interaction with it and assess the extent of its efficacy and usability?
The Littlest Tutorial on Agile User Stories
TL;DR User stories are a generally accepted tool for modern software development. They have a specific format:
“As a [persona],
I want to [do something]
so that I can [realize a reward]”
This format is designed to help the story writer be descriptive about the target user experience and, crucially, include a testable definition of user success. The purpose of user stories is to drive better, more intentional discussions with the rest of their team at least during the following processes:
a. before the fact as you’re designing
b. during development as you’re implementing
and
c. after release to make evidence-based decisions about whether to hold, modify, or drop a given implementation of a user story.
Done right, the format helps prompt the following important questions:
Stories are often organized under ‘epics’. These epics are user stories (same format as any other story), but they summarize the user experience where the balance of the user stories (I refer to them as ‘child’ user stories) detail the individuals components of the experience. For example, let’s say you had this storyboard of an HVAC technician finding that they need a replacement part to complete a repair:
An epic that summarizes the interaction might be something like: ‘
As Ted the HVAC technician,
I want to identify a part that needs replacing,
so I can decide my next steps.’
From there, you’d move on the detailing individual user stories. If you’re starting from storyboards (which is a great idea!), don’t force a 1:1 relationship between the storyboard squares and the user stories. You may find some squares require more than one user story and others are just background and don’t really need their own story.
There are two additional items I recommend including as you draft any story at the child level:
1) your first take on how you’d prompt a usability test subject to engage with a design of the story
and
2) analytical questions and metrics you’d use to answer them.
While you should not prescribe the UI design in your user in order to have those conversations at the right time, the very best thing you can do for your child stories is to make sure they’d defined around an observable definition of user success.
You can think about it like this: if you want a test-driven approach to finding the ‘Right Solution’ (and you do), then your user stories are doing the job of framing testing your solution or solutions. And taking a test-driven approach means that you draft tests first and use it to focus your implementation work, like designing prototypes you might take forward to development.
Here’s an example of user stories drafted with testability, both for usability testing and live analytics.
EPIC: As Ted the HVAC technician, I want to identify a part that needs replacing so I can decide my next steps.’
| CHILD USER STORY | SUBJECT TESTING | ANALYTICS |
| I know the part number and I want to find it on the system so I can figure out next steps on the repair. | Moderator Guide: Hand the user the applicable prop (HVAC part with a part number visible) and ask them how they’d look it up on the system. Ask them to use the paper cheater to enter the part number. Then (on success), progress to the next page and ask them to ID the part and its pricing and availability.Output: Did the subject find the correct field for the part number and click the right button to advance? Did they ID the pricing and availability of the applicable part on the following page? |
How well does this search type work relative to the alternatives? How often is this search used per transaction relative to the alternatives?Metrics: Searches of this type relative to others Sequence of this search relative to other search types Conversion to order from this type of search (%) Candidates for A/B testing: |
| I don’t know the part number and I want to try to identify it online so I can figure out next steps on the repair. | Moderator Guide: Hand the user the applicable prop where the make and type is clearly visible and ask them how they’d look it up on the system.Output: Did the subject find the correct fields and click the right button to advance? Did they ID the pricing and availability of the applicable part on the following page? |
(see above) |
| I don’t know the part number and I can’t determine it and I want help so I can find out its price and availability. | Moderator Guide: Prompt the user that they aren’t able to ID this part and want help.Output: Did the subject find the correct fields and upload a photo? Do they understand what’s going to happen next? |
(see above) |
| I want to see the cost of the part and time to receive it so I decide on next steps and get agreement from the customer. | Moderator Guide: Tell me what you’re seeing here? What the price for this part? How soon can you get it? Given that, what would you tell the customer?Output: Result on whether the subject understands and can complete the applicable search process. |
How often does this lead to a part order? How will techs that do this perform relative to others?Metrics: Conversion rate to order Customer satisfaction per job of techs in a cohort that use the tool vs. baseline (mean customer satisfaction per job) Billable hours for techs in this cohort vs. baseline (billable hours per week) Candidates for A/B testing: |
For more on stories, check out the online tutorial: Your Best Agile User Story.
Managing Product with ‘INVESTable’ User Stories
Writing great user stories and (more importantly) driving great discussion and development with them takes practice. Below are a few key focal points for you to consider as you draft your stories. These are based on the INVEST checklist by renowned agilist Bill Wake.
Individual
Is the story integral- does it have all three clauses ([1: As a persona], [2: I want to [do something]] so that I can [3: achieve some kind of reward/conclusion.]?
Does the story make sense as a stand-alone item? Could you sit down right now and prototype it? Making sure you’ve broken down the experience into discrete pieces will make your design better. Making sure you can work in small batches will make your practice of agile better.
Negotiable
Is the story about the experience you think the user should have vs. how that experience should be implemented? This is key to your functioning effectively with your team and generally making sure you stay focused on doing what actually makes sense for the user vs. checking off items on a to-do list (which is easier but not effective).
Here’s an extreme example: ‘As a user, I want to push a red button, so I can submit the contact form.’. This story has no reason for existing- if somehow you’re just utterly convinced you need a red button, do a prototype that shows how it would look. Instead, this story should be more about what this user is hoping to achieve but submitting the form and how they’ll know if they’re going to get it.
Also, avoid terms like ‘easily’, ‘quickly’. No one is every tried to build software that’s difficult or slow- it happens all the time but it’s not because someone forgot to add ‘quickly’ to their user story! If there’s some kind of benchmark you ultimately need to reach with users, that’s a good item to add to your test cases.
Valuable
What validated learning do you have that suggests this user story is worth building? What measurable outcome would it improve? See the above section Learning What to Build for more depth on this.
Estimable & Small
These aren’t the same, but they’re tightly related. If it’s hard for a developer to roughly estimate the level of effort to implement a story, it’s very likely that the story is too vague (IRL, this is what I observe 92.4% of the time). It’s also possible (though far less likely) that the developer or team is thinking of an approach to implementation that’s more expansive than is necessary or product. For perspective on this, check out Bill Wake himself discussing ‘YAGNI’: Bill Wake on Yagni.
Testable
Personally, this is my number one with a bullet. Why? A highly testable story is very likely to score well on all the elements above (in my experience). Key to this is the third clause of the story: ‘…so that I can [achieve some kind of reward/conclusion].’. Making sure you can think through the user experience to a testable conclusion is not only key to moving forward with implementation and launch of this functionality, it will help you be more thoughtful about your design.
