Duygu Karaboncuk (Amazon) on Lean UX

Alex:
Thanks for joining me, I’m your host Alex Cowan. Today we’re going to be talking with Duygu Karaboncuk, who is a senior product manager at Amazon. Duygu, thanks for joining us.

Duygu:
Thanks for having me.

Alex:
You are a senior PM and the last product you worked on was the Woot product. For our listeners who don’t know what Woot is, can you tell them a little bit about it?

Duygu:
Sure- Woot is a startup that was acquired by Amazon. They have amazing deals and what makes Woot unique is the funny and quirky content that they have on the website so that’s what brings people in over and over again to the site.

Alex:
What would be an example of funny, quirky content?

Duygu:
We have sometimes make fun of the products that we’re selling. Very recently, we were selling a kind of leather. And our creative team was making fun of the leather by having a picture of ‘pizza leather’ in it. Kind of lot of people shared and it was internal joke for us. That kind of thing. We sell very cheap and great deals- at same time we make fun of ourselves selling it.

Alex:
What’s it like to be a product manager on Woot?

Duygu:
What I loved about working at Woot is you get to really make a change in customer experience on the website. I was responsible for improving the customer experience on mobile web and the desktop version of Woot. Hearing from customer feedback and doing lot of customer surveys and analysis and understanding what they really want us to improve on the website, you could go and make the change and you could see it through. I think that why as for action and putting products out in front of the customers, that was most fun thing about being a product manager at Woot.

Alex:
Can you give us an example of how that cycle works for you?

Duygu:
Sure. When I first joined the team, we got feedback from customers through a survey we conducted that they really wanted to see best sellers separately in Woot. We started to dig deep into that idea and did further customer testing and surveys and we ended up launching best sellers widget in addition to the ‘Last Chance’ and ‘New Deals’ lists on Woot’s gateway.

Alex:
So there was some kind of idea that they wanted to see what everybody else was buying, and they wanted to see that differently than they wanted to see the stuff they were already seeing. Is that about right?

Duygu:
Correct. Yes.

Alex:
You got evidence from surveys or feedback on the site. Can you give me an example of the kind of things you heard that led you to think, “Oh, the customer is saying this and I think what might be good for them is seeing most popular list or something like that.”

Duygu:
Sure. For example, I’ve started this voice of the customer program at Woot where we collect the customer feedback to the feedback page that we have and also through the community on the social media. We would too looking into the reports on customer inputs, we will be able to identify what are the things that they’re asking from us. That would be one input into our decision making. And also, once we have some idea about what we want to launch, we would bring customers in and test different concepts with them. Those would provide inputs for us to decide on the products.

Alex:
You mentioned that you have this voice of the customer vehicle. Can you tell me more about how you operationalize that, how you make that work?

Duygu:
Initially, when we started this project it was very manual, so going over the feedback and categorizing it very manually. But then, as we started to see lot of positive results from this program, we started working with internal teams to apply machine learning into categorizing and analyzing the feedback. So that now you don’t need to go and figure out what are the terms, but you can ultimately put it into main buckets and categories so it makes it faster for you to analyze the feedback. Also, you don’t need to depend into one person to give you the results. You can just go in and analyze and slice and dice the data.

Once we started automating it, we also grew the team. Now there’s a team in Hydrobad that manages this voice of the customer program so we can see the customer trends.

Alex:
Can you give me an example of a place where you use this program to learn something that was actionable to make the product better?

Duygu:
Sure. I’ll give two examples. One is when we launch best sellers widget on Woot, we got lot of feedback around people did just not find it as engaging, as fun. So they were asking what makes Woot special is its core brand value proposition of being fun and the product that we launch didn’t come across that way. Getting that feedback and through A/B testing, we worked with the user experience team to make it more fun and quirky for Woot.

Our creative team started adding descriptions that makes these best sellers and new deals more funny for Woot customers. For example, on the last chance list, a recent description was “Your ex didn’t give you a last chance, but we are giving you one. So, here you go: last chance items.” That was one of the feedback that we received and immediately acted on it.

The other example is that Woot is very well known for its iconic monkeys. We’d recently changed the monkeys, and we got a lot of feedback saying people just didn’t like the new version. So we learned from it, we improved, and we’ve sent another version, which people loved. I think getting that constant feedback and iterating on the product is the most fun (and key) part of being a product manager.

Alex:
Once you learn something like this that you think you want to act on, can you talk a little bit about the process where you take that evidence or that observation and then you translate it into action with the rest of your team?

Duygu:
Sure. We have this process called Working Backwards at Amazon. You start with the customer experience or the feature of the product that you want to launch. Starting from that final launch and how the customers are going to interact with it, you go backwards and that’s when you start developing it. We usually write a press release. If I were to launch this project, how exciting would that be? And then you start answering all the relative questions regarding the product, like marketing, development, other management-related questions. Once you prepare that document, it goes through approval process. It needs to make it through the organization and once it’s accepted that we will invest in this, you go into more execution mode as a product manager and you start working with the UX teams and development teams to develop the product.

Alex:
You write this press release and then you try to think about the implications for the different functional areas at Amazon like finance and marketing, and then that’s the vehicle through which you get their notes and their feedback? Is that right?

Duygu:
Correct.

Alex:
How do you brief your UX, your design people on these things?

Duygu: You always work with them very closely whenever you’re writing a PR FAQ document. But then, once it’s approved, your input to the UX team is like, what are we trying to achieve with this design or with this product? So, it’s very helpful if as a product manager you provide them the design goals. Are we trying to help people find products easier or are we trying to help people bounce less from our website? What are the design goals? Once they go around, they come up with several variations. You would evaluate those based on the goals that you have as a product manager.

Alex:
How do you evaluate them? Do you just, with your intuition or through testing? How do you do that?

Duygu:
I think you initially, you try to understand whether the design meets the goals you want to achieve. Of course, that’s just one input. Then, afterwards what we do is we bring customers in for focus groups and usability testing. Let’s say we in Woot we had a new design to help customers browse easier. In the designs that we were evaluating, we brought customers in and asked them, “If you were to find a pet’s toy for your friend, how would you go around finding it?” And then you would observe customers interact with the new design, and see which one helped solve their pinpoint. I guess, afterwards, like after you vetted one design, you go through AB testing and then data and so forth.

Alex:
How many of those alternative designs do you take to production?

Duygu:
I think it depends on the complexity of the product you’re working with. Sometimes if it’s a very complex product, some teams might choose to do just one kind of alternative. Or if it’s an easier change, you’ll do a lot of variations to test. Usually what we did in Woot is we came up with one alternative after doing comprehensive usability testing and surveys with customers. Because those are developing that takes time at same time too.

Once we choose one design, we would develop it and then build an A/B test. When you’re doing an A/B test, that’s when you start getting lot of feedback and learn from it and you iterate further too.

Alex:
So the other variation, is it just the baseline? What you have in production already?

Duygu:
Yes, controlled versus the new experience.

Alex:
You’re a Darden alumni. What did you learn at Darden that’s been helpful to you in doing this job?

Duygu:
I think I really enjoyed the elective courses that I’ve taken in my second year. Like, how to develop products, how to work in an agile development environment, how to do software design. Those courses have been super helpful for me to understand on day one being able to contribute at Amazon. But at same time, what I’ve learned, taking a step back, is this approach to products that you won’t get it right for the first time. You will fail, but you need to incorporate customer feedback, learn from it, move faster, iterate again. So I think that approach of failing is okay. That’s how you learn. That’s how you improve. I guess was the most valuable lesson that I’ve learned.

Alex:
What’s the hardest thing about being a good product manager?

Duygu:
I would say it’s stakeholder management. You have a lot of stakeholders from marketing to development team to UX team to management to just aligning everyone and getting their buy-in into the product vision so that everyone is contributing and marching toward the same direction. I think achieving and aligning stakeholders is the most challenging. I also think it’s the most fun part at the same time, so that you as a team are trying to achieve this goal for the customers. I like it.

Alex:
How do you do that? How do you align stakeholders?

Duygu:
In my experience so far, if everyone is involved from day one, and you add their contributions and their feedback into your documents and the process. They are part of it, and they contribute throughout the process that they are also as a part of the team. That’s when you align everyone at the same time. If you leave it to the last point to get their approval, then you might also see a lot of surprises too. That’s what I try to do.

Alex:
And the PR FAQ vehicle, that press release, you find that a helpful way to do that early on?

Duygu:
Correct.

Alex: Well Duygu, thank you so much for sharing your story.

Duygu:
Thank you so much.

Alex:
This has been the Interdisciplinarian. If you want to learn more about how to apply modern product design methods, and execute in agile alliterations, check us out at bit.ly/hiagile, our agile specialization on Coursera. Thanks for joining us.