Session 4 Recording

So, There we go.

So welcome to day 4 in recording here, which you will find on clarity flow again later, where Aaron knows how to find it and and we can help get it later in Matthew 2.

but yeah, post class email with links to mural, linked to clarity flow, um, I'm gonna do a PDF of mural, uh, the mural board, uh, and, and make sure that that's linked.

I will at least copy that to clarity flow.

And um an updated version of the deck, uh, since it's changed since the thinking of the course, uh, so all that will come your way.

Um, is there anything else you you you would like to have me send you or link to to make it easier from like the, the course and any of the materials or anything that we talked about? I think those are the major things, right, recording, yeah, and output of everything we've covered the material we've generated, and then.

Yeah, the side decks, I think.

Unless there's something else we've been missing out on.

No, I don't, I don't think so, but you actually triggered something else for me.

Um, I will get, uh, links or PFs for the um lean product canvas deck.

It it's an easy thing to find, but if you've got it quickly available to you, that's just another good one to have.

Um, but that's all I can think of right now.

I just, I'm trying to think if there's any other concepts that we talked about that I should like to.

You can always ask, right? You can always reach out to me and say like, hey, you got something for this? Is there, uh, you know, oh, I remember I was talking about this, I was watching the video, you got a link for that, I will help you out there, right? So don't worry about that.

Um, yeah, yeah, like, in re-engaging with my students, always a pleasurable experience for me.

I'm like, yeah, cool, you're doing stuff, this is good.

So that's cool.

Um, the other thing that, uh, we love, so you can, you can go through this, uh, at some point I'll try to make a little time in the class later on, but, um, if you can fill it out it'd be great.

I just put in the Zoom chat a link to a um a class survey.

Uh, and so this helps, helps me, this helps, um, sense and respond, uh, kind of learn what, what are we doing well, what do we need to work on, uh, where else should we go with this material? Um, is there anything else as well that you would have to say about the class, um, that others might want to know? Um, and that kind of thing.

So, um, that tally link that just went in there, it's short, there's like a couple of NPS scale items with some opportunity for feedback, uh, on those pieces, um, and any other recommendations, improvements that you'd have to suggest.

So if you could go through and give us that feedback, be super, super helpful.

Um, also down at the bottom, there is a list of, uh, a place where you put your name and email for a certificate of completion.

I think I was messaging Natalia about this earlier, um.

Yeah, so that do do do do, ah, she actually wants you to fill that out in order to get your Credley badge.

So there's a certificate of completion, I think a PDF that you get and then as well, um, we've worked with or we've uh connected with Credley so you can get a little badge on the post and let everyone know, like I've done this thing.

OK, thanks.

Um, all right, one last bitches, I, uh, I want to talk with you more in the future.

I know you said like you'd love to get more people engaged in this kind of thing.

Um, lean product management would maybe be an interesting class there.

Uh, the product discovery course could even be good, right, just as a note.

We can catch up on what more of what it's about, but this class is is um really about being a product manager, right, and leading into discovery.

The product discovery course is actually more for teams that would be doing the work all the time and so still cover a lot of these similar concepts, but I think like we don't, we don't get it so much in a strategy.

It's kind of more assumed you'd have one.

there's still going into proto persona, still going into experiments, still thinking about how you incorporate this into your workflow.

And like, and going through that whole lean product canvas of like popping in all those pieces.

So if that feels more appropriate for your team, um, You know, that's cool too, but let's, let's maybe find a time, um, when, when things are a little less crazy for you to chat about them further.

I, I, I was actually thinking about the course on OK hours if you do anything with.

You know, a deep dive into OKRs and how to do that, um, that might be useful for the rest of the team, even if they have no no idea about anything else.

Um, that might be useful as a tool.

Um, and, and people who are.

People who take the product management course might be able to guide the rest of the folks with just the OK at least that's, that's, that's the idea I had in mind.

I don't know if it's uh if it's valid.

I wanted to chat with you about that too.

So we can, we can talk about it offline.

It is, and I'm working with, um, so two things.

I mean, I can always talk with you about a private class offering if that's something you want to do, um, but I am scheduling some public OKR classes um for November, December time frame, um, and I'm looking into 2026.

I just haven't looked at my calendar for that yet.

Um, those, it's, uh, the course is about half the size.

So, um, with that, I'm gonna do 23 hour blocks, um, and I'm gonna schedule some that are like a week apart, but I'm also gonna schedule some that are like do your first class on Monday and the second class is Wednesday, so much more, yeah, yeah, more compressed time box, you know, time frame right there.

Um, so yeah, it, it is like a one day in person or like a, you know, that Jeff and Josh do.

22 hour classes and then offload they say about half the material.

I, I just put an extra hour for one if we get into questions, topics, things like that, we have the time and if again if we don't need it, we don't need it, we just end early.

It's never, no one's complained to me ever, ever ending a class early for the day.

Is this a, is this a uh are you going to publish this or is this more like a private thing for The ORs class.

I'm going to publish it.

Um, so, so yeah, so we have at least with Lian within Lian product management and OR's class, um, it is essentially who does what by how much.

It follows that book and patterns.

Um, that class, so some other interesting things about that class, we dive into OKRs like 3 different times, right? We go into A lot of the exercise is about outcome impact mapping.

We use that to drive um OKRs a little bit more.

We talk more about what makes a good OKR.

Um, we, uh, and we talk about it from a, a work perspective, a personal perspective, right? Um, and so you can try in different contexts and see how OKRs can apply, um, to both product and process management.

Um.

And then the class some some key differences, other things that get added.

Uh we talked about OKR cycles of review and trying to learn and adapt based off of um our OKRs and, and know when to change them.

Um, and that course at the end of the Think if modules as a um as a quiz that got built into it, um, Sensorrespond is partnering with OR mentors.

Uh, to provide an OR mentors certification, so a third party certification of your knowledge based off of the course.

So they're still working to adapt and learn about that, but it's, uh, it's a, it's a little something extra, which I think is attached to that course, which is cool.

Um, so I don't have those published yet, but I will email you when the public ones are live and we can definitely talk about the private ones too.

OK.

Cool.

The private ones might be next year, but the public ones, uh, I'm gonna do it.

I'm gonna ask Aaron if he wants to do it, uh, of the OK one especially.

Yeah, so, uh, it'll be um a little review for you in some ways, but also a refocusing, maybe a deeper dive too.

So, OK, um, cool.

Anything else you want to chat about before we Get into today's materials.

Right.

Well, let's roll.

Uh, so, working this way, that's what day 4 is going to be all about here.

Um, we're gonna get into, like I said, we have gone from talking about strategy, where we're gonna play, how we're gonna win, the main business problems that we can work on refined multiple times throughout the course, right, the objectives that we want to uh go after, the key results that will help us know, the, um, the assumptions and hypotheses that we've been Making about our work too, right? And then trying to actually run experiments and today we're gonna talk about it as well.

How do we actually start like doing this consistently, right, and incorporating this into our work.

Um, my question for you, as we get into is your experiment.

From last time.

So this is mainly gonna be and now she had to drop early before we uh before we got to most of that discussion.

I unfortunately will report and do not complete our homework in that regard.

OK.

Um, all right.

That's right, yeah.

The purpose was in my mind as we left, right, trying to validate, you know.

interviewing what we believe are representatives of a proto persona and validating the assumptions we've built in in the mural board, how many of them actually are relevant.

Yeah.

Yeah, I unfortunately I lost track of time, and so yeah, we didn't get to the interviews and I originally was thinking I could have the time to at least quickly come up with a survey of questions.

And I was kind of at that point I was like thinking more on the execution part of it and it's like, how do I mix in kind of more like the basic yes no questions and I know, right, you had emphasized making sure that, you know, having those open-ended questions that allow you to have that story, you know, allow them to tell the story.

Um, Yeah, but yeah, unfortunately I think it's anything.

OK, OK.

Um, so it happens and you actually, you, uh, you, you covered the key point, right? The whole point of the exercise is to learn quickly, right? To learn if your assumptions are correct, to learn if these people really exist, um, and to do that, um, quickly and, and cheaply in a way, right? Like the, uh, if you were to run this.

Take an hour, 2 hours, 5 hours, I don't know, something like that, and you'll learn something, and it's a really, uh, when I've done that in the past, it's a really cool experience, especially if you don't interact with your customers ever, um, which I haven't heard much about there like, but that's, you can get all this cool information in in a short period of time and tightening that feedback loop before we decide to make all these other investments and build all these other things, um, is so valuable and so powerful.

Um, now, I want to talk a little bit, uh, not, not that I'm like angry about not doing the homework, but I just want to talk explore some of the other things about, um, what caused it not to happen, right? So you just said, you know, I kind of forgot about it.

Sure, as you were talking before, you got this big thing coming up today, uh, and there was a lot of other work that was associated with it and some changes there.

Um.

So, you know, some of those are like, there's chaos that got imposed upon you.

Um, or like just the timing wasn't quite right, but what, what is there anything else that prevented you from doing that kind of experiment? I mean, I know, at least double checking um based on what was in the recording and at least referencing the original slide deck that's on I think if it.

Um, The thing the main thing was just trying to remember what the actual homework assignment was and then they.

You know, since I wasn't immediately executing it on after a previous session, you know, you know, after that weekend, it's kind of like.

I know we had to execute the experiment, but what was the exact I know we had discussed interviewing, but then it was kind of like figuring out the exact details of like, you know, what I just described.

Was it to, you know, validate the proto persona and the assumptions we made there? Was it to follow up on the questions and the different topics we had put into the mural board regarding what do we think are the key items that we need to validate? Mhm, yeah.

Yeah, that that's a good point and like how do we know when to do that? So I, I know I highlighted week one when it was all new and fresh and we're like, whoa, what's going on that the um putting some time aside, right in the, in the middle of that week to do that is important, um.

For our next group at 0.

I will, like, I think I'll try to get that information out as well earlier, right? Um, and including in a post, uh, pre-class email like, hey, get ready to schedule some time um to do this, you know, this kind of experiment because of the, then you won't forget as much, right? Um, and that's, that's just a different aspect because you might even want to schedule as you're doing your regular work.

Right? And so we'll talk a little bit more about that today too, of like, when do you schedule this discovery? When could you do it? How do you make it part of your general flow? I, I have two other things.

The one is the project management thing.

This is for my personal, if you task management.

One of those is that you, you mentioned this in the first class, you said, you know, you need to schedule this right now.

But because it was so far away, I pushed it off and with the mode we're running in right now, anything that's not uh needed in the next 20 minutes is not fire.

So I, I, I pushed it off.

That is completely on me.

Uh, the second thing was that we even yesterday we were at the, at a dealer um where we might have.

We could have, we even discussed uh interviewing one of the technicians there because we, we know that they're likely gonna be dirt bike riders if we could have interviewed them there.

Um, I don't know about Aaron, but in me, I was, I sensed.

Just as you asked me this, I, I, I, I went back to that moment.

I distinctly remember having this hesitation to just go talk to them about it.

Uh, and, and I had this thing that, oh man, I have to prepare for this.

We have to get a, we have to get a questionnaire out, we have to get the survey out, and we have to prepare a whole list of things that we need to ask them, and then we can go talk to them.

Uh, I wish that weren't the case.

I wish we could have, you know, even if it was, uh, we had, I think we had enough of their time.

We could have, we could have spoken to them about this for like 30 minutes, uh, easy, and we could have also done that with, uh, some of our colleagues.

Uh, not yesterday, but, you know, uh, earlier in the week.

Uh, there's there's some.

At least prioritization things that need to be fixed there, at least.

For me.

Yeah.

Um, you don't.

So, I'll give you an analogy.

Firefighters should spend most of their time cleaning the engine.

Um, you think of like, when there's a true fire.

You need to go fight it.

Uh, but the, but hopefully, we're actually not fighting fires all the time.

We're not like constantly in a state of chaos, um, cause that it's just that, that will wear on you over time.

So I hope, you know, for your, you said you're in this mode right now.

Um, think about.

Not related to this class at all.

My personal recommendation is, is, is think about when can you get out of that and how do you, how do you get yourself out of that? Like, what are the things that will start to trigger, we need a shift, right? Or what are the things that trigger, like, if we stay in this mode.

Bad things might happen.

Um, cause I've worked with companies like that, and I've talked with a lot of people and it's like, you just, you've seen people leave, they get burned out, they, you know, they can't, the, the quality of workshops.

Um, we don't know, we actually don't have focus on what we're working on because we're just used to respond to emergencies, um, so.

That's my suggestion is um To, you know, if you can take a little bit of time in the near future, think about how you'll get yourself into the next mode of operation.

That's fine.

So you mentioned Hesitation.

Um, and he said you, you, and then it sounded like part of that was you wanted to feel more prepared.

Yeah, definitely on my case it was.

You know, the uncertainty of like, well, I don't want to feel like I'm wasting, you know, their time, um, especially since, right, this was originally uh a dealer visit we're planning to do, um, for following up, you know, with the getting a beta version out of our application to the technician, our main user, and getting their feedback and kind of doing an AB test of the existing application versus the new one.

Um, that was our original plan, and of course, We had several roadblocks with that, so I think we're already kind of weary from travel and driving for an hour and a half up to to the location, up to the dealer, and then the fact that we forgot certain things, um, like the programming cable we needed, it's like, well, we're already so frazzled, and it's like.

OK, what's one more thing.

Mhm, yeah.

Yeah, it's um I I think it's fair, right, to force it into a situation that what didn't make sense, uh, might have been awkward, and then it adds to your stress, right, in that kind of way, um.

But the, the, the, the crazy thing is that um from my personal experience, uh, you're, you are aren't often wasting your time, uh, their time, and you, the planning, you do need to plan, but you maybe need less planning than you think, right? Sometimes just popping over and and just asking like, hey, you know, seeing somebody like run around, do you, do you commute often, you know, or like do you ride on these shows, you know, like much.

Are you just here today with a friend? OK, cool.

Well, not my target, right? And that's a key piece too is when you're starting to talk with these kinds of people and, and do these types of sessions and while I You know, the uh years ago before this class existed and before I was doing this kind of thing, I worked with groups um in my previous consultancy, uh, that did stuff like this.

They would just kind of like roam around and say like, hey, they're working on mobile apps, do you use these, like, do you use the um power tools at home? And they're like, um.

No, it's like, OK, not my person, move on, right, so finding that, OK, I, I could ask you my questions in this interview, but it doesn't make sense.

And so when you start to think of your personas and, oh, do you use power tools on? Yeah, yeah.

What, what kind of tools? like what kind of work do you do, right? Oh, you know, we got a drill or whatever, and I do a lot of woodworking.

I, oh, interesting.

Tell me more about the woodworking, right? And uh what, you know, what kind of things to me? Tell me a time that you just struggle with settings and so this was all about um.

We were building, we're helping Milwaukee Tool uh build an application that configured their tools.

It's called OneKey.

Uh, so if you've ever seen, if you ever go out and see Walky Tool and Oneey, the company I used to work for created the initial launch of that.

Um, and so it was neat just like, and why, why does anyone think about using their phone and a power tool at the same time, right? So, um, yeah, you can just, you know, sometimes chat with people and and see what's going on.

Um, I did a little bit of that for you over, over the week, um, over, over the weekend.

Um, so we talked about the company and, you know, because just because you attended the class with Jeff and Josh on Lean uh Lean AI, um, they were curious of what's going on and everything, and I asked a little bit about getting around and like, oh, do you all walk? You guys taking the subway very much? How you do, you know, do that and somebody mentioned, oh, well, you know, we have our, our motorcycle.

Oh, what, what kind of motorcycle is that? Right, um, oh, how far do you go? What do you do? And like, oh yeah, we don't actually walk very often here anymore cause it's so easy to just hop on the bike and, you know, even put the two kids on the back and zip around.

I'm like, oh, OK, interesting, right? And, you know, started asking, and some people are like, I don't speak good English, and I can't help you, but OK, so but it was just neat like I, I had a couple questions and started figuring out like, yeah, they're they're um.

Uh, they just hop on a bike and take it for short jaunts.

Um, no one talked to me about Uh, like little tours, like out to certain areas on their bikes.

They didn't do that kind of thing.

It really was in the city or within different districts and such, um, and, and they talked about like getting around and it's easy and like, and we just, you know, you can park them anywhere.

I was like, really? Yeah, like any, I mean, we just ran on the sidewalk, right, right, right on the street, like all the time.

That's, you know, just go and you don't have to worry about finding parking or what you're, you know, or how far it is a walk from the the station, right, the subway station to wherever you want to go.

You just park right in front of the thing and and you're good to go.

I'm like, really? And so I then went to the next part of my experiment, which was observational review.

Yes, there are bikes.

Everywhere.

Right, like, literally everywhere I see people on bikes and putting bikes out and about all over the place.

Yeah So I thought that that was really cool.

They're just all along the sidewalks, um, and in various other little spots and uh I think, yeah, that was my last picture of the mics there.

Um, so it was neat, like they're parking all over the place here, and I started learning something about this, which I thought was neat.

You're talking about EU commuters.

I'm in the EU.

Here are people commuting on their bikes, and there were some patterns that I noticed, right? Um, they do just park on the sidewalk.

In the evening, it looks very similar.

OK, uh, I didn't get a picture of that.

There were less bikes, but there are still a lot of bikes just out there with these like they got a little lock that kind of comes down right here, that connects to the handle.

Um, so there's a lot of them they're just sitting out there like that too, uh, which made me think a couple things like where, where do they fill them up? Where would they charge them? If this was the case in EU, right? Um, how far would they need to go? They, they can't just bring them into their apartment.

The, if so, the elevators were really tiny.

Um, there was, there was a lot of underground parking, so I didn't see what that looked like.

So I don't know how many people were moving, uh, underground at night.

Um, I also noticed that this bike here and this bike here, and one more way back over here.

Or some of the few like what I would say look like standard kind of bikes, they look like most of them had more of the scooter, more of the scooter style, yeah, yeah, and so people were riding around on a variety of scooters, all of these little boxes on the back, right, so many of them like that, uh, just sit around, go to work like she's just for work, right, right, and then got nice shoes on and section this guy in a suit.

I was like that's, that's cool.

Um, so that maybe tells you a little more like, what, what is an urban commuter in Europe expecting? From their back.

So I just thought I'd share that with you.

You know, you're so focused in your class and you can and you mentioned that I was like, I can add some context, you know, I'm not running the same experiment, but I thought it would at least be kind of neat to bring that up and see what you want.

And, and one of the key points is right, you highlighted, you know, I think at least I was approaching it from like it's a very formal structured process, but right, you're able, you're showing that, you know, just by going out into the street and having a general idea of like, oh, you know, I wanna find out about these things, just go out and do it and it doesn't have to be like, yes, I need a step by step like this is the exact list of things I need to cover, and this is, you know, the exact feedback forms I need to follow.

I think yeah, allowing for a more natural, you know, just.

Not, um, spontaneous is the word I'm looking for.

You know that, yeah.

Yeah, the, um, the, the downside of that is if you just if it's totally spon uh spontaneous, uh, the structure of your results is not as good, right? Yet, um, I I certainly was at a point where I needed structured results.

I was looking for themes.

Right, like, what kinds of bikes are people on? Where are they? How many people are actually riding on bikes? A lot, right, a lot in that area.

So like, oh, OK, cool, right? So you, you can start finding out some very quick themes just by making some basic observations and a couple quick conversations, and then use that to design more structured.

Interviews or surveys for round two, right? Because even just that very light touch proves whether you need that or not.

Yeah, right, you know, if the, if you're able to identify a theme, you know, that's opportunity to be like, OK, do I need to actually then formalize something and dive into that theme, that's something I need to now track following forward or what part of the theme is that theme tied to.

Yes, but, yeah, exactly true, right, like you've identified, yeah, there's a theme of You know, there's a heavier bias towards the type of scooters versus the traditional motorcycle we think of in like North America.

Right.

And, and what, what came out of the question for me is there's a bias towards them.

Is that what they want? And would they accept a different kind of bike, right? Cause it clearly there are some that are out there, and I walked past a couple of bike shops, and it seemed in each bike shop, I wish I got a picture of that.

There was a blind.

Right? But if there's, there's a blend, and they're not like, are they selling so many of the scooters and just a few of these, or did I just not see them or like, there was obviously gaps in the information and more things to go and discover.

Um, but, you know, observation, I'm like, OK, well, if we were to build a bike like this, whether we wanted to or not, there's clearly like a bias towards people wanting to ride something like that.

It feels familiar, and we can just off we go.

Right, um, and you might find out, you know, you could run experiments like that too, or, you know, is it just, why is, why is the market used that? Like why do they care? or you know, how many people are riding on bikes with multiple people.

If it was an adults, I didn't see it once.

If it was with kids, I saw it often.

Lots of kids riding on the backs of bikes.

Uh, one or two deep, uh, not as many as I saw in India, like, whoa, like, like there are a lot of people on bikes everywhere and, and like to, you know, at least 2, maybe 3 adults and sometimes like, you know, an adult and 3 kids all riding somewhere.

I'm like, how, how, how, I don't, I don't know.

Yeah, I remember when I was in India, my coworker and I, it's like at some point we'd see, yeah, you know, you see an entire family, like 55 people on a on a motorcycle or a scooter we're just like.

I don't, you know, safety aside, I'm just impressed they're able to pull that off.

Uh-huh.

Everyone's leaning in the right way, it's at the right time like I, it, it was impressive.

The, the thing is in India, it's, it's different because the speed of movement is also much lower, so it's not like uh no one's, no one's riding at 90 miles an hour.

It's At most 20 kilometers an hour or 25 kilometers an hour.

Yeah, when we, when we were doing 40 kilometers an hour that felt bad in India, and I was like, and it just it blew my mind when I realized later how what that speed was.

I was like, oh, OK.

Yeah, most times you said it's at fastest 25 kilometers an hour, and more like 5 to 10, right? OK, um, so a little bit of, of insights I say, just, um, get out there, talk to people.

it's amazing what you learn and um here's the other one, right? uh.

Yes, it, it feels uncomfortable, especially if you've never, if you don't do it, right? If you haven't done it before, if you're not used to it, the more you do it, the more comfort that you'll get in just having these types of conversations and just going out and talking to people.

And what what helped me with those is, um, you'll be amazed how few people get asked questions.

About like their own habits and like what they do, and for like just general, you know, anxious showing up as a as a human, curious about what they do, and seeing if you can help them in some kind of way or or change things for them in a positive light, um, they're often more than willing to provide that kind of feedback.

If you know, you were to come in like, hey, you know, um, we're from zero motorcycles and, you know, we make bikes and we're trying to figure out how we could sell more of them to you.

Yeah, please walk away, right? It's like, hey, what's the problem that you're having with bikes? And then when you, when you focus on that, people feel seen and um and they often appreciate it, and it feels good to like talk to these people that way.

OK, um.

I'm gonna move us on to the next phase, if that's OK with you guys.

Mhm.

Yeah, OK.

So, uh, let's, let's get into some tactics here.

We've done some a bunch of strategy and we're gonna get into a little bit more on, like I said, how do we do this work? How do we start making these connections, how do we incorporate this type of product discovery work, um, and, and focusing on outcomes for people into our general workflow.

Uh, so, first off, we're gonna talk a little bit about OKR based road maps.

Um, I got a question.

Do you do anything like this now? No.

OK, which is, which is why we want to do the street.

Awesome.

So, uh, there's, man, there's, uh, this reminds me of a time.

Did I tell you about uh the Gantt chart product owner insecurity? Did I tell you that story? OK.

So, um, I, I worked with a product owner once.

He was in charge of security at um uh an investment company.

And he uh and when he took over, it was like he was part of a small part of security, and he ended up taking over everything, um, because he had a good view of like, where do we need to be and how do we prevent intrusions and all this stuff.

And so he was asked by leadership in his organization, uh, or by the executives to Give up, we need a roadmap to look at where, you know, where the initiatives and what's going on and, you know, lay it all out quarter by quarter, what will happen, OK? Um, and so he had this thing, um, 8 quarters long, 2 years' worth of work.

I think it was, yeah, I think it was, yeah, 8 quarters long, 2 years' worth of work, um.

At least 10 different initiatives, uh, at the time, only 3 that he had direct control of.

But the others that he knew uh impacted and like correlated, and then, you know, bubbles of like when they would get worked on, which quarters, what was happening and kind of gang chartish looking thing.

And he added one extra row at the bottom, and it said confidence.

How confident am I in this plan? And the first quarter, the confidence was medium, right? 0 to 3 months medium.

Quarter 2 was low, and then every quarter after that was none.

So he delivers all this and then uh Ahmed, one of the the leaders in there, love this guy.

He goes, he says, um, so I gotta ask you about the confidence here.

You have prac you have practically no confidence in this entire plan that you just showed us.

He goes, yeah, right, Bob the proctor, yeah, so why did you do it? Like, why did you, why are you showing this to us? And he says, because you made me.

You said you need to deliver this kind of flow and show us what's going on, and I just want to show you how useless it is.

Because so much is, we're gonna learn so much along the way, and it's gonna change what we do.

So I, that's why I have no confidence that this is going to happen at these times, uh, and that this is even important.

And he's like, huh.

OK.

Gotcha.

So, It would have been better if he did something like this, OK? So, um, Another way of of running road maps.

Something I actually, I think is pretty cool here.

Uh, time, right? Time going from earliest on the left to furthest on the right, uh, we should start laying out the types of things that we've actually been pulling together here in this class, right? Like our longer term strategic theme, the objective, the where do we play, how do we win type of stuff, right? Like actually listing that out, uh, this is what's driving all of these ideas that we see below, right? Um, and then per quarter, let's start listing, what are the key results that we're aiming for, right? What are the quarterly, OKRs, the quarterly goals, uh, that we're trying to go and achieve? Because some things need focus down, some things we maybe will start focusing on later, or we need time before we can get to it, or there's other things maybe we'd want to build, so.

You know, let's start seeing what makes sense to do now versus later.

We want to know what are the features that we're looking to discover, right? Um, and the discovery questions that we can go out.

So, right, longer term strategic themes, uh, as I mentioned earlier, you know, your objective should make and your business problems should make sense for your context.

So if you have different teams with different focus areas, it's OK to have multiple road maps here, uh, OK our roadmaps for different groups.

Right, um, then again for quarterly goals, uh, we, we commit to these on a quarterly basis.

What I think is important here is, and I said, I'm a big scrum guy, uh, and in scrum we talk about committing to goals and forecasting work, right? Because specific scope, I don't know.

It's like it's uh it'll be done when it's done, it's the one thing that I know.

But committing to a goal gives me focus.

I will do everything I can to try to make that objective happen.

I can commit to that, right? Um, so that's, that's what I see as the difference there.

OK.

Then um we think about quarterly, you know, the features, right? Things we may build, not things we're going to absolutely build, things we may build, um, are, you know, all the hypotheses, all the best guesses about what we can build in order to hit those goals.

So going with what we talked about last session, right, here's our OR now, what are the things that we need to learn and what experiments might we run to go and do it, um, write those hypotheses about if we Made, you know, um, hot swappable charging, you know, or batteries.

If we, you know, um, I'm trying to think what the all the, what all the other, um, experiment, you know, features were, but that kind of stuff, that was what we can list here.

And then the discovery questions, the things that we need to learn, right? And how we might learn those, we would list at the bottom.

OK.

So, um, the further we go in time, the less we're gonna have on the board.

Cause it's just we don't know, right? We can We can see out there, there's maybe some things that we're gonna build and when we get really far out, when you're you're out, I, I don't know, we might learn so much, we might not even go after that that quarterly KR anymore because we've learned earlier that the, you know, maybe in this section that this doesn't even make sense and we need to shift.

So, you know, we don't want to.

Yeah, we can't predict it very accurately, so we also don't want to make a huge investment.

And that stuff.

OK.

Um, and every quarter, minimally, we should be checking in, right? see how are we doing? Did we see any change in user, uh, outcomes, right? Did we see their behavior shift that, uh, indicate any progress towards these key results or not, right? Um, and then what might we do, how might we shift? The next, you know, quarterly, uh, goal based off of what we learned.

Um, Jeff gave me a great example when I was in class with him about this, and I just, I want to recount his example, um, because it was so very relevant.

He said like, as they were working through this, they, um they were looking at a really low.

Um, employee and job seeker connection rate while he was working at the ladders.

So the ladders was all about trying to go and find job seekers, um, you know, and, and who are paying for access to this platform to connect with employers who are looking for good people, um, you know, like you might do on LinkedIn or something, and, and say how do we find and pair these people up to get jobs filled, to give people these opportunities to help employers staff their teams.

Um, and this connection rate was really low.

Um, it was coming at about 14%, which he thought was like, that's really bad.

I mean, they're both paying for this, and we have that low of a connection rate, like this, something's wrong, right? We got to change this.

So, uh, they set a goal of getting it up to 75%.

We want 75% of the of the employers and job seekers that we're trying to pair up to actually have a meaningful connection.

Um, and, and so it's ambitious, right? It's way more than 14%, but it's also not like everyone will do it, not 100%.

All right.

And with that, they, they ran a couple different experiments and things that they built to try to make that shift.

They did some discovery along the way, and in one quarter, they got up to 68%.

OK.

So what do you think about that? Set a goal of 75, they got to 68.

What comes up for you? That was the dishes.

It was fantastic.

That is, that is the kind of stuff that we should go after.

But I, I'm wondering how did he get to 75%? What did he based it off of? They just Did that like big, big audacious number.

Let's go, I mean, if we can get 3 out of every 4, you know, people pairing up, it'd probably be worth their money.

Well.

Yeah, and then they got, they didn't get to 75, they got the 68.

Yeah.

Erin, what do you think? It just shows an environment where You know, it's, it's OK to say that, you know, the The objective itself, right, that's technically a failure.

But as a whole, right, being able to say that realistically, yes.

Um, operationally, this is a fail, this is marked as a failure, but As a whole, right, we've gone from 14 to 68.

If you're ignoring that and saying that's a failure, just because we didn't hit what we initially said, then, you know, we're missing the plot here.

Yeah, very good, right? And you're right, like, it grew, it grew quite a lot, right? And no one, no one guaranteed 75%.

They just said we're gonna try to see if we can, we're gonna do everything we can to get as close to that as possible.

And they came pretty close and they made a huge improvement along the way.

And so at this quarterly check-in, their leadership took a look at this and actually that's pretty awesome.

Like 68%'s pretty good here.

It might grow a little bit more.

We could probably do a whole another quarter, just trying to hit that 75% if we really wanted to, but why? Um, so instead, they just said it's better to move the team onto something else, right? Rather than try to squeeze out that last 7%, and they shifted, right, towards a different key result and to a different goal for the next quarter.

OK.

Erin, are you there? Your screen froze.

Your face and your eyes moved, so that's good.

I'm like, not just on my side.

I'm looking at on the.

And the little thumbnail.

Yeah, I, I wonder if it's, yep, yeah, there you go.

Something happened.

We will see him again here soon.

I said laptop just made a funny sound and my screen is frozen.

On your side? He, no, he said, he sent me a message.

He's he's texting me.

Oh wow, this laptop looks bad.

OK, it might take a minute.

That's fine.

OK, we will, uh, yeah, we're, let's see, what's the next thing I just want to take a look at where I was going with this next so that I don't accidentally do things that will take them away.

I think it's just like the have everything fade to gray over there, you know, like, don't worry about that stuff, focus on the stuff on the left.

Um, Yeah.

Yeah, OK.

So we're, we're gonna, we're gonna jump into an experiment here in a minute, so I, I can really use him being here for it.

Uh, or the exercise.

So, um, I, thoughts, questions, anything you want to chat about before he comes back? My trouble with this is, how do we negotiate this? With people who might not be.

I'll, I'll give you a real example.

Um, we have new bikes coming out.

That bikes Um, sales folks want.

Apps for that.

The SVP of sales is asking me, how the heck don't we have an app for uh a dirt bike in 2026.

It's not 1990.

So from that point of view, sure, maybe that's uh there's some validity there but the thing is we don't know what we don't even know what value we're adding to our customers' lives, the benefits we're delivering with the app that we have for our.

Of, you know, full on expensive premium bikes.

And to just release an app for the heck of it for uh.

For a dirt bike is inherently risky because if at best what we'll have is a check box.

Because we haven't, we don't know what value we're going to deliver.

But the thing is, if we don't do a good job of even the checkbox, the, the trust factor is gonna take such a large, uh, hit.

It'll just torpedo our reputation straight away.

So we, we did discuss all of these things and given the constraints we had and the, you know, the time we had in to figure out what value we could actually deliver with the app, we said, first version, let's get a rock solid bike out, and then we can figure out what what people are using it for.

And then we'll see if we need to add, add an app.

Now, I'm, I'm.

Whether or not an app is needed, I don't know.

I want to find out, but it sure as hell isn't.

Isn't gonna be found out by someone just telling me how, how, why is it that we don't have an app in 2026.

Yeah, um, you negotiate that.

Let's say I, I have an OR, uh, I, I become an OR expert tomorrow.

I want to find out how do I help me negotiate this thing and drive, is it an education problem to be solved? Is it, uh, is it, uh, is it me connecting it to a dollar value that, that they might be able to see? How do I do that? I can't be.

No, uh, so you, you brought up, you brought up actually some really interesting points so like you don't know, um, and There are, there are two conversations that come to mind, um, with the salesperson, right? And I think walking through with them on this discovery is actually really key, right? Not just coming back to them with like, here are the things we're gonna do cause discovery is an activity and building an app is an output, and neither one of those might actually change the outcome, which It's the first question to to work through with the um the the the user there, right, or with the salesperson it's like you're asking for us to create something, some output, right? But what is the thing that we're actually, what do we expect that it might do for us, right? Um, what, what's the goal? Well, if it had a, what would, you know, why would you have a bike that doesn't have An app.

Again, I don't see user behavior changes.

What does this actually create for our users, and what does that in turn create for us? So we had an app, right? Um, we, we have a a hypothesis.

About some OKR and so you know, you can maybe help them focus on what is the what is the OKR that's important here, right? What that users um are more likely to buy and and pay for extended service with their bike if they have an app that lets them manage things more along the way.

um, and we can see that by a certain percentage or we expect that um if, you know, as a hypothesis like.

Uh, if you're talking about sales of bikes and things of that kind of nature, by having an app, we actually create a more attractive version of our bike.

By making it by having an app that allows users to modify the ride experience of the bike, we will sell more bikes, right? I think we were talking about that in terms of the, the dirt bike one the other day, right, our last week.

How does it help them get to the um to the, the trail where they're riding? How do they tune it for different kinds of trails, right? You know, how often are they using it, those types of things.

So it's easy for a salesperson to say, if I had it, they would.

I'd make more money, right? Yes, so we, in fact he had a, he had a great anecdotal reference point too of a dealer dealership somewhere in Utah.

Where people walk in to see the 0, they're interested, they want to see what it is.

It's uh it's slightly more expensive than the competition, but they can see that it's a premium product.

And then the technician goes or the salesperson goes over a list of all the things, and then an app is clearly missing there and.

They don't even wanna know what's in the app, right? He, he's, he, he, he's really good at this, and he's telling me it's, it's an unemotional purchase.

It's not like, uh, it's not like they're, they're thinking deeply about this and then, you know, considering pros and cons and so on.

They hear, uh, no app, and then it's, it's just one more thing to, uh, one more thing that sticks out against us and then we have to.

That is what they're dealing with, so I don't know if it's a marketing thing.

Yeah, well, there, there's an interesting question you can dive into that with them, right, of, um, like I said, it, it's an investment like getting an app is not free, right? So we also need to make sure that building whatever set app, if we want to do it actually creates the type of results that we're looking for.

But before we build it, um, one, we want to know if we need to build it, and two, we need to know what should go into it, right? And that's, that's, that's many of these items.

Right, um, and that might tie to a key result of, um, you know, like increasing customers, um, uh, connectivity with the bike, you know, um, you know, something like that.

Uh, and so some discovery questions are like, yes, what, what would they do with the app and how would we go about it, and like, what are the kinds of features that they would find it's missing.

But another interesting question, right, one for your sales team and two for your customers, um, is, do they care about an app? Right, and so if you were, if in the sales process, you're listing all the features.

The discovery question I would want to have right off the bat is, how, how often are people asking for the bikes app? Right? Cause if we just tell them there's no app, then they feel something's missing.

If we, if we just tell them all the features, how often do they respond back with and how do I like, and, you know, how do I reconfigure the bike? Where's the app? it's like, is it just a general question like, oh, there's a mobile app, you know, tell me more.

Like I wasn't aware.

It's like I thought it was just a motorcycle, right? I thought it was just a motorcycle.

And if you tell them about the motorcycle and you don't talk about the app necessarily, right, cause it's, it's not a feature of that bike, um, but it's a feature of your other bike, so maybe they see that gap, and then if how many times do they ask, how much do they care? That would be a discovery question that I'd want to get into.

And if they're asking a lot, I'd be like, oh, you know, where's the app? Actually, we're we're currently looking into that.

Now we get another discovery questions.

What would you do with the app for a dirt bike versus a road bike? What would you, you know, what, what are the important parts of this here? What kind, how do you ride your bike, right? And I can go into all these other questions that then, like, I'm not saying we don't, we're not going to build an app.

I'm just saying when we go to make this investment.

I want to make sure that this investment is actually worthwhile because customers want it, and it's targeted based off of what customers need.

Right? So really, that will help me understand their personas a little bit better, that will help me understand what are the different things I can do again, to get to that same OKR just in maybe a different way, versus like I have to build that.

Um, so I would do some discovery in that way and and do it with the salesper, maybe even get them to help you figure out what are the experiments to learn about customer behaviors cause Sales people are most focused on the customer behavior of get them to buy.

But now we can say like, what are the different things that they that we need to know about them that trigger their impulse to buy.

Mhm So that's where I would take it if I were you.

That's that's actually.

There's some pointers that I can try with uh.

They had a sense.

OK, some of it also when we get into the the next little bit here, um, we might find other ways to to show that that work a little bit more, um, but it's not totally listed here yet, OK? Um.

Now you're back, Erin.

I want to get towards our, our next last little piece here before I turn you all loose, uh, with the experiment.

Um, so just something else to consider, we've gone through all of these 4 things so far.

Business problem statements, right, um, and for that affect our users and, and talk about why, what's the value prop there? The OKRs, uh, what are the objectives, uh, what are the outcomes that we're looking for the key results, the hypotheses we have about how to go and solve them and understanding the risks towards actually working on that hypothesis like hypothesis, we need an app.

That's a hypothesis.

Right, so what are the risks of doing that? What are we, uh, what's in the app, how much you needs to get built there, all that kind of stuff.

So, um, we bring this up to show, right, uh, business problem statement is kind of a bit above your longer term strategic theme.

It, it might, I think there could be an error that goes here too, but, you know, your top level OKRs might also fit into that longer term strategic theme, right? They also definitely fit in the quarterly goals.

It all depends on which ones you have, how they tear with each other.

You know, that kind of thing.

Um, your hypotheses are all of the things that we're considering building, all the features that we're considering them, and the risks are in your discovery questions.

Right, so all these pieces that we've been talking about, you can map to a board like this and start to see where might they fall out.

And what does that mean for the direction that we're taking is? How do we use that to communicate with salespeople, with executives and say we're not just doing this stuff because we want to do it, we're doing this stuff because it helps, we think it helps us achieve this goal.

And here's what we're going to work on to figure out if that's worth making an investment in, so we don't waste a bunch of time and money building the wrong thing.

Right, so, I, I've found in my personal experience that that that kind of tie, um, It's what your higher level leadership are certainly thinking about.

They might ask, like, what are you doing? What are all the activities, what are you building? I want to see the detailed plans and all that kind of stuff.

They are somewhere trying to relate to how does this keep us in business, how does this find us new clients, what do they care about, what are they buying? What should we, you know, what do we need to do, and how do we not make the wrong investment.

So if we start doing some of that work for them, show that like, yes, actually, You're doing that up here and we're starting to tear that down into our teams work in certain kinds of certain areas of work like diagnostics tooling, um, the OS, whatever that might be to help users get a better outcome, uh, and to help us get a better outcome from that, or a better impact.

Um, it makes it a much better story, like we're speaking their language, right? So I think that's, that's a really cool thing to do.

OK.

So, Why don't we go? Yeah, hypothesis, turn this into many things.

Um, let's build a road map.

And see what you can come up with.

For your cares and hypotheses, and where might the next two quarters go.

So maybe this you might even be able to use this to help you get away from the chaos and the firefighting that you're doing right now.

OK, uh, so here's what I do.

Let me pull up the mural again, um, and just get you, you're both already in there.

I was like, OK, I'll get you the link, but I don't have to worry.

Um, over here on road mapping, um, we've got an opportunity to start listing that out.

This talks about now, next, future.

Um, you can consider those to be quarters or something like that.

Um, that's just another way that if you don't want it to feel so.

So rigid.

Now next later is another way to, to look at that, that bit there, right, to go and say, here's what we're focused on now, this current quarter.

Here's what's focused on next quarter, this is out there, right towards the future, OK? Um, so how about this? I am going to start, uh, a 10 minute clock here for you guys to work.

We can go longer, but I'm gonna check in on 10 minutes just to see where you're at, um, between some of this mapping and what else you discover as you start to mapping it out.

Like maybe you find there are other features that we need to work on.

Maybe there are questions that you learn to discover, and maybe there's even a no care or two that you want to write, um, uh, in addition to what you came up with before.

OK, um, do you have questions for me on that before I turn you loose on trying to map out your OKR roadmap.

What's AML? or is that AIML? Oh, this is just uh over them for AML analysts.

I don't know.

What is that? RPP for the seamless effect for AL analyst identify machine learning on car rails by the end of the year.

I don't get that example.

I wish, I wish I could say, I don't even know what card rails means.

I'm gonna now I'm gonna follow up with Josh and Jeff like, yo, what's this cause that was not in my board when my class ran through this, um, but I do know one of the things that we talk about in this case is, um.

I'm talking about like a learning management system and perplexity and like the integration between those kinds too.

So my guess is there's something related to that in there.

But what I'm gonna do in the meantime is I'm gonna delete it because that's not your objective.

Um, so I can let you map your objective over here so you can start looking at your OKRs features and questions.

All right.

Yeah, more, more for me to figure out about that actually.

Let me just undo this really quick and copy this data because it's clearly weird like that paste.

All right, now, I'm gonna dig more into that.

Maybe I'll have an answer for you, uh, in under 10 minutes' time.

All right, uh, I'll stop my screen share.

You can all jump on the board.

Uh, I'll give you 10 minutes before I check in.

We'll see how it's going, go from there.

OK.

All right.

So I guess for now we can continue using the OKRs we made around the XCXP.

Right.

Plugged on let's just take the at least directly copy paste the XCXP uh yeah injection.

Right, and then also cause we can tie into, you know, I guess for you to catch up on.

When, uh, at the end of last session in the last session when Rituals walking us through the experiment planning and right, you know, so part of that is that.

You know, the discovery questions of, you know, we have all of these things we need to answer.

And alongside, right, the behavior that we wanna, uh, just.

Alongside, right, we have the, you know, 2A and 2 B's.

On this where we were targeting specific user behaviors.

In that frame, I imagine.

We would want to Great, OK, what's our, what would be a road map for if we're targeting.

Let's say, you know, you use your confidence.

What's something we would try to do now versus later.

Um, So actually The example Rich had on the slide deck.

So our business problem statement, we start off with, we want.

We want to solve why people aren't buying the XCXP at the rate.

We expected, right? I cannot copy the entire.

Doesn't let me copy it.

So I wonder if we say.

Start off something like.

Just Objectives and key results is.

So that we make.

Just kind of feel it out.

So we say, OK, if that was her.

OK.

You know, we need to validate that.

OK, we're testing.

Confidently complete purchases.

Right.

Target market confidence and purchase.

And, and when you say target market confidence and improve target market confidence and purchase, do you mean are models change or uh models and hence our goals change or are you talking about actually making a change to.

Um Well, so that's where right it ties into like when we look over here back in our OKR Swifty.

Right, 2A and 2B, you know, right, so these are some of the objectives we had tried, uh, you know, the the graft and it's like, OK, what can we do regarding converting preorders to purchases? What can we, you know, what are some leading indicators on.

Oh You know, a likelihood of that conversion actually taking place and how can we support that? Mhm.

It's kind of what the the latter two are.

Right, so what are, what are things we can target that would improve this.

And then hypothesis would be, you know, does do these actually matter, you know.

You know, it's like validate, I don't know.

Well customer referrals actually matter.

So in that original slide is indicating.

Risks.

I'm not quite sure what we would put there as a risk.

The, the risk is for 12, and 3.

12 or 3.

Which one? Are all of them? Well, because uh cause right normally at least the chain and the slide that goes.

We have the Business outcome.

OK.

Yeah.

Um, hypotheses and risks.

I guess what else can we, I mean, I guess how would we flush this out it's like.

What what details would we provide here? was it more of, more indicating Additional parallel work would be the best way to do it, or is it do we need to? Explicitly.

You know, provide more details on what each of these items are, as we're kind of creating this road map.

So What I don't What I didn't get is um.

So the initial thing, initial idea that you had of, you know, dedicating one month for each of those, that made sense to me.

So if we have.

That's a risk here.

I pop in a little earlier.

Pause the hour.

No problem.

Um, so the things you listed up at the top.

Um, identify reasons where ICXP is not being its target, improve, uh, confidence.

So where did those come from? So that was based on, I guess kind of at least rigid trying to rigidly follow the pattern that at least the layout you try to describe is Um, right, we initially.

Uh, talked about, right, I guess kind of right, the business problem statement was right inverting, and we talked about the OR is kind of the inverse of business problem statement.

Um, so going back to business problem statement we crafted with F is I guess really the last sentence we were targeting this understanding and solve what is preventing buyers from completing the purchase.

Yeah.

Um, so right, this, this initial thing, this identifies kind of more of the tactical, like, what do we actually need to do to accomplish this, you're right, identify why.

Why we're not meeting a sales target, let's say, I mean.

In the business problem statement we do directly call out, you know, we see that at least initial indicators, the conversion rate.

Um, so maybe is it better to, you know, be more specific to directly tie into the original problem statement? I know, so, um, here's what I was thinking.

This, this item almost feels like a question.

It's not a like a direct question itself.

Right, but it's uh why, why is it not being the target mark? What's, there's something to discover.

Yeah, right, um, but it's not an OK car in and of itself because it, it says nothing so much about the behaviors and and what's going on else um, there's, uh, so if I were to take what we had, right, just looking at what you've already built, um, you've got tris, you know, uh, hitting their confidence.

Let me just go to some of these here.

Um, 85% of pre-order buyers scheduled demo ride, right, up from 65%.

I could put that one up here, right? So maybe that's a high level outcome that we want to work towards, or um they complete it quicker, right? So, um, quicker would be great, but maybe it's not as important as actually doing it, right? Um, so maybe I'll move that one next, or, you know, so it's, it's in that one month pace or maybe it's, it's in this next column here.

OK, um, so I then, if I were to go towards some of the other things we did, we thought, hey, how would we, how do we do this? What are some of the things we think could actually get us there? Um, so looking at our hypotheses, we had some stuff about hot soppable batteries, isn't a is an objective right on something that would help them do that.

Um, it's not, you know, do they want to see how that works from a demo ride.

Um, what about, um, you know, switchable traction control.

Maybe that's another one that comes in here.

In fact, I can also take.

Just the item that we brought up here about, hey, we believe that, you know, this discovery, this feature right of hot solvable batteries is very tied to this one, right? That's a hypothesis that we have directly related there.

And now, how would we know that? What are the questions that we need to do for that? Um, well, we need to, what do we say? Um, we want to watch, we ask people how long do they, do they ride in a day? Uh, we need to see how far do they go before they charge.

Um, we want to actually go take a ride ourselves, maybe, right, or we want to track rides and use capacity.

We thought that was a cool one.

It was just that was more than a day kind of thing, right, of, uh, planning it would take a little longer than doing one day.

Um, That's the quick way to fill your room.

So, so what we actually needed to use there were product outcomes which are leading indicators which we actually have control over and then figure out uh features that supported that.

OK, I think I'll see, see where we missed it.

OK.

Yeah, so yeah, all these maps directly too, and, you know, I gotta say there's one good thing that you didn't do, uh, which I've seen come up in filling out things like this before, um.

A lot of times, uh, you know, people will go and say, well, I already have a product backlog, all these things, we gotta build a map, it's gonna have these things in it, whatever, and so you just start listing those in features and then be like, where are we gonna put them? And then you kind of maybe try to work backwards for the outcome.

Remember, this is about discovering what actually is needed and what works, not just a new way to manage the work that we have, but to actually try to say, this seems to be important because of this outcome and this OKR that we're going for.

Um, these are the things that we need to learn about it, to know if any of the stuff in the middle, any of these features are even worth doing.

Right? So this is here to help us minimize waste and really find the right product that drives the actions in our customers that we desire.

OK.

So kudos for not doing what I've seen so many people do, like I I'll just map it backwards.

No, no, no, no, no.

We don't know if any of that's even correct, right? Those were all other assumptions we made without even understanding why were we doing it in the first place.

Um, so, Here's a question uh for you all now, um, kind of show you where some of this stuff maps out.

Do you want to go further on this? Do you wanna try to add a little bit more on your own and get some of that experience, or are you feeling more comfortable with How this is supposed to lay out.

Yeah, just really just need to let it percolate a little bit, a little bit longer.

OK.

Uh, Mm I'm OK with percolating, letting it percolate, but I'm, I'm thinking, uh, how much time do we actually have? Yes.

I was trying to be mindful of that too.

I wanna like so I was like, I don't know if 20 minutes is really the right exercise.

Um, so here's what, let's do this again.

Yeah, yeah.

Um, do you all need a break or do you wanna just keep rolling? Can we get like a 5 minute break to uh just get some water.

I think that's a great idea.

So, uh, here's what we're gonna do.

We'll, I'll put a new 5 minute timer back up on the board.

Um, if it's less than 5 minutes and we're all here and ready to go, we roll, OK? 5 minutes so I'll see you guys soon.

See you, bye.

Yeah Miss something, disappeared.

OK.

Here we are.

30 seconds section.

Um, all right, ready? There I see you, can you hear? Yep.

I'm looking for the Uh, yeah, right.

I'm sorry, I'm I'm really slow today.

It's OK.

Um, I think, can you say something for a second? Yeah.

OK, good.

Hey, I saw your mouth was moving before but no audio came through on my side and I didn't know why, and I wanna make sure it was everything was good and I think it was just a synchronization thing.

That or might not have, I was double checking if I had muted myself on Zoom or not.

OK.

All right, um, let's go to our last little piece here.

So this is more, um, let's exercise, more concept, uh, but I think she also maybe gets into some of your questions like when do we do this stuff and how do we start getting this to fill in, um.

So the thing about integration, right? At some point we've probably got a backlog we're working from somewhere, right? List of features, stuff that we build, um, like I said, I, I am heavy in the scrum camp, um, and, and medium in the combine camp backlogs our way of life, right, knowing what's coming up next.

Um, so we'll talk a little bit about bringing this stuff together.

If we think about You know, dual track, uh, actually like dual track road maps, um, which is a concept that's like Jeff and, and Josh bring in a lot here.

I think it's important to understand, but also like where they blend, um, right, you've got the same kind of thing that we're just looking at what's the roadmap look like and and the delivery stuff is really on executing on those features, right? What are all these ideas, things that we could build, not must build, but could Build that help us go towards our quarterly goals.

We have to, we deliver on them.

We see if that affects behavior.

Are we getting closer to our goal, and then sometimes there's things that we need to learn, learn about should we build those, what else might indicate a better key result that we could be going after a missing one, something like that, and that's discovery work that we have to be doing on our product and they need to go on simultaneously.

Right, with each other, um, where, cause you, you can't just do a bunch of discovery and feel like you've got all your questions, and then let's go build it.

And if all we're doing is building things and not coming back to learn from them and do some discovery along the way, we might end up building the wrong thing.

OK.

So we're focused here a little bit more on on bringing those things in parallel and trying to run them at the exact same time, OK? Um, the original concept for dual track agile, um, it looked like this, right? Uh, and so this was actually created by a woman called Desiree C.

Um, Jeff Patton and Marty Kagan took the concept and ran with it for a while.

They often get credited for what it is, uh, but, but, uh, Jeff, uh, actually pointed out to a paper that that Desiree wrote where she had this concept of like, when designers are getting in, right, and doing all this different work, um, they, they would often make designs, hand them off to developers later on, go and do that.

Turns out we didn't actually understand it or we couldn't build it.

And so she envisioned this, this kind of dual track piece here, where um we start getting some customer data, right, and then we, we start going through and You know, at the same time, let's implement some of those things that have very low UI costs, right, um, to start getting that there while we're designing for the future, and then they sort of cross, right? We, we go and hand our test group and our design group what was built and let them figure that out and start designing for the next cycle, and we take all these other designs, we say, let's go implement those designs in the next phase, right? So that's the original dual track actual diagram.

What's, what do you think about that? What do you think about that pattern? I mean, it's basically the sense learn respond circle, it's just right, we're tying in different explicit activities together, you know that.

Talking about, you know, parallel activities need to happen, how they'll be different.

You kind of like creating a giant Venn diagrams essentially what overlaps, what doesn't.

Yeah.

Like, and, and what are the specifics about what's happening in those areas, right? Mhm.

OK.

Sure.

Um, give me a minute.

I'm, it's something's bubbling up.

I'm trying to figure out, I'm trying to Put words to what's coming up.

OK.

Um, First, let me see if I understand this correctly.

The goal is that the, the The idea behind this is that uh both the developers and the design folks start with an initial guess of what needs to be done and the developers.

Um, so cycle one for the developers could be something like architecture or the infrastructure that's needed to pull off future experiments.

Is that fair? Yeah.

OK.

So, and when, when they're done with that, then now they can take inputs from designers and Implement things that can be ordered for the next round of um.

Experiments that we want to run.

And So, in this, what I wanna then understand is when do we start polishing things? From a discovery point of view, this is great.

Yeah.

At what point do we go, OK, we've, we have something which answers our answers the questions that we had uh framed for our experiments, but we need to get this thing ready for prime time.

Which is now back to a lot of um.

Development and maybe some input from uh uh designers, but uh so what happens then, or am I missing something here? No, it's not on here, uh, explicitly.

I, I mean, I will give you my, my personal opinion on that, like, when do we start polishing things and, and my, my answer is always and never.

Um, and the reason I say that is, uh, again, if I go back to my scrum roots, every, every time I build something.

It's got to be usable.

I actually have to have something I could put in front of a customer, um, and it needs to meet whatever quality standard we have for that.

Doesn't mean it needs to look pretty, but it has to work, right? It has to work at a, at an expectation that matches what the customer is.

So like, if it looks totally different from any of our other applications and it makes them confused and it doesn't associate with our brand.

I wouldn't release it.

That's actually not, it doesn't meet the right quality standard yet.

But then when it comes to like, well, let's learn about this thing, we build it, we ship it, we were, you know, we, we get, we sense their their response, you know, their reaction to it, and it's like, actually this is cool.

It's just like, I don't know, it's pretty clunky.

All right, so that works, and they have it, and I've now learned, I, you know, I sensed that polish would be better.

That's more work that we could do.

That is a value-based investment, not a quality-based one, but a value-based investment at that point, right? So let's make, let's make it a little less clunky, less button clicks, whatever, um, more performance.

I don't know what that needs to be, um, in order to provide more value to an end user so that they could use it as it is, but it would be better for them if we polished a little more.

So I always, I treat polish like.

Something that I I do with because I've validated its need.

OK, so I think you clarified something for me here.

So the, the cycle 1 cycle 2, cycle 3 here, those are um Small chunks of work that we're actually shipping, right? It's um it's like a end of a sprint or whatever those time boxes are, so the polish happens within implement designs.

It's not implement designs is not just just to gather more data but to actually finish that uh piece of work.

Yeah, yeah.

OK, that's fair.

Let me tell you, this diagram drives me nuts.

Um, And I, and part of why is uh is on the next slide, um, but also I see a lot of delays that happen here, right? So we do some stuff that that these people, they don't even know that what they're building on top of.

They're just designing for the customer.

And gathering some data for it says for cust cycle 3, like we're going way further out, like what stuff helps us implement here because we're gonna take this data and feed it in to design for that cycle and move ahead, right? So there's this lag and so we gather data in cycle 1, which feeds into designing cycle 2, which gets implemented in cycle 3, which we then deliver towards people to test in cycle 4.

That feels like a immense lag and like a mini waterfall and, and a lot of context switching to that is the thing that as a manager now if you think about this now the design team and the for any issues that come up.

For bugs that come up, now we have to resolve that out of band, we have to break the cycle for the designers and designers on the other hand, are doing a context switching constantly between two time scales.

So I, I can, I can see why.

Why it drives you nuts.

I don't know if we could pull this off without him.

Well, good.

I don't want you to pull this off.

OK, actually I'm gonna give you a better way.

So the, the, um, this, I worked at a team or a company that did this.

Um, we had a a design track that was going, there was a design team, they were working for things, right? They were working on a track at the same time development was going and our team, um, we had to fix.

The user experience of um a charting area in an application.

It was just really clunky and there were a lot of low hanging fruit items to go and start knocking off and um and when it came to it, our team was like, that we volunteer, we'll we'll work on that.

I mean this has clear customer impact.

Like, you know, in Valium, yeah, sure, we'll we'll do that.

Nobody wanted to touch it because the team that I had been working on had actually was told to stop working on this product, we need to get you into training, OK? So they, we pulled them offline and we're like, well we, we'll we'll, we'll deal with that, we'll get into this mess.

And when we went to go and do this, we're going to work on these things.

There is no, there was no design for it.

It was just feedback that was saying like, hey, we have to go and improve the way that users interact and experience these charts and to get the data that they need, um, and, and answer the questions that they're after.

And so we're kind of curious about what those questions are.

And so, uh, a member on our scrum team had the wonderful idea of saying, well, let's go talk to the designers.

And maybe we can get their help, right? Making something that's just got better user experience, cause this is really not our forte.

We have the technical chops to do it.

But that doesn't mean we're gonna create a good design.

Um, so we go off to talk to the designers about this.

And we come in and we say, hey, we're working on this charting area and we're going to try to make some, we're gonna be making some improvements in here.

We would love to get your help.

And the guy we talked to said, Oh yeah, actually, um, we have that planned out in in this board, let's say, uh, for cycle 3, like it's going to be about 6 weeks until we start diving into those designs and then we can get back to you on, you know, what we've discovered so you can go and build it.

And our, no, you don't understand.

The design, like the design is changing this sprint, the product that we are actually doing this work like today we're starting on it, and it will look different tomorrow and the day after, and by and in 3 weeks, we are going to show a different charting application to our stakeholders.

And like, well, we're not ready for that yet.

And we're like, we did that that's not why we're here.

It's going to happen.

We want to know, would you like to work together? Could you help us, right? Because we don't understand user experience and design nearly as well as you all do.

And they're like, oh, we, uh, here we got all this stuff going on, and one guy volunteered, right? Uh, he said, yeah, I can swing over.

um, can you, I'm a little busy today.

Can I pop over tomorrow? Yeah, sure, right? So he came over, we'd already started working on things, and he's just like, so what's going on here? We explained the problem.

Here's what we're hearing from customers, here's what's going on.

We want to improve the way in which that they that they that they use is by helping, making it easier to understand where they're interacting on the chart because they didn't know which chart they were using, make it easier for them to find the the the metrics that they were going to bring up, uh, those kinds of things.

Oh, OK.

What if we like highlighted this chart? Can we do that? Sure, we made it ugly, like bright red, you know, somebody's like.

Yeah, that's garish, like maybe we can do something else here.

Oh, I see we have a border, it's not very visible.

Can we like make the chart slightly bigger than the other one? Yeah, we can do that, right? So we just in the moment, working with the designer to craft a different solution, right, um, was pretty cool and then like launched it, it was a much smoother result, right? So we didn't think of a a design team and a developer team.

What we thought about was creating one team.

One team that oscillates between all these different things, between, we did a bit of discovery, and now we have to like actually discover and start delivering something, and then let's build it, and then we kind of pop back up.

Well, is this working? No, let's go back more.

Let's get it out there in front of customers, right? Um, it's two different types of work, but it's one cross-functional team moving together, right? Like, like testing out different things at different times and supporting each other in order to achieve the outcome, not about getting the output done, but to say, hey, we can all, you know, we all have different perspectives, different skills that are helpful here.

How do we as one team, work to go and make this objective happen? In earlier, the 1st 5 or 10 minutes when I was driving, which we were talking about what changes we've made in the team.

This is the direction that the big change in direction that we had for, for the diagnostic tool that we're actually gonna review today.

Uh, one of the, just to go back to what you had in the previous slide, one of the things that I've noticed is that it's tempting to assume.

That maximizing um employee utilization.

Is Strongly correlated with delivering value, they might not be the same.

Fast is not fast is not quick, so.

My martial arts teacher used to tell me that all the time, fast is not quick.

That is definitely not quick.

Um, God, I saw that in in a movie recently too.

I thought that was really good.

Uh, I can't remember the movie, but it probably was somebody else's martial arts teacher that brought it into the movie, um.

When, uh, Yeah, are you struggling with that at all with anyone? Appreciating that concept.

Aaron says yes.

I can feel on your face.

Uh I think I have that trouble.

It's not so much within my team, but outside the team, it's even more difficult to convey that because for most people.

The, the way I, I used to do it in the past was to play this game, right? You you one person holds their hand and then the other person taps, right? Most people when they wanna tap, they start moving faster, but in the process, they're telegraphing so much motion, they're looking at it, the the it, it doesn't always lead to better outcome, uh, but, but just slowing down and seeing minimizing action and then doing it slowly is, is more.

Is much easier than uh.

Then trying to do it faster, but I, I don't think I have.

That's, that's one thing that I want to convey to both inside and outside the team.

So I, I, I agree, like it's, it, some people are really stuck in that mode of thinking.

Um, I try to think of a, a couple of pieces there.

I Um, Try to always understand, do you think that people would like are just gonna work less? Is always a concern and it's like, you know what, they, they might, and that's actually OK.

But you know, cause, but I'm gonna keep paying them as if they're working 40 hours a week, like, no, you, what you're doing is you're, you've hired them to make your company achieve success, right? I didn't bring these people with like, here's the plan what they need to do all the time.

Maybe if they're working a support line or if you know, you're, you're doing literal like factory kind of work, that makes a little bit more sense.

But when we're trying to build things that haven't been built before, and, and I'm trying to say like you were hired because you've got a great mind, you fit our culture and your skill set can help us achieve more, right? So what that more is, I want to focus on that more, right? And so those are outcomes, those are objectives right here, right? So tying it back to that, so if I can get my people to to achieve these objectives.

More efficiently, right, or and and in whatever time it it means for them, right? If I'm learning that it's actually taking longer to do that, right, they're putting in all this time and effort, um, maybe that one actually I don't need to invest in this same anymore, but if, if I just let them run.

And they, and they're not working 100% of their time.

We still get the objectives done and maybe quicker than we thought.

That's a good place to be in, right? I don't care how you did that, right? Like you made the, the good thing happen? Cool.

Hey, um, well, if that's not actually, if that's only keeping you busy 20 hours a week, um, and I mean, you know, you're a full-time employee, so I kind of expect like 40 hours a week out of you.

What else could you do? Let me ask, I try to ask.

Employees that question.

What, what do you think that you could be doing that could help us achieve more outcomes? Not what can I give you to keep you busy, right? Um, and where is the right slack time to learn? To discover, to deal with emergencies, right? Uh, that kind of thing.

And so I say Slack time very, very specifically, um, Tom DeMarco's book Slack is a great, great read, uh, for help, like, and it includes a lot of data and math around why 100% efficiency, right, and using employees' time is actually one of the worst things you can do for most organizations.

Uh, yeah, I think.

I think I can see that.

I also, I've read somewhere that uh servers, if they hit a hit 80% utilization, that's like a All hands on deck kind of situation because it's it's red water, uh, red water mark levels, so.

It's the same.

Why should be different for human beings? And those are repeatable kinds of tasks, right? Yeah, so you, if you were, if you get to that case, right, um, once you get over that 80%, and I'm, I'm a computer engineer by trade, so I know exactly what's happening under the hood, um, the queuing time to organize where processes go starts to actually affect the workflow.

Right? So that's where some of the other 20% is, and it, and so things can't just flow because they have to like be reorganized to fit the right cores and find the memory and all that kind of stuff.

And then when you hit 100%, things are processing and other stuff's waiting and not getting done.

And then that comes in, which means that this thing has to wait and so you're just in this constant, the oscillation back and forth.

Makes everything take longer, right? If you want one thing done quicker, the best thing to do is make it the only thing that gets worked on.

And that might mean that you're only using 10% of your capacity, but the focus says, look, this got done and now I can go do something else, and I can go deliver on this thing and put it out there and wait for the market to go and respond, and I can go do some other discovery or go do some other work, right? So if you don't have slack, In some kind of way, you do not have the opportunity to to learn and you slow everything else down.

It's like asking, asking kids, you know, when they're, when there's, uh, when they're playing.

If, if we went to them and said, hey, only 80% of the time you play, what are you doing the other 20% of the time, they won't even understand what that question means.

If you're in a state of play, we're playing all the time, you, you'll find something to do.

Right.

And if you're, if you're, yeah, if you're in a state of play all the time, right, if there if um I was talking to somebody recently about this in the lounge yesterday, they said they sent their kids to a Wharton school um and there are a lot of just outside time and there's a lot of time where they're like they're just like sitting out.

Sign in nature board.

Right, and there's this kind of like, you like your kids should be bored from time to time.

And the main reason, right, is because in this moment of like if you just, here's a, you know, here, here's an iPad, go play games or something like that, that all they've got is like an activities, activity activity activity all the time.

What they don't do is they don't discover what do I actually want to be doing with my time.

They also don't go and say like, well, I'm sitting here, right? Like, oh, there's nothing to do.

I'm so bored.

Cool.

Yep.

Good luck with that.

And then sure enough, 5, 10 minutes later, there's like a pillow fort that starts to emerge and we created invented a whole new game like that creativity happens in the gaps and we let our minds roam.

So we need to do the same thing in our real work.

That's when we find our best solutions.

OK, I'm gonna move on here cause I know you're short on time and there's just a few last little things left, OK? Um, patterns, uh, uh, within dual track agility even with one team, right? Um, you can have separate backlogs, one for discovery, one for delivery.

Um, you can do one backlog where each thing is its own item, like this is just a discovery piece, this is just a delivery piece, you know, that kind of thing.

Um, and then there's blends right where the discovery work is actually in the stories, the product backlog items wherever they are that like in doing this, as we get it, move it up higher on our backlog, we have to do some discovery and when we go to do the work, like I was saying before, we're going to start building this thing and we're going to like interact and learn from our customers as we're doing it.

So you might have some things that are, you might have things that are like a lot more discovery or even full discovery or like, you know, mostly delivery, but there's a supposed to be a bit of a blend on all of these items because we don't know exactly.

What what gets the job done until we're done.

Do any of these look familiar to you and your work? Yeah, we're trying to get as much as possible on the, the one on the right.

Uh, one of the things that's actually helping us is that we have two teams, one here and one in India.

So based on, especially for this tool that we, we were working on, uh, based on the work that we had to be done for, let's say the next 2 or 3 weeks, the right kind of pod would be formed.

So if we had to do a UI revamp, then uh designers and the UI developers would get together, right? They would form a pod.

And then And then we do because we are, we have a time uh uh time zone difference.

We, we, we can, you know, the designers design something and then, you know, they, they sync with the developers and then by the time they come back, they have something that they can play with for the rest of the day and see uh what the users like and don't like and so on and go back.

So my personal preference is to do something on the right where people are working together and talking together all the time.

Um.

You also, though, it sounds like you're doing this a day at a time.

Mm, perhaps, yeah.

If your UI people and your designers and whatnot are working on something and then it goes off and they can build it essentially over the night and then the next day you can see something.

You know, it's a very short cycle.

But that's what it sounds like to me.

Right, um, But at least in this case, the work, like, we have to make the work visible, we have to create some, you know, some visibility to it.

And if you're planning on doing it that integrated, right, or even like that closely integrated in time, not necessarily in team, um, then yeah, on the right, you're you're getting closer to this right side where it's like, we take it on, we're doing some of that work, we're delivering at the same time, we're feeding the cycle constantly.

Um, OK.

So you're trying to get more to the right, you're maybe more leftish right now, but you're trying to get closer to the right or middle maybe right now? Is that correct or no? We We, we definitely don't do what's on the left.

Uh, we're trying to, uh, in the past we used to.

I'd say maybe between The center and right is where we, where we are.

There are some of these things which are just discovery only, uh, like design only, sorry, that, that we still do.

Um, Yeah, I'd say from 2 to 3 is what we're trying to do, trying to move.

Oh.

Um, that's good, cause yeah, we definitely don't want the one on the left, right? Cause now, this, this is the left is the easiest way to create multiple teams focused on different work.

It's the easiest way to say like, how do I know this discovery work actually influences this delivery in the right way and vice versa.

Um, who is managing these backlogs, right? Um, what do the teams pull from? Do they choose? or do we choose, right? At least having one backlog there helps make sure that we know why we're doing this work.

We know that this, this is coming up next because it helps us answer a key question.

It helps us deliver on, um, uh, a solution that can help towards to create.

Uh, an outcome that we're after, right, you know, all that kind of stuff.

Um, the one on the right, there's something I think is really important to also call out here too, that in my opinion, Um, again, like I scrums my kind of thing, like, you know, teams might pull in the these top three items in their current sprint, but we know as well in that case, they don't really work on just these 3 items, they're refining these other ones and discovery as refinement.

Can be a really good thing to do, right? So if I'm looking at something that is A couple sprints out worth of work a little bit further down the product backlog, and we just don't understand exactly what it is that we need to build yet.

That's OK.

And as part of my like, what should we build here, we made this, we may come up with hypotheses about these items that we think that this will actually help users in a certain way.

We may go out and talk to customers to learn about.

You know, what, what do they do? How much do, how far do they drive their bikes, right? Um, do they run out of charge, um, during, uh, uh, an outdoor trail session, that and, and that they're sad about that, right? Do they run out of gas, you know, one of those, those pieces that they're running into, um, that then we can come back to in another refinement session, uh, you know, a week later, a day or two later, and say, hey, we actually have some data which can feed.

Some of the information about this product backloggram, and even though we're not building it yet.

We are doing active discovery work on it, um, that informs where it goes, that helps us prepare for that solution, um, and ties us closer to the outcome that we're after.

So it, that's a key aspect to me is that discovery and doesn't have to happen at the moment you take that item on.

Sometimes it's in the refinement.

There are two things that you didn't mention here, Rich.

I, I, I wanna check with you.

Um, so one of the other things we have tried is to integrate the QA or the testing folks into these teams too.

So as the designers and developers are working on something, they also immediately tested and they give feedback immediately the same day and so on.

They're not the same as the people who are certifying the bill and saying this is ready to hit the, you know, go to the customer, but we're doing some initial testing, so that is working great for us.

One of the things that we're, we've missed doing in this pattern is to have some kind of release management integrated into this.

So as, as we come up with them um.

They come up with a small.

Um Batch of work We know it can be shipped But we've not released it because we either don't have the bandwidth to um set up the CICD we need to do like a continuous thing or in some cases it's because we actually need to test the whole bill which takes like in the case of firmware.

They might do some small scale testing for a week, but then by the end of the week we can't just deliver that firmware without testing it without doing some integration testing to our customers, right? So if we have to do that, how do we while they're testing.

We, we still end up with some convergence time, right? We, we're still looking at some baking time at the end of the week.

So in, in a 5 day week, maybe a day or a day and a half is dedicated to just that system integration.

How do we, do you have any advice on how we can go about doing that in an integrated fashion? Yeah, um, so there are a couple things that come to mind.

Uh, I love that you're trying to get the teams integrated a little bit more together, and it sounds like right now, You don't have any kind of CICD and release pipeline.

Is that correct? No, no, we do, we do have CICD for one of the projects that the, the one that we were working on.

It, it was one of the cornerstones for our uh.

Uh, swiftness, but, but I think we can, it can have, uh, It can be far more integrated in even this project and then there are other projects where, you know, there might not be as much penetration.

Yeah, and um and when you integrate that work more and you have teams that have more that blend of skills, two things tend to emerge, right? Um, uh, one is more people will be sitting on their hands.

Right, if they don't know how to do other types of work, right, which that will also happen eventually they might learn like you're testing people might understand more about how to release, uh, or they might understand more about discovery, right? So if I'm going to be testing these things, how might users interact with them and what what kind of discovery work can I do, um, and talk with customers that actually influence the way in which we test the product and also thus design what we're building to be tested in the first place.

Right, so like, people, I think, grow their skills over time, but even if they were only doing the very one thing, right, then at that point, you're optimizing, again, not for the individual and their busy, but for the team.

To actually create more usable results at a quicker pace, right, because there's not waiting and and there's not that those handoffs to another team.

Um, and as a team, then you're deciding when do we do different levels of validation.

Unit tests can be done on a developer's computer and on the build server, right? And they're very fast feedback that happen often.

But running your full integration test suite, if that takes a day, you said right now, or a day and a half, I don't want to do that every day, cause I, that's, that's too costly.

So now the questions come in to, is it the right way to do it? Right, and, and how might we increase the speed of our delivery, uh, you know, and like of our, of our integration testing, right? Um, and there's, there could be lots of answers in there.

I've seen times where maybe there's things we can directly automate, maybe it's that we need to actually push our testing left more, right, more into the unit level or or other ways that we can automate it.

Um, versus, um, you know, when it's, I mean, at some point you said you have to deploy this to a bike and a phone and try to get everything to connect, and that's cool, but like where does that happen? And do we have a bike that's just sitting there ready for the build server to deploy to, right? And the discovery of that, of how we do that, um, is a team capacity issue, right? they maybe need some time to work on that.

But the question then becomes, why do it? Do I do what? Why, why try to speed that up? Why try to make it, you know, what if you had an integrated team that could do this kind of work, so that, um, why would you Give them the space to not build new features that could go and create more value for a customer, but instead go and speed up their integration testing.

In my mind, I, I don't even want to prescribe that.

That's up to them to figure out what is the best use of their time, right? If, if they're waiting.

So I'm, I'm thinking about it because there are some, there are like 3 or 4 teams and each of them have different needs.

I, I personally can't prescribe one way of doing uh to them.

So I'm I'm.

I, I don't have a good answer for you uh for, for the question that you asked me.

I, I think You, you, you answered half of it already, um, right of you shouldn't choose how.

Right? So all the features, solutions, if we did, if we did this kind of thing, it would make us, you know, integrate quicker, like, and that's cool, right? Um, and I want the teams to do this because this is important.

So but the why still matters.

What's the outcome? It, it's, it's the same thing that we were talking about earlier as far as I can see, fast was quick.

If they can deliver faster and they're not waiting, which is frustrating for any kind of uh creative person to just wait.

Uh, and not deliver value.

The, the, we break the flow of things by so uh that is one thing I can think of.

And and so that's an interesting thing, right? Like if, if we had something like, you know, how might we uh let's go back to business problems, right? And but look at it internally.

We have internal customers that we're trying to go and help.

They demand lots of new features and capabilities on our stuff all the time, um, and we're actually kind of, we've noticed that we have a problem in which our designers get out of flow.

Um, when we have to go and go through these testing cycles and we're waiting to find the results, how might we improve the speed of delivery, um, in, in our build processes so that we can test and learn quicker.

To deliver the most value to our clients, right? So again, no, no solution prescribed, but you know, the same concepts we've been talking about throughout this whole class, right, of focusing on the problem, trying to understand the outcomes that would would say, are we actually doing that and then letting our teams discover the right solution to get there can be applied here too.

So I, that was, that was what I was just like thinking I'm like, yeah, we can do the same things here.

So I've seen it done in lots of different ways.

I didn't want to prescribe a solution other than give you ideas of this could be done, um, but yeah, when you, I, I think, you know, another key value there is um.

When teams can integrate their work a lot quicker, when they can get something done faster, right? When integration takes minutes instead of days.

That increases our ability to learn.

Alright about our customers and learn if our solution creates the value that we want, it increases our overall uh quality of our product, cause I'm not like trying to do something while this test is happening and then like, uh oh, that didn't work.

What did I do over here? How does, how do these all affect each other? Um, it increases our organization's agility, right? So we can say, yeah, I mean anything that we work on, once we're like, you know, we wanna make sure that this is something that can ship.

I can have that done in 30 minutes versus like, yeah, we'll do that and then we have to take a whole day to test.

It gives us more options to test our market, to create better outcomes for our customers, and to create better impacts and results for our business.

So those kinds of things are are really key and that that's the kind of stuff I lean into when I'm saying we're gonna change the way of how we work.

We don't maybe know what that is yet.

And to resist perhaps upper management, that's like, but your teams, we got so much stuff we gotta work on and do, why would we have any time for that? By making this time, we expect it creates this kind of outcome, right? And I can I can try to instead focus them on the outcome about something that they may care about.

And then that's the story that gives me the opportunity to work on it.

Right.

I need to hop off now.

Yeah.

So, but, but Aaron can stay, right Alan? Yeah.

So we, yes, we are literally right here.

I mean, this is just wrap ups, right? So I can, I, I can go through any of the wrap ups and questions that you want, Erin, but we, yes, we are, I mean we we just talked about a lot of those right there, right? Outcomes over outputs, OKRs drive everything.

It's all hypothesis, tell those good stories, right? That's super, super key.

And then, yeah, we get to.

Cheers, have a good time.

Um.

So, let me pause and say, Che, thank you for bringing Aaron and Matthew along, and this is really cool and fun for me, and I'm glad you got a lot out of it.

Um, let's you and I catch up sometime in the near future.

OK? I will, I'll I'll, I'll, I'll, uh, send you an email and then maybe we can do a quick chat and figure out what, what our next steps can be.

I, I can't thank you enough.

This has been great and I'm truly thankful that Aaron joined every single day.

Even when it was 6 a.

m.

in the morning, which she absolutely, absolutely hates.

Thank you so much.

Oh yeah, I appreciate the opportunity cause, you know, again like when we first introduced yourself, right, this is, you know, Shay is saying, hey, you know, I think this is a good, you know, this fits the lines of your interests.

You should attend this, see if it interests you, continue down this path and learn more, you know, there's a lot of new concepts here that, especially since I don't necessarily have the existing management context and background it's kind of like, oh these are whole new foundations I can build off of, which is a lot of fun so far.

It was awesome to watch your engagement with it too, right? And like redoing things and coming back in.

I think he made a good call bringing you guys in.

Right, so, uh, thank you, Aaron for like the, the passion that you seem to show in the class too.

I thought that was really, really awesome.

No, just it's a good teacher, right.

I appreciate that.

Thank you so much.

Um, OK, so, yeah, we are, we are like I said, we're done with the class.

I've got some wrap up things that, you know, that I need to prep to send out to you.

She, I know you gotta run.

Erin, if you want to hang out and talk, you know, anything else, other questions, I'm willing to, to, to chat.

Otherwise, we can all say our goodbyes now.

That's, that's totally cool with me too.

But she, at least minimum.

For you.

Goodbye and take care.

Bye for now.

I'm, I'm, I'm definitely gonna get back in touch with you.

I, I, I loved it.

I love the course.

I loved your pacing.

I love the, the, the, the presentation style and, and the, the thought you put into every single slide and, and flow of, uh, you know, the sessions.

I, I, I really enjoyed it.

I really, really enjoyed it.

I, I actually wish we could do it like a 2 day, hands-on, do nothing else, 2 full days, just do this.

Oh, I wish I could do that maybe 2026, right? We fly me out to, I don't know, one of your locations, one of your dealers, we could do it there or something like that, go with that.

I'm keeping my fingers crossed.

Thank you.

Thank you.

Bye-bye.

Aaron.

What about you, man? Anything else you wanna dive into? Any questions? I know, I know we were going quick there in the end, I let him drive a little bit.

So, I mean, you know, my time is yours here.

Uh, so what can I help you with? Yeah, it's kind of, you know, one of the things when Treas first started, um, almost became his direct reports that, you know, he handed me a list of books and said, here's some reading materials, I think, you know, we'll try to gauge your interest and that's that was the fun part, at least for me it's also in the think of it because it's like, right, it lists some recommended reading materials and all the supplementals like, oh, you know, watch this TED Talk cause, you know, have like this how to tell a story TED Talk, right? um yeah and so it's like, you know what.

What else can I, you know, get my hands on to try to read about and figure out that ties into all this cause I know, right, you know, there's the entire sense and respond, you know, book series that you pointed out in the first session that Tres has on his bookshelf, yeah, outside of that it's like, what else? So, oh, OK, you, you're looking for some specific recommendations on other things to go and look into, read and get and go further.

OK, um, well, I mean, like I said, I gave one, right? I, I think it's really good book there, um.

Yeah, another good TED Talk, um, Astro Teller, uh, he is, uh, he was, um, Google's X Labs, uh, and he led that whole initiative for a while, um, and he talks about, um, Uh, that's like the surprising like the truth of failure.

I'm trying to think, um, the unexpected celebrating failure.

Yes, the unexpected benefit of celebrating failure.

Uh, really cool TED Talk, and he gets into like, again, some of these discovery things and, and for them it was, um, as much about learning about users as it was about learning like tech technically is this feasible and what, what does that mean for us, right? Cause he didn't, he's like, we don't have infinite budgets, we can't just do anything, right? It doesn't matter how much the sales guy wants it, it's um we're trying to solve a problem and and and we say, what's the Best way that we can kill this effort right now, right, uh, to say that like if we couldn't get this done, we, um, we might as well just go work on something else.

And so they actually, he talks about succeeding with a lot of those for a while and some of those projects that have since also gone away, and some other really cool things that they that they were like, this is still viable, but it's not viable for us.

We just don't have the funds, you know, to continue with it.

Um, so yeah, check that out.

Um, books.

I have a lot of books, uh, check, just pull over one of the key ones that I wanna dive to, um, hm.

Yeah, was good.

Mhm When When you work with leadership groups, um, so like I read this as in a um a leadership team at an organization.

I, I eventually became part of the the core management, um, the advantage, I thought it was a really cool book.

Lai's good, uh, but he gets into a lot here into, um, essentially establishing clarity.

Right, so leading with leading with goals, leading with the objectives, leading with values.

Uh, this was a really cool book to help frame that so that when you're going and and trying to work with your directs, right, it's, um, and actually I don't get it.

Why am I doing this? And if you've ever felt that, it's maybe because they're missing some of the pieces that are in that book, um.

Antifragile is a really really neat read, um.

So this book um was The whole key thing here and it's, it's, it's dense, it's like it's a, it's a very, very dense book, um, but the whole concept is in studying and understanding systems that improve under stress, rather than ones that fracture, right? So you, um, if you think of fragility, right, you take a, a glass, you drop it, right, it breaks and shatters, um, you know, it's fragile.

Uh, this cup.

is not anti-fragile, it's resilient, right? It's robust.

So like I can drop it and it's not gonna break, but I might dent it, and that dent's not gonna go away, and eventually that might affect its performance, and I can crush it and it'll go and it'll be, you know, ruined.

But so anti-fragile systems are things that improve where like um Your immune system.

You get sick, you're ill for a while, but now your body knows how to resist that in the future.

Um, a restaurant is fragile, but all of the restaurants in an area are anti-fragile.

If one of them undergoes like a health code violation, they all benefit in some other way from it and learn how to improve their quality because people are gonna be looking for that, right? Um, taxi drivers, right? Uh, antifragile, right? It's like, oh, I'm not getting a lot of businesses in this area.

Where do I go? How, what else can I do? I don't have to sit here and wait, right? So I think that those concepts apply really well in How you work Right, and try to figure out like how do I develop antifragile systems around me to learn and improve myself, um, and, and towards what I work on, um, as well as in business, right? What can we do to Apply stress to our business in order to help us learn and improve.

Even a lot of the concepts we've been talking about here in this class are mechanisms that can introduce that stress and help you actually improve and grow from it.

Sure, I'll definitely check these out.

Yeah, um, yeah, I mean like data books, facilitation books, like I, you know, I've got other things to get to dive into.

I'm just trying to think of what's the best.

What's the best thing for you? Where, where, what are you interested in? What, what do you feel like is a gap that you're discovering? I mean, part of with, you know, at least of the entire training is kind of showing that, you know, at least right, it's the hard part is always figuring out how they can immediately apply it to the projects, right, at least, you know, thankfully since, you know, the class was just people from zero we're able, you know, directly tie into, you know, the, you know, on the mural board what we've done.

And then for now for me it's also thinking about, OK, you know, we've done this one vertical slice, you know, how is this, how can I start doing the same to you know the other projects like Dres I was talking about the diagnostic tool we've been working on.

Um, yeah, and of course, as you pointed out, you know, at what point do you finally have the time to stop firefighting and do something else? Yeah, yeah, and, um, and making that space for your team, right, to learn and, and grow, and um I have a feeling he, he will try to give it to you at some point, right? Um, and, and it's something that I, I envision in the future, you will have teams that you're gonna be, that are gonna be reporting to you and like thinking of how do you, how do you give them an objective, right? Give them a, a, a purpose, something, give them a hill to climb.

Um, give them some constraints about how to get there, and then like let them run towards it and figure out how do you support them to do it.

Those are the best teams, right? Um, and they're the ones that are gonna keep coming back and finding something new and, and, and, and want to work with you, versus the ones that are like, yeah, I'd rather be doing this other thing, I think I know better, you know, whatever.

If you do, prove it.

Go ahead, give it a try, um.

So that that would be kind of interesting towards uh my two of my favorite books on teams and and that uh you go to my library here.

Yeah Um, work rules, Lasso Bach, he was, um, senior VP of people operations at Google.

Um, This is a really cool management book and like all like at least a third of it, if not half of it is about hiring people, um, and like how how so many people do it wrong.

No, definitely, yeah, it's pretty easy to see how hiring.

The wrong people can definitely tank a project or just team morale in general.

Mhm.

Yeah, but they, he also talks like about some cool things about, you know, like letting your teams run and learn in different ways and trying to um like how you How you find out where you like where you invest for performance, right, um, that like you're gonna have some people that are just, they, they do substantially better than anybody else.

Great.

Pay them substantially more than anybody else and like let it run, right? And, and then like don't improve the things that are like, oh this is like not working that well, we have to get it better.

Maybe don't, maybe actually it's not a key factor and need to find something that is working really well and make it even better, right, so that you can like get to that level of what's working versus is that worth investing to bring this up to speed.

Yeah, yeah, so like that that's a cool one.

And then when it comes down to like what do you double down on and how do you grow that over time, um, it's, there's a lot of ways in which you can apply like the stuff that we've learned or, you know, uh, actual patterns or anything to every aspect of your business, every team, and not every team really needs that, or not all of it.

Um, and so, uh, scaling up excellence was a really neat read about that.

Um, So the cool thing here was like like really finding excellence and and understanding how it gets copied, right, is, is what I I pulled out of this book and like the very first chapter gets into that, I really think.

So this is um uh Rob Sutton and Huggy Reyo.

I just uh feel like at scaling up excellence.

The, um, we talk about setting the stage, right, and the, the, the, the first thing that they that they cover is.

Uh, Buddhism versus Catholicism.

Right, and it's like what, imagine applying like a religious view to scaling, right? And the um Uh, the, you know, the more the Buddhist way, like being in harmony with the environment and trying to like, you know, figure out how to grow from there, um, and letting people work with it versus like the Catholic, like, this is the dogma, we will do, you know, this is how you run a mass, all that kind of stuff like that, that creates excellence in a way, and it also stifles it in ways like they both some, you need a little bit of each of them in your workflow.

So, as you're trying to find out like what works.

You know, the thing that I, I got out of this is like recognizing, it works now.

For this thing, it may work again in the future, but it may not, and being open to identifying some of those core elements that lift up your teams and and help you achieve like more excellence at scale, um, it was, it was a cool read for that.

Nice.

Yeah, yeah.

So hopefully I, I at least, you know, made your bookshelf thicker by that much.

Definitely.

Yeah, at some point I can imagine at some point, yeah, I'll get to the point when Shrey's bookshelf.

Yeah, oh God, I know.

I'm, I'm looking at mine, it's not even like that, that thick, it's like it says, um, but yeah, the, uh, oh, the goal, the goal was a good read.

The Phoenix Project is maybe a better read um for you actually.

Yeah, that was one of the first things she has handed me was the Phoenix Project.

OK, good, good, good.

So yeah, you understand that then theory constraints, I mean, I work the automotive parts manufacturer versus an actual like.

Automotive manufacturer.

That's yeah, no, that was that that was definitely part of the reason why tras was just like read this and it's like tell me how you feel after it.

Mhm Yeah, that that was basically his first intro like, all right, here's a taste of what I'm gonna have you start working on, start learning about, and I'm like, cool.

It's definitely a fun thing to read on the flight.

That's that's cool, yeah, oh God, I, I remember the first time I read it.

I got a couple chapters in and I had nightmares.

Like I, I just, I was like it triggered previous things like, you know, that I've done.

I'm like, oh God, like I, I like PTSD got results.

Yeah, it's just like, you know, the, you know, right, starting off the premise is like, oh, I'm this guy and all of a sudden I discovered I'm in charge now.

It's like, uh, No, I know.

What do I do? Yeah, yeah.

No.

OK.

Um, what else can I help you with? curious about.

I think it's just for right with the entire, this entire training it's.

What are some easy ways, you know, the small slices and begin, you know, chipping away at it, like, right, you know, definitely we've seen like how, you know, we spent a good amount of time trying to refine the business problem statement and figure out what are the OKRs that we can create from that.

And it's like, what are some What is a way, you know, start taking maybe a smaller problem, you know, sca reducing the scale of the problem maybe or it's like.

at least we approached this as a business, a business problem statement is that do I just abstract that to like, OK, is my business now this specific application I'm working on and You know, these are the things around that, or is that still, you know, Yeah, trays would describe it, you know, the tactical level versus strategic level, you know, that level, that kind of thinking, yeah.

Uh, good question.

I think, um, not, not everything needs a business strategy, right, associated with it.

You don't need to understand the way we're gonna play we're gonna win all that kind of stuff and, and, um.

Even the business problem um is overkill for some situations, but I think a key piece, you know, to mention is there's always a problem.

There's always a problem that's trying to be solved, and discovering what that problem is.

And and figuring out how we might know that it's been solved, those are good techniques to use, even if you're not trying to craft a business problem statement.

So, when you mentioned before about like creating this application.

Um, that's an output, right? It is a, it's an assumption, it's a guess, it's a way to do it.

So when you're, every time I'm prescribed a solution to build, um, and I'm, and I, it's happened a lot.

I was a consultant, so people hire me and if I don't do what they want, I might get fired.

Um, but every time that I saw it, I'm like, OK, to build the best solution here and to really make sure that I am as effective as I can be on these things, can you take, you know, explain to me what are the the context, right? Yeah.

Give me the context, let me know.

Like yeah, what led you down this path to, you know, tell me to do this, right? Yes, yeah, so I'm asking, getting curious about that, right, and, and not in a like, um, I wanna know if you're right, right, you know, but like I, I just, I just need to know, like the more I know the better I can do, right? So just help me, teach me.

Right, if the context is like, oh, you know, you, this is the specific problem that the customer is facing, and you want to make sure that the solution absolutely focuses on that, right? That's important information for me than that, but yeah, OK, when I, you know, build a solution that you're dictating that I do put an emphasis on, right, whatever.

The pain point that you've had already identified and you've communicated to me.

You know that I'm addressing it.

Yeah.

Then the next part is finding out how will they know, like how do you, how do you know that this pain point has been eliminated, um, because usually like out of those three things, the solutions easy one to come up with.

Finding the problem is probably the the next easiest thing to come up with.

It's the, the more obvious somebody's yelling you about, you know, yelling at you about it.

And then the thing I see missing the most is how do we know that we actually did the right thing.

Um, but when you start asking these questions, if one of those things is missing, whoever you're talking to might feel uncomfortable with it, right? Um, and so help them discover, right? So it's like, oh, well, here's the problem, like, so that's, that's why we're doing this thing.

Cool.

And just to make sure, like, so that we maybe we can even develop our solution to give us the the the results and like how will we know this is actually like working.

And we made the right investment or that we should keep investing in this or whatever.

If they don't know, like, it's like, well, well, cool, let's like, let's come up with a couple of ideas and lean on these techniques of like how what what behaviors will we see change, right? Uh, are there any leading behaviors that might lead to that, right? And start coming with the, you know, with, uh, well, if this changed by 30%, if we start to seeing, you know, people actually lying on the mattress, you know, whatever, um, then we can do that and like.

Actually, uh, someone I'm working on tomorrow is, um, next week, is it next, oh my God, it's next week already.

Next week, I'm giving a talk with um Jeff and Josh on the combination of OKRs and nudges.

Uh, do you know the concept of a nudge? Not familiar with that.

OK, so, um, one other book and uh I don't have it on my bookshelf right now, um, Thinking fast and slow, it's a dense read, but it's a good read.

Um, it's got my copy.

That's that's something I gotta figure out where the hell is my copy.

Um, so thinking fast and slow, uh, cover talks about the concept of, um, humans, humans process information in two different speeds.

Some things you think about really fast, and they react to it without sometimes without even recognizing that they reacted to it.

That type of information processing is, um, they call it like level one thinking, and it is uh cheap for the body to go through.

Right? So that you're, you expend very little energy making that decision.

Um, there are, when you have to think about it though, and actually like, well, do I want this and what do I do? And, you know, is this the right choice? Um, that, that's a slower, deeper level of thinking.

Something we're obviously capable of doing, but research shows actually causes physical pain in people, like, like it's, it's uh You, you, the agony of that choice.

Right, that word is is chosen on purpose cause it feels painful at times to go through it, and it takes a lot of energy, right? So, um, if you want behavior to change and somebody has to choose to do something out of the ordinary for them, and let's think about it, then that's a painful choice for them to make, and they're most likely to not do it.

Right? So you want them to get to buy bikes and you start giving them all this stuff to think about and choose.

No, right? They're they're not going to.

So instead, the idea is you create a nudge and a nudge is also a choice and um uh work rules talks about nudges a lot too.

There's some really cool stuff in here that and the concept is, um, you want to, you wanna do a nudge is an experiment or an a and an opportunity.

To provide choice to somebody, but in such a way that the choice that you want them to make is now a little bit easier than the choice that they would have made before, and the choice that they would have made before is just a little bit harder to make, OK? So, uh, the, a, a simple example of that is, um, we want the outcome of people um living healthier lives through healthier eating.

OK, um, and we know that in our cafeteria right before they go to check out, there's the rack of candy bars that I could pick up and buy, right? And so it, it's a very easy and obvious choice of man, that, that, you know, pack of skills.

It's delicious, like it's pretty good, yeah, right there I can just grab it, right? And, and off I go, um.

And so, you know, people won't eat as healthy because the the simple choice put right in front of people is a candy bar that's right within reach, right at the moment of checkout.

I don't even, I barely think about it.

I just grab it and go, right? It's um, and that's why they go away, then it's a more of a mental, it's like, oh, do I wanna actually walk over there, grab it, come back, stand.

Yeah, so maybe we actually keep the same racks of things, but we move the candy bars, like you said, somewhere else.

We, we put them just out of reach.

We move them to a different spot, so you have to like, oh, hold on, I need to go back, I need to go over there and do something to get my candy bar.

But what's right at checkout? A whole bunch of fruit.

Right, I can just pick up an apple and be done, right? So the key here is that there's still choice.

That's being made.

I can still choose the thing that I want, but because it's a little harder than it was before, and the thing that I'm, that somebody wants me to do instead, it's just a little easier than it was before.

It's a little bit more like, I, I don't even have to think about it.

I can just grab this apple and go.

I start to nudge behaviors in the right direction, but I can't, like, remove the choice.

I can't say like, well, we're just not gonna stock candy bars anymore.

Right, people get angry, they won't do it, they'll, you know, they'll they'll go out and get them anyway, they'll stock them in their drawers, right? We will not change the behavior.

Um, and we can do that in a lot of things like with our products too, like we can make it easier, and I think, um.

We're talking about that with the bikes, like the demo application, then trying some things out and like just be like, hey, um, why don't you, why don't you try these settings on our local bike, right? And just like hit a button and we'll schedule a demo for you, right? And um that's like, oh, I, I don't.

Do I want it? Oh, I mean, it's right here.

I just but I it's like I'm already here.

It's already set up, might as well, right? Yeah, and like versus like coming in and like having somebody say like if they come into a store, how do you ride and what's going on? What do you do? All right, well, let's like, you know, configure this thing, you know, a certain kind of way versus like putting.

You could ask them that, but you can also put it right in front of them and be like, here, play with this, and configure it be like, hey, you just configured the bike, you want to go ride it? Like it's it's right here, it's ready to go.

And like that choice, and they could say no, they be like, well that's it I you mean how I set it up? Yeah.

Sure.

Right, and now we lead them towards the things that we want.

Um, so those are, those are cool concepts.

And so thinking fast and slow gets a lot more into the brain science about why, if you're curious about that, and it can really help you think of like leading towards those those good outcomes and and getting people to If you, you know, what I try to do is I try to nudge people to think about problems and outcomes.

Over solutions, like, yes, it's easy to to come up with a solution, right? If I make it harder, like, I'm not gonna do this until we understand why and like we can do it, but like we, you know, it's still we, we gotta, I gotta like this up, I gotta do that.

If you just tell me what it is that you're after, we just get going on it right away.

And that might shift them for a smaller thing.

And last thing I ask, is that right? And then the other part that, especially early on, we focused really on.

Avoiding the use of placeholders or You know, making sure that we're producing numbers, you know, it's backed by data or some analysis of Some proof of indicating we currently do that, we are we have X Y Z, you know, we have 30% fewer sales than the plan of record, right? I think, you know, you highlighting the strength of directly indicating that no, you know, we don't need to say we have why models being sold every year.

It's like if we already have the data and we know that's a problem or you know, we know that this is what we're looking at, just toss them in there like what's the point of hiding it.

Yeah.

What do you wanna know about that? Or where, where are you going with that? And I think I realized I started that as a question and I started expanding this and I'm trying to get back to the original.

Um, At what point does, you know, if you don't immediately have that information in front of you, at what point does it, you know, the, the value of actually going through and digging and collecting all that data to produce these numbers at what point does that become prohibitive to, you know, that's kind of like, you know, OK, the output is I'm producing in, you know, a statement with hard numbers versus I just need to say this is the problem I want to solve.

I don't quite have all the data I need to justify this yet, but, you know, at what point is it like I need to do one or the other, no, or I need to give up on.

The data collection and analysis.

Um, OK, I, I see where you're going there.

So my thoughts are, um, the data is there to make the story more compelling.

And to give us some guardrails, right? So, um, the, the, the assumptions about we're gonna improve it by 30%, right? That's, that's a guess, right? And it's just, it's something to show that we're gonna make a movement like you said with the example of we improved by 58%, like that's great when it was at 14, like, no, no question this was worth the time.

Um, so having data.

Helps make that story more compelling.

Um, and, and having a compelling story is important if the cost of investment.

is going to be higher, right? So, um, ah, yeah, I mean, I, obviously it's the problem.

I mean, like, haven't, have you heard anybody talk about it? Yeah, OK, at least one person? Cool.

I want to go and spend 4 months working on this.

Oh, it's a lot of money.

Like, now I'm curious, should we really do it? When I can make a direct connection for somebody like, yeah, we've seen this problem, here's evidence of it, and here's also what's happening because of it for a business like Data about the customer, data about our business, um, people care about that, like, they start to, it's like, it's hard to refute data.

Um, I'm not saying people don't do it every freaking day, but like, it's like, here it is, oh, you don't have data, go get me your data.

Right, like, show, you know, you don't think that this is true? Cool, go get, go, go, go get me data.

You're telling me this is a problem I need to solve? Well, show me the impact on this, is this actually, you know, we're on the value chart and say, you know, this high effort low value, yeah.

Yeah, um, something we didn't cover here, um.

In the class, they used to have a slide and it's uh Missing, um.

Truth curve.

Have you ever seen this before? No, I'm not.

OK, so, um, I, I'm gonna find out why this got taken out of the course, cause, um, I know it's in a different course, but so the idea is really about evidence, and you're saying like how do we, what do we need for this evidence and this data, and if we're just, all we're doing is talking about is this a problem? Like, like I think there's a problem here.

OK, well, like, you got any proof of that? I saw somebody that has this problem.

I don't need a lot of evidence yet, right? But also my investment is just a conversation.

Like if we're like, no, I don't, I don't agree, don't go any further, right? But if we're actually getting to the point of like, we're gonna get something out there, we're gonna release, right? Um, we need more evidence that says like, yes, this is actually worth and, you know, doing the, you know, the work, that kind of thing.

Um, and so that the whole concept here is like, if I don't have enough evidence for the things I'm working on that I'm going to be putting out in the market, that is very risky.

It's, it's unnecessarily stupid, right? I can also collect way more evidence.

That is necessary for the thing that I'm doing, like, if I'm just trying to have that very first conversation, Um, mountains of evidence are great.

If it was cheap to get it because I'm already collecting that data, I've already instrumented my products to have it available to me, and it took me one quick question to an AI tool that already has access to all that data to spit out an answer or like I'm looking at uh Tableau or whatever right and I didn't have to invest much.

Cool.

Now I've, I've already got the data, like it's, it's actually I've got loads of evidence and very little effort to have that conversation move ahead, and I can make me kind of jump to some of these higher states, right? So, um, so you're, you're really trying to get, you know, like in this case, we're trying to make two things happen.

One, Um, that we give evidence to create a compelling story, to decide to actually pursue this thing a little bit further.

Um, and then afterwards we also want to use it to build, change the way in which we build systems like our products and the processes that we put around products to make the data that would help drive these decisions available to us cheaper so far, yeah.

Easier and cheaper to acquire.

When you don't have it, it's gonna take time, and it's gonna and the effort is bigger, but the more that you put that in, the easier it is to put that data in there in the first place of like, we actually, it's not just anecdotal, like people, you know, we have clear data that's showing there's a problem here.

Um, From another perspective, like, uh, I, I think I maybe mentioned Etsy's monitoring and data-driven solutions in the class or or no? I don't think so.

OK.

Do you know Etsy, the website? Etsy? OK, so Etsy is their masters of continuous deployment, um, hundreds of times per day, but I, I like one time, um, they, they do, they monitor some really key things, uh, the modern stuff like, um, the frequency of logins.

They monitor um volume of of posts in their forms and support, you know, channels.

Uh, they measure volume of sales, um, and there was one other key metric that they're getting into.

I can't remember what it was, uh, but like these are things that are just on like up in a, you know, uh, an operations room and Etsy all the time, and they're just like constant graphs, they're just, you know, flux.

And they also have, because they have so much data, they have data about like where, what's the range? That they should kind of be in cause it's never like, you know, exact, right? But they know like, what's a typical range on a 2 p.

m.

on a Friday, you know, kind of thing, right? And so they monitor all this stuff, and because they, they developers have the agency to release anything at any time, like there's constantly stuff that's going out there and changing.

They wanna know like what's what's moving and how do they correlate that to the changes that are coming back on.

Uh, so they had a problem that they didn't write a business problem statement, they just, they came through it much quicker than this, um, where they noticed what, you know, all of a sudden once uh a the the volume of sales.

Started dipping down, right, like just it's unnaturally.

The volume of logins.

Dipped down, like people were just not coming in and buying anything anymore.

And then, but also the number of support and for posts rose.

Right, and it's like, wait a minute, this is people are engaged.

But they're not buying, and like, what, what's going on here? What happened? So they started looking at data to find out what's the problem that we're trying to solve.

We have evidence and the evidence in this case was that um people are saying that they cannot log in to complete their purchases over and over and over and over again, right? What do you mean they can't log in? Can we, do we have logins? They run all their tests.

Yes, login works perfectly fine, right? Somebody get in there and actually log in and go buy something and let's see what the heck is going on.

Um, and they found out the problem.

Somebody made a change to a CSS style sheet.

Where they were modifying the color of uh something that was blue in one spot and they wanted to make it white.

It's totally different, right? But what that meant was, is that the blue login button.

That had white text on it and that was on a white background, also used this style and now the login button just disappear.

So people didn't know, like they didn't think I click enter to log in.

They, I have to click the button and the button's gone, or it wasn't.

You just couldn't see it, so nobody knew how to log in anymore.

And thus I was trapped, right? So, OK, we can find that, revert that change of roll forward and I wanna go.

Um, but they, you know, they, they designed for this information that helped them identify potential problems from the very beginning, and it, it's part of everything that they do so that they still designed to get this information and they can still release things hundreds of times per day.

And get it there, right? So I think like.

It can be a journey, but that's, that's also why you want those numbers.

It's like, if you don't have them.

You know, and your stories are compelling, you need to figure out how to get them, and you wanna make sure that that is as easy for you as possible, so that we can actually focus on the problem, find the problem, focus on the problem, solve the problem.

That's sound good? Or any, any more questions around that? This one It's gonna, you know, try to start seeing where I can put these in the practice, right? Yeah, um.

You know, a, a, uh, a political variant of the story, right? Not having the data right there, right, um, obviously compelling action, but also compelling lots of different kinds of action.

Um, so I think about all of the troops getting deployed to support violent crime in Chicago right now.

Right, um, OK, obviously, you know, the president can just choose to paralyze troops and send them in and not, you know, there's a change otherwise, right? But there's no like, you know, we see violent crime as evidenced by blah blah blah blah blah.

And in fact, the people who had the data created different problem statement with we have all of this evidence that says this is not a problem.

And the more compelling story is that, right? And so that's creating this distance, right? And if, if everyone could see the same data, it's like, yeah, actually there is a problem that we would need to do something with.

Um, so let's go do it, or, you know, we all see that there is no problem, so we're gonna put our energy somehow, so.

Data helps tell the story.

It just makes it that much more believable when you go and get there.

Right.

Um, we are also, so another one, I just got a preview of a storytelling class that we're building with Sensorypa.

Um, so that was fun.

We did like, um, ethos, pathos, and logos arguments, right, like a appealing to authority, uh, appealing to logic or appealing to emotion, um, not in that order, right, but, uh, those things, and then other, other formats and stuff, so.

It's another cool way of like taking these kinds of information and formatting it in different techniques and using those things to, I said, engage people and say, yep, let's go work on this problem, and let's focus more on that, rather than your luxury list of ideas on how to solve it.

Or it's like, yeah, you know, let's go.

If you got a laundry list of ideas, let's try and see which one of those we actually need to focus on, right? Exactly, exactly.

All right.

Well, um, Erin, I just realized I have a call in 20 minutes and I have got to figure out lunch.

Um, so I'm, I'm gonna get rolling, but dude, I, what I said before, I really meant it.

I really appreciated like seeing what you're coming up with and your engagement and like re-engagement with all the material.

It, it was cool to watch you think about it right on there and, and to your questions got more intricate as we went through the class.

So like that was, that was cool for me to see.

I'm at your disposal, man, like if you ever want to talk about any more of this stuff, and if I can help you out in any way as you continue to, to learn, grow and advance in your career, please let me know, OK? Thank you, it means a lot.

Yeah, you know, just in Patras, you know, I really appreciate, you know, the way you've, you know, set all this up, you know, tying, you know, slide deck content, all the different experiments and giving us, you know, the space to be like, oh, OK, let's go ahead and actually try to do what we just talked about, you know, I really appreciate that.

Yeah, you being so willing to provide feedback.

Cool.

Well, I will try to keep that up and I really appreciate that that feedback there.

That just, that means a lot to me, you know, I makes me feel good, makes me want to keep doing more of it.

Yeah, thank you.

I appreciate it.

Thanks, Erin.

All right, take care out there and um yeah, I will, like I said, you'll hear from me again soon, but you can always reach out any time, OK? Have a good day.

Enjoy your lunch.

All right.

You too.

Take care.

Take care