Hut 8 Labs

No Deadlines For You! Software Dev Without Estimates, Specs or Other Lies

In Coding, Fast and Slow, I talked about one of the deepest challenges involved in writing software: the near-total inability of developers to predict how long a project will take.

Fortunately, as that post mentioned, I believe there is a way to work, where the software you write ends up being valuable, and the business people you work with end up being happy. And, critically, this way of working does not involve committing to estimates of how long work will take (which is good, because, personally, I suck beyond all belief at such estimates… even for work which I initially believe will take no longer than a single day).

In a lot of ways, this is The Most Important Thing I’ve learned in my (let’s just say many) years of being paid to write software for people.

The core idea is: put uncertainty and risk at the center of a conversation between the developers and the rest of the business (instead of everyone pretending such nasty things don’t exist). Doing so allows the entire business to tackle those genuine challenges together.

To show what such a conversation might look like, I’m going to develop this approach in detail, in the context of a story.

Welcome To <Company X>, Here’s Your Spec

Let’s say you’ve started at a new job, leading a small team of engineers. On your first day, an Important Person comes by your desk. After some welcome-to-the-business chit chat, he/she hands you a spec. You look it over—it describes a new report to add to the company’s product. Of course, like all specs, it’s pretty vague, and, worse, it uses some jargon you’ve heard around the office, but haven’t quite figured out yet.

You look up from the spec to discover that the Important Person is staring at you expectantly: “So, <Your Name>, do you think you and your team can get that done in 3 months?”

What do you do?

Here are some possible approaches (all of which I’ve tried… and none of which has ever worked out well):

• Immediately try to flesh out the spec in more detail

“How are we summing up this number? Is this piece of data required? What does <jargon word> mean, here, exactly?”

• Stall, and take the spec to your new team

“Hmm. Hmm. Hmmmmmmmm. Do you think, um, Bob (that’s his name, right?) has the best handle on these kinds of things?”

• Give the spec a quick skim, and then listen to the seductive voice of System I

“Sure, yeah, 3 months sounds reasonable” (OMG, I wish this wasn’t something I’ve done SO MANY TIMES).

• Push back aggressively

“I read this incredibly convincing blog post ¹ about how it’s impossible to commit to deadlines for software projects, sorry, I just can’t do that.”

Here’s the thing about all of the above: they’re basically guaranteed to fail. By which I mean, specifically: no one is going to be any kind of happy about the software that gets written from the above starting points.

OKAY, ENOUGH STALLING, SO WHAT DO I DO, DAN?

I’m going to suggest something that may sound a bit odd: while this Important Person is standing at your desk, use this opportunity to, politely but ruthlessly, interrogate them about the business you have just joined.

What is the business model? What are the biggest challenges facing the business as a whole? What risks does leadership worry most about? What are they hoping happens, if everything goes just right? Who is the current customer for the product? What motivates that customer to buy? Are they happy after they buy? If not, why not? What other customers would the Important Person like to go after, if he/she could?

One way to understand this is: there is some central problem or challenge which the business is facing. Your first job is to figure out what that problem is, and, just as importantly, what words the Important Person uses when they think about that problem.

A very important thing: it usually takes a considerable bit of effort to get beyond the proposed solution (e.g. the report), to the actual underlying problem. Laura Klein summarizes this marvelously as “[People] will tell you that they want a toaster in their car, when what they really mean is that they don’t have time to make breakfast in the morning.” She’s talking about user research, but I find the same perspective is incredibly useful when talking to, e.g. CEO’s.

Returning to our example, let’s say that, as you talk to the Important Person, you come to understand that your new business, which sells software via a monthly subscription plan, has a serious problem — too many customers are canceling every month. What’s more, you’ve joined a startup, and, although it has a solid chunk of cash in the bank, the leaders very much want to ramp up how much they spend on sales and marketing. Of course, doing that will burn through their cash, and thus require raising more capital sooner than later. And getting VC’s to invest more money with that high cancel rate is going to be very difficult, if not impossible.

You’ve been hired, at some level, to help solve that problem. Even if the people who have hired you don’t think about it that way.

Now that you understand that central problem, take one more step: figure out exactly how this proposed development effort is supposed to solve that problem.

How and why does the business believe that this report is going to lower the cancel rate? What makes the Important Person think it’s going to work? Are there any ways they’re worried that it might not work? Are there any key questions they’d like answered sooner than later?

Oh, How People Love To Hear Their Own Words

A key tip for these conversations: at each step, it’s really helpful to echo back what the person just said to you. E.g. “Okay, let me make sure I understand — you’re saying this new feature you want is critical because it’s going to help us upsell existing customers, but we’re not so much expecting it to help us get new customers? Do I have that right?”

At each of those little checkpoints, if you’re right, the Important Person will feel this rare, pleasant sense that someone in development actually seems to understand how the goddamn business works. If you’re wrong, you’ve just narrowly avoided basing your dev efforts on an imperfect understanding of the business (which is a path straight to misery).

Note that template: a) “I’m going to echo that back, make sure I understand”, b) echo it back, c) “Do I have that right?”. I say exactly those words, basically every time I talk to someone about a new project — so much so that my partner Edmund calls it “pulling a Milstein”. You don’t have to be clever with that template, is what I’m saying — put all your cleverness to work really listening and trying to understand the problems facing the business.

This whole process takes practice, but is INSANELY VALUABLE. You can (and should!) start by asking everyone you work with about how they understand the overall business you’re currently in, and what challenges it’s facing. Do the same with random people you meet. Be curious, don’t stop being curious, and don’t be in any way afraid to say “I don’t understand that, can you explain it to me?”

Now, The Knockout Punch

Once you both understand some central problem facing the overall business, and how your proposed bit of development effort fits into a possible solution, you wrap all that up and deliver it back, repeating as many of the words they used as possible, e.g.:

“Okay, if I understand it properly, we’re adding this report, because we think we can use it as a key feature in a new, higher pricing tier. This more expensive tier is not really for acquiring new customers, it’s more for upselling existing ones, so we can extract more revenue from our most engaged customers. If we can do that, it’ll have a potentially big impact on our revenue churn ², which is the most important number in our business right now. And, we really need to see that move in the right direction, in the next 6-9 months, so we’ve got a good story to tell investors when we go out to raise our next round of financing.

Do I have that mostly right?”

With even a modest bit of luck, at this point, the person who handed you the spec will have a cautiously hopeful expression on their face, and they’ll nod as they say, “Yeah, that’s… um… that’s pretty much exactly right.” ³

You then say, “Great, let me look into the tech we need for that report, and I’ll get back to you with more info.”

Note: you haven’t promised any date by which the report will be finished. Instead, you’ve demonstrated that you are going to work with this Important Person to solve the actual problems the business is facing. And those problems involve very real, very hard, external deadlines (e.g. running out of money by a certain date).

One way to see it: you’ve taken a key first step in earning their trust.

Now, notice, too: instead of you having made some promises to deliver on a spec, which promises are now hanging over you and making you nervous, you’ve directly engaged in a real problem for the business. And you have plenty of room to be creative about how you solve that problem. Yes, it’s a hard problem, but that’s why you got into this business in the first place — for the joy of solving hard problems that actually matter to someone.

Man, If Only We Knew What To Do

The next day, you meet with the team, and discover that the new report is mostly straightforward, except for one thing: it requires a periodic import of data from a new social network with a complex API. The team has just started working with that API, and they tell you that they just don’t have enough information to make the call on the 3 month deadline, either way — it’s certainly possible they could hit it, but there’s every chance things could blow up.

What do you do?

You could tell the Important Person that you don’t know. That is, at least, honest. But it doesn’t really help them (aka help the business solve its problem — move revenue churn in the right direction, before the next round of funding).

What would help you solve the business’s problem?

One key is that the business as a whole is trying to make a decision — about how to spend your time.

If you knew for certain that you could get the report built in 3 months (and that existing customers would happily pay more for it), the right decision for the business would be: build it.

Conversely, if you knew for certain that you couldn’t hit the deadline (or that existing customers wouldn’t pay more), the right decision would be: stop immediately, start some other plan to reduce revenue churn.

Given that you don’t know which of those two alternatives you’re living in, what you (and the business) need is: more information.

If you could obtain that information, you could make the right decision, which would make your business a great deal more money than the wrong one.

In the presence of uncertainty, acquiring information is often the best way to generate value. And, yes, this is the point in this blog post where I tell you to go read Donald Reinertsen’s Principles of Product Development Flow.

So, what you do is, you pick what you work on next, to gather as much information as possible, about the things you are most uncertain about. If you’re clever (and you are! That’s why you got into development in the first place), you can find a way to gather information as part of the process of actually building the thing. Meaning, you usually don’t need to conduct some separate, research-y phase—instead, you can gather the information you need by doing your work in a careful sequence.⁴

And, crucially, you have to be completely up front about all this with your counterpart on the business side.

The Meeting Where You Earn Your Salary

In our story: you schedule a meeting with the Important Person, and, in advance of that meeting, you bury yourself in the technical details of what the team has found about that new API so far. You also do some chatting with the sales and marketing folks — you want to understand the target customer, and which of their problems the business is hoping to solve with this new report.

Then, at that meeting with the Important Person, you say something like:

“Right now, we’re feeling optimistic that we’ll have that report ready in some form within 3 months — but our biggest risk is working with that new social network’s API. From the initial investigation we’ve done, it looks like, at the very least, we’d definitely be able to show them <minimal data foo>, which, from what I understand of our engaged customers, might be enough to trigger upsells, but sales and marketing aren’t certain.

We’d like to propose the following: we take two of our best devs, and they spend 2 weeks trying to build a full integration with the social network, purely on its own, so we’ll have a better understanding of just how much data we can pull in. While they’re doing that, we’d also like to have our front-end devs building mockups of a report with just the minimal data, so that you’ll have something to do some user research with, and possibly even use for sales demos if things go well.

Does that plan sound like a good way to go?”

This little speech is, basically, the most important thing you’re going to do at your job all month. So I want to unpack it in some detail.

First off, note that, because you’re thinking in terms of risks and information, you propose sequencing the work to get as much information, as quickly as possible (e.g. information both about how much data you can get from the social network, and also about whether or not customers will be satisfied by the minimal data set). When you’re facing a chain of risks, you’re going to generate the most valuable information by attacking the biggest risks first.

Second, it should be clear that you can only pull this off if you deeply understand the overall business problem — that’s what lets you propose the minimal data thing. Generally, those opportunities emerge bottom up, as, e.g. a dev figures out what data is / is not easy to obtain — but the value is not always clear to those devs (the very best way to run this game plan is to make it so that all the devs really deeply understand the overall business problem).

Third, it’s important that you’re offering the Important Person an actual, meaningful choice. You’ve clearly stated your current knowledge of what is possible (e.g. the technical risks and opportunities), plus your current understanding of what is valuable (to the business). You’ve framed that in a way which lets the Important Person now make a choice about what to do next (which will often result in you learning that your understanding of what is valuable to the rest of the business is no longer accurate — that’s a very, very good thing).

Fourth, note that, when you work this way, there are good risks (we call them “opportunities”) as well as bad ones. You discovered something unexpected — you could quickly and cheaply build a simpler report that might work. One of the most fun things about this approach is finding those wins — it’s tremendously exciting.

Finally, notice how you’re explicitly operating with a full knowledge of the hard, external deadline facing the business. You’re not talking about deadlines for implementing a spec, but you are talking about deadlines for the overall business… which are the only ones that actually matter.

What Happens Next

Now, the Important Person might say any of the following:

a) “Great, go for it”, (you say “Thanks, sir/madam, we’ll see you in two weeks with more information + options”)

b) “That minimal data would be a fantastic report — I’m certain we can get upsells with that” (you say “Awesome, we won’t put the devs on an exploratory full backend integration, we’ll sprint ahead on getting the minimal data ready asap, we should have an early, crude prototype to look at it within a week or two.”)

c) “That minimal data is absolutely not enough” (in which case you say, “Okay, would you like to see other options for restricted data?”, or, “Hmm, I’d love to better understand what questions we’re trying to answer with this report, since I don’t feel like I quite get it yet”, or even, “Well, in that case, maybe we should explore some other options for reducing revenue churn in parallel, because there’s a real chance we won’t be able to make this report work in time.”)

Note that that last situation, is not, in any way, a failure. You’ve learned something very important — the business folks believe that the current plan centrally depends on something which has a great deal of risk associated with it. Armed with this information, you can both try to drive down that risk, as aggressively as possible, and also start working with them to prepare other plans, so you’re ready if things blow up.

Overall, what this approach means is that you will be constantly adjusting your understanding of what is the most valuable way to spend your time, and constantly keeping the business folks in the loop + offering them meaningful choices. This is not, in any way, “we don’t need no stinking estimates, we’re code cowboys, just trust in the full force of our awesomeness.” It’s turning the entire process of software dev into an ongoing conversation with the rest of the business, where information is quickly getting into the hands of people who can make decisions about it. And, where “information” means both things that you know/have learned, and also an understanding of what you don’t yet know — i.e. important risks.

As I said in my previous post, writing software means learning something in such precise detail that you can tell a computer how to do it. More broadly, if creating new software is important to a business, then the business as a whole must engage in a learning process — not just the developers.

Hmmm, This Doesn’t Really Feel Like a “Process”

Inevitably, my solutions to this feels somewhat personal — but that is not an accident. Fundamentally, we’re talking about two groups of people having to build up trust in each other. Trust about things that they will not, in general, be able to verify.

Specifically, developers have to trust that what they are being told about the rest of the business is true — that customers want what they’re building, that the long hours are actually needed (and aren’t just some middle manager showing that he knows how to crack the whip — I wish I didn’t see that happen, far, far too often).

And the rest of the business has to trust that the developers, when they go off into their weird, opaque world, are honestly reporting back on what is possible, how much effort is involved, what they’ve achieved, etc.

Any means of building up that trust will always have a personal flavor — it exists between human beings who have learned something of each other. It’s not a thing you can mandate or fix with an imposed process.

Absolutely anyone who has done any real work on either side of that divide can immediately call up instances of that trust being betrayed — of discovering that all your work for the last half year was meaningless (and that someone knew that and didn’t tell you); or that the repeated promises that some system was ready to launch collapsed in a fiery wreck as soon as the first user tried to login.

Sometimes, The World Is Telling You To Polish Up Your LinkedIn Profile

A severe warning: this whole plan can fail, badly, if the Important Person is, well, not very important. Specifically, say you have a strongly hierarchical structure, where some middle manager is the only person you’re allowed to talk to. It can be the case that such a person perceives their job as taking proposed solutions from upper management and getting a bunch of developers to implement them. Such a person can be very threatened by the idea that you want to get beyond the proposed solution, to the underlying problem. They can hear that as “I’m going to have to go back to my boss and tell them that a bunch of developers think their idea isn’t very good.”

When a boss says “Jump!”, this kind of person prides themselves on saying “How high?!” Since I’m instead proposing “Why are we even jumping, here?”, you can see how there can be problem.

Furthermore, such a person will often put a really strong value on preventing the flow of information up (from developers to people who can actually make decisions). They may think of that as “Not troubling the boss with the details”. But, as I’ve described above, such a block on the flow of information is absolutely deadly to software development.

So, what’s your best option if you find yourself in such an unfortunate situation?

As I see it, there are two paths ahead.

Option 1: Try to get the middle manager to see this new way of working as something that will make them look good.

I rarely see this work, but it can be worth a shot. My partner Edmund reports some success trying this by way of: a) find an ‘internal’ thing, where the middle manager is, like, a user of the thing , b) propose to them that you work on that internal thing this new way, and then, c) if that produces a thing they find really useful, help them see that their boss can feel the way they feel now.

But, as above, that’s something of a long shot. Which leads us to…

Option 2: Quit.

I don’t say this casually. If you’re stuck in the situation I’m describing, it’s overwhelmingly likely that your project is going to end in some form of unpleasant failure. And, what’s more, it’s extremely rare that you can get higher-level leadership to see any problems with such a middle manager — in general, such a person has that job precisely because they fit into higher-level leadership’s mental model of a manager. In which case, the entire org is going to be set up in a way which makes it hard or even impossible to write useful/valuable software.

If you’ve been stuck in such a situation for a while, I’ll just say — you may have forgotten how great it feels to solve meaningful problems for people. Go find a place where you can do that.

Your Mission, Should You Choose To Accept It

In summary, I’m saying: 1) become a student of the overall business you are in, 2) sequence your work to extract as much information from reality as early as possible , and 3) make risks and opportunities the centerpiece of an ongoing conversation with the rest of the business.

There are no certainties in this world, but that approach will let you tackle the uncertainties together.

And that, I can tell you from fortunate experience, is a profoundly satisfying way to work.

But, Wait I Want To Learn More

I’ve stolensynthesized just about all of the above ideas from a bunch of very smart people. You should totally go read their books and blog posts. ^{5 6 7 8 9 10}

And, this December, I’ll be speaking at the Lean Startup Conference, in San Francisco, on “Risk, Information, Time & Money”. Major bonus: almost everyone I list in the above footnotes will be speaking there, too. Last year’s conf was great — I gave a talk on How to Run a 5 Whys (With Humans, Not Robots), learned a ton from other speakers.