A growing number of developers, rallying behind the Twitter hashtag #NoEstimates, believe that estimation is a deeply flawed software development methodology.
Estimates, according to these developers, are not only imperfect and ineffective but an unnecessary evil invented by businesses to pressure developers into meeting unrealistic deadlines. Naturally, as with all great programming debates, advocates believe they are fighting a Holy War.
As someone who, at different times, has been a developer, manager, product owner, and now a business owner, I can confidently say that estimation is not your enemy. In fact, estimation is one of the best tools we have for measuring velocity, efficiency, and productivity, and therefore is one of the best tools for dramatically improving your performance as a developer. Peter Drucker, the author of The Effective Executive, said it best: “If you cannot measure it, you can't improve it.”
The NoEstimates crowd is blind to these benefits because they believe a series of untrue myths about estimation. These myths are rooted in a misunderstanding about the purpose, execution, and value of estimation for developers, managers, and clients. The following series of articles will dive deeply into software estimation starting with these myths of estimation, why developers should learn to estimate effectively, and the proper mindset one must have for estimation. Further articles will expand on why estimation is necessary, why understanding the work is important, how to estimate effectively, and finally how to work against estimates.
The narrative from the NoEstimates movement is that estimation benefits management while developers lose. But estimation is not a zero-sum game, all parties can and do benefit. Estimates allow management, among many things, to gauge developer efficiency, manage product owner expectations, and create realistic timelines. All of these are extremely valuable to any business, which means that developers who can provide accurate estimates are highly valued. Further, developers can easily demonstrate their productivity and ability to finish a product on time, by consistently completing work within the estimate. Providing a highly valued skill to employers will result in increased leverage the employee will have over the organization as well as create separation between themselves and other coworkers. In short, estimates help, not hinder, developers in garnering higher wages and job security.
Developers do not need to worry about whether estimates benefit managers and businesses. As long as they focus on themselves, getting better at estimates and hitting them to garner better wages, everyone will win. This can nicely be summed up by term, The Invisible Hand, coined by Adam Smith:
“...he intends only his own gain, and he is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention.” - Adam Smith, The Wealth of Nations
Sadly, some managers do use estimates to force unrealistic deadlines on developers. But this says more about the culture of the management team than it does about estimation itself. A team that abuses estimates will likely abuse other methods to force unrealistic deadlines anyway. The problem is with management demanding unrealistic deadlines, not estimation as a tool. Hammers can also be used for harm, but nobody argues that hammers are not effective when used correctly.
First, stress is not unequivocally bad. One technique in Cognitive Behavioral Therapy (CBT) is to list all your negative emotions (angry, fearful, anxious, stressed, etc) and assign a percentage for how much you are feeling that emotion in a specific situation. Then go through each one and assign a percentage that you think you should be feeling. The point of this exercise is to realize that in every situation there is an optimal level of stress we should be feeling. Feeling stressed or anxious about an upcoming job interview is not only perfectly normal, but it is also necessary for you to be on point. Achieving 0% for all negative emotions is an unrealistic goal in a modern world filled with bills, relationships, and family dynamics all out of our control. Unless you plan on becoming truly enlightened and sweeping a monastery for 12 hours a day, you should plan on optimizing your stress levels to help you achieve your goals.
While estimates certainly can cause stress, they by no means need to increase stress beyond an acceptable and manageable level. By understanding how estimates work, how to think about estimates, and finally how to use estimates effectively we can reduce stress to a minimum. This article, and subsequent articles, will cover each of those points in sections below.
The phrase “a task takes how long it takes” implies there is an objective, absolute, set amount of time a task will take to complete. This may be true if we’re talking about the absolute shortest amount of time it would take to complete a task, in the same way, we calculate the absolute shortest amount of time we can get from LA to NY using different modes of transportation currently available. But the maximum amount of time a task could take is infinity, or the lifetime of those involved, due to procrastination, roadblocks, and so on. How long a task actually takes lies somewhere between these two extremes, depending on the productivity and focus of the team.
The goal for all teams should be to get closer to the minimum and always seek improvement. Developers who refuse to estimate and say a task will “take how long it takes” risk falling into the trap of Parkinson’s Law, which states that “work expands so as to fill the time available for its completion.” I encourage you to try an experiment to test the two methods. One you plan out, estimate, commit to hitting the estimate and make pragmatic decisions towards the goal of hitting the estimate. For the second one, simply work until you are finished. I guarantee you will have a much higher chance of finishing the first project quicker than the second.
We wouldn’t call them “estimates” if we were able to perfectly gauge the amount of time a task would take. Scientific measurements are only accurate to a specific decimal place, yet they are still useful. Medicine is not perfect, yet it still saves lives. There are many things in the world worth doing where perfection is impossible. A basketball player will never be able to make 100% of all their shots, but 45% is much better than 25%. In the same way, estimates can still be incredibly useful despite being imperfect.
Both of these views are perfectly valid to hold. There is nothing objectively flawed about either argument. They are subjective statements that both sufficiently describe estimation. The question is not if they are valid, the question which one is more useful to you?
The myth that various skills are natural or inherited is found in almost all possible activities. The reality is that skills are earned through hard work and practice. Anders Ericcson’s book Peak illustrates this point excellently by dispelling the myth that perfect pitch is innate by telling the story of Mozart’s childhood.
By the time he was six, Mozart was performing for royal courts across Europe. One might assume that being so skilled so young, Mozart was born with perfect pitch and the innate talent for music. However, Mozart’s father was obsessed with music and the first few years of Mozart’s life completely revolved around music. His older sister performed for money from an early age. Mozart himself was being taught how to play by the time he was three. Essentially all of his childhood was spent listening to or practicing music. It is no surprise, with that amount of exposure and practice, that Mozart had perfect pitch at such an early age. Ericsson gives another example of a teacher who successfully taught an entire class of students to have perfect pitch with an hour or two of lessons every day for a few months.
As with perfect pitch, estimation is a learned skill, not an innate talent. Anyone can improve their estimation skills if they deliberately and consistently practice.
Now that we’ve debunked the myths of estimation, let’s look at why estimating is beneficial to developers.
The most efficient, most productive, and most valuable team members have the most leverage because businesses obviously do not want to lose them. With leverage, comes compensation, freedom, trust, autonomy, and pretty much every positive aspect of work. The equation to gain leverage, therefore, is surprisingly simple: consistently grow, take on more responsibility, maximize your value, and therefore maximize your leverage. Finally, use that leverage to ensure you get what you want out of your work life.
If you want leverage and to take your career seriously, then you need to have an active interest in honing your craft. To this end, developers are endlessly obsessed with honing their technical chops. They go to conferences, read blogs, do side projects, follow influencers, take courses, and more. But none of those activities will have as drastic of an effect on your career as honing estimation because it is the one thing in software development that affects all others. Being good at estimation requires that you work in a consistent manner, have strong self-discipline, and be able to deliver on time. Just like any other skill, you can improve your estimation and delivery on the estimates with practice, measurement, and adjustment. Very few give anywhere near the kind of rigor towards getting better at estimating as they do to increasing their technical skills because they do not see the immense effect it can have on their careers.
In short, learning to estimate effectively is the best way to grow and demonstrate value to your employer, and therefore is the best way to maximize your leverage and reap the benefits therein.
The reality of a programming career in the modern world is that it is nearly impossible to avoid estimates because the majority of companies and clients want them. No matter how badly you want them to not exist, most businesses use them for quarterly planning and budgeting. From a purely practical standpoint then, it is wise to accept the inevitability of estimates and take learning how to estimate effectively, seriously.
This article is for developers, not managers, product owners, or business owners. It is directed specifically at the hard-working, go-getter with aspirations of being the best developer on the team. It is not for those who want to skate-by, blend into the background noise or otherwise disappear into the depths of some large organizational monopoly that allows for dead weight. If minimal effort is your desired goal and “good enough” is your motto, then these articles are not for you.
This article will also not help you find existential meaning in your work. Occasionally people will say: “If you don’t love what you are doing every moment of the day, you are doing it wrong.” In my experience, this is a naive sentiment, every job, no matter how glamorous, can be difficult, monotonous, boring, challenging, undesirable, or “beneath you.” Accepting that you will not always love every part of your job is important to finding happiness and satisfaction in your work.
The steps in these articles are not shortcuts, hacks, or magical formulas. Jocko Willink, former Navy SEAL and author of Dichotomy of Leadership, says it best.
“The shortcut is a lie. The hack doesn’t get you there. And if you want to take the easy road, it won’t take you to where you want to be”.
The “road” that this post describes is not easy, it is difficult and that is the point. If it were easy, everyone would already be doing it and it would be a requirement rather than something that makes you stand out. This famous quote from JFK describes how I approach everything that I do in my work.
“We choose to [do these] things, not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone...”
JFK perfectly describes what we should all aspire to achieve in our work life. Work should be challenging. This challenge should serve as a goal to “organize and measure the best of our energies and skills.”
Following the framework described in this article is a similar kind of challenge that requires effort, dedication, and having the right mindset and ability to overcome excuses.
The two biggest challenges I have seen developers face when adopting estimation described in these posts is not having the right mindset and allowing excuses to defeat them before they have even started.
The criteria for this to work is as follows:
1) You want to be the best developer you possibly can be.
2) You are willing to put in the work consistently to get there.
3) Once there, you still work are willing to work hard every day to be at the top of your game.
People know how important mindset is to sports and athletics but rarely apply it in other aspects of their lives. If you ask any athlete they will tell you that in many cases the difference between extremely successful and mediocre can simply be their mindset. A major difference between the two mindsets is one gives in to all the excuses their brain comes up with while the other doesn’t.
Excuses are easy. Excuses are a dime a dozen. There are literally infinite excuses that you could come up with for why something didn’t get done. Excuses almost always sound totally reasonable despite the fact that there are still people who manage to run marathons, spend time with their kids, and continue to excel at work. This is the same dynamic at play when you approach an estimate. While working, your brain constantly tries to convince you that there are many valid excuses as to why you cannot finish on time or on budget. You must actively fight these excuses.
Excuses always sound like legitimate problems that really prevented the work from being completed. The problem here is that the job of the developer is to get the work done in spite of excuses. Remember that at the core, your job is to solve problems and achieve the goals of the stakeholders. Most of the time, excuses are simply a list of the things that need to be done. For example, saying: “I couldn’t get my work done because it was unclear which encryption we were supposed to be using.” Figuring out which encryption is the job and is part of the task. What one person could legitimately consider a blocker, another could consider a challenge that they can and will overcome.
Periodically there are real excuses. Excuses that could not be avoided and are understandable, however, this is rare and the exception. You had a death in the family and you had to take time off work or a third party vendor broke their API and delayed your release. If you find yourself leaning on excuses for why you are consistently missing reasonable estimates, you are either the most unlucky person in the world (hint: you are not) or you need to reframe your thinking. You have agency to overcome all of these issues if you approach your work with the right mindset and ownership. Do not be tempted by the easy road and excuse every miss. Take ownership over your work.
The best part about having the right mindset is that the folks who have the wrong mindset are your competition. This is great news for you because they are going to be easy to outperform. The correct mindset combined with the estimation framework described in this post will set you on an ever increasing upward trajectory.
This article will go into more depth about the underlying motivations for estimation and explore a detailed framework for estimating effectively. Implementing these changes into your workflow and life will be challenging, but do not confuse difficulty with complexity. The framework described in this series is simple at its core. It is analogous to losing weight. There is nothing complex about getting in shape: eat fewer calories, eat healthier, and exercise more. Thousands of people struggle to lose weight effectively because, while it is simple, it’s also very challenging to show up consistently. Consistently leveling up your skills, efficiency, and productivity to provide the maximal value to the stakeholders is as simple, difficult, and boring as getting in shape.
Not spending sufficient time to fully understand both the requirements and the goal behind the work is one of the biggest mistakes developers make when estimating. Often they will look at the technical aspects of a particular story or feature and ignore the business or project concerns that underpin the motivation for the story in the first place. Other times they will do a cursory scan of the story without diving into all the details particular to that project. Estimating is difficult enough when having a full grasp of the scope of a story, not having this knowledge makes it even more difficult. Therefore, the first step to providing a realistic, achievable, and accurate estimate is to spend the necessary time to fully understand the story’s requirements, the goals behind the story, why the story was prioritized, and why the story was assigned to you specifically. I call this goal-based thinking. The following will cover goal-based thinking in a series of steps that must be completed before providing an actual estimate.
Stakeholder: A person, group or organization that has an interest or concern in a scope of work.
For any given scope of work, you will need to ask yourself:
Asking these questions will help you understand the goals behind the project. Many developers assume that the product owner or customer is the stakeholder, but this is a mistake that will lead to problems down the road. If the true stakeholder is the CEO of the company then they will have a very different set of expectations for the work than the product owner. If the stakeholders are end-users of the product as well then they will have very different expectations for how the software should look and feel. If the product owner is pushing for certain changes that do not fit with the vision from the stakeholders, this might cause problems down the road. Identifying who the real stakeholders are will help you make better decisions about the product and ultimately how work should be estimated. Let’s walk through a scenario and see if we can identify some stakeholders.
A product owner asks if you can help him build a new workflow for an accounting system. They give you the high-level overview: “We would like to create a workflow that allows our accounting employees to add invoice payment details into our system.”
Let’s answer the questions provided above:
So now we have these stakeholders identified:
With a little digging, the whole project reveals itself to be much more complex than it originally sounded. But we also have a much clearer picture of who and what is involved with this project. The most important part of the work is building the correct foundation and mindset for each party involved, not the actual programming. Without identifying the stakeholders this would not be clear and the team would be completely focused on the technical aspects of the project instead of the stakeholders. Taking this step seriously is vitally important groundwork for the next step: Goal-Based Thinking
Imagine a story, like “Add test coverage to onboarding workflow”, is assigned to your coworker. The coworker reads the story without asking any questions but proclaims they can finish it within the sprint. Nearing the end of the sprint, the work is turned in for your review. Running the test suite reveals the test coverage to be 95% over the section of code they were assigned, which is obviously a great start.
Digging into the actual test code reveals the majority of tests contain assertions that, while touching lines of code, are not meaningful. The majority of what was added was frivolous, brittle, and unnecessary tests. Your colleague responds to these comments by saying that the story was to “increase the coverage” and that’s exactly what they did. While technically correct, this is clearly not in alignment with the goals of the stakeholders. One could argue that this is an issue with the story itself not indicating that the tests need to be meaningful, that is, the story was not fully specced out with all the details. But anyone who’s ever specced out a story before knows there’s no way to add every single detail to a story to cover all possible interpretations and requirements. Part of a developer’s job when they receive a story is to clarify the acceptance criteria so that they understand exactly what is expected.
Everyone has had the experience of describing what they want only to receive work that “technically satisfies” the requirements while completely missing the desired outcome. Goal-based thinking greatly improves the probability that the result is what we want. When a developer understands the goal behind the work they will have a much greater chance of turning in work that aligns with that goal.
To nail this point home, let's look at another profession where goal-based thinking is crucial: construction workers. A high-level goal for most construction projects is to ensure the safety of the people who will use that building. Nailing boards, screwing in screws, putting up walls, setting tiles are simply the implementation details of the overall goal. If they accomplish all of those implementation details perfectly, but in the end, they create an unsafe building, then they’ve still failed.
A developer’s job is not to turn in code but to solve a problem, usually a business problem or project problem. It is not to have a high degree of test coverage or to follow a specific pattern. Your job is to solve problems that achieve the goals of the stakeholders. Keeping this in mind is crucial to becoming successful.
Figuring out the goal behind a particular task can be difficult if you’re not used to goal-based thinking. Let’s use the example above to illustrate how to find the goals and reasons behind tasks.
In a surface level reading of the story, the original task, “Add test coverage to onboarding workflow”, can be expanded to mean: “Increase test coverage in the project.” This a great first step because it reveals that the original task is a part of a larger goal for the whole project instead of one specific aspect. But we need to dig deeper and ask “why do we want to increase test coverage in the first place?” A naive answer is to make the CI and test coverage metrics look better or fit some arbitrary numeric metric that the programming community has decided is a good number to hit. But higher test coverage in itself is not useful to a project and certainly not to a stakeholder. Looking deeper into why developers and project managers push for higher test coverage reveals at least two, more meaningful goals:
Stopping at this level for a moment, it is already obvious why the work our coworker did above is completely useless for these two goals. His tests focused on increasing metrics of code coverage but not ensuring there are fewer bugs or making the codebase easier to maintain. In fact, his work gives a false sense of security against bugs by artificially increasing the test coverage metrics. Additionally, the team now has to maintain brittle, useless tests whenever new features are introduced.
But we can dig even deeper and ask: Why do the stakeholders want fewer bugs and a more maintainable codebase? Fewer bugs mean more time for developing new features, implementing new designs, etc, etc. An easier to maintain codebase makes developing new features easier and onboarding new developers. The reasons here are extensive and I won’t attempt a comprehensive review of all of them, but the benefits of increased coverage impact much of the overall product goals and outcomes.
Each task has a whole hierarchy of goals that will never be fully described in each ticket description. Several layers of goals need to be kept in mind for each story to ensure that the work you are doing aligns with each one. Failing to meet one goal of a project can be detrimental to the whole project. Take these series of articles about estimation as an example. The goal of these articles is as follows:
If these series of articles about estimation achieved the first three goals but tanked efficiency and productivity for the businesses, this whole framework would be a failure. The business would eventually die despite you having leverage because the business would not be productive or efficient. Achieving all the goals of a task is important.
After understanding the stakeholders and the goal behind the task, you need to fully and completely read the story. This might sound trite or trivial but many developers do not take this step as seriously as they should. Read all of the acceptance criteria, review all of the mockups, read all of the notes and comments, and read any discussions that led to the story thoroughly. During this time you need to make a note of anything that is unclear or confusing and bring this to light to your project manager or coworkers. If anything has been left out of the acceptance criteria this is the time to make that known.
Once you are confident that you understand the story completely, you need to verify any assumptions about what is expected. The stakeholder or PM will have specific expectations about what work should be done and what the completion of the story entails. It is important that your general assumptions of the story and scope of work are in alignment with what the stakeholder and PM expect. A good pattern to follow comes from Steven Covey’s book, The 7 Habits of Highly Effective People:
“Seek first to understand, then to be understood”
Most people approach new work with the intent to understand enough to get started. A better approach, Steven Covey would say, would be to understand the problem well enough to be able to explain it to your colleagues or stakeholders. Many teams have found success by requiring the developer to restate, in their own words, the problems they are trying to solve and the goals that will be achieved by completing the work.
The last step is to ensure that all parts of the work have been laid out and expressed fully. The goal of completing the work is to satisfy the stakeholders so that you can move onto another story or task. Making sure that the requirements for the completion of the story are fully specified is vital to stemming scope creep and making measurable progress towards the next milestone. Scope creep is a notorious problem in software development and a savvy developer will mitigate against this during this step.
When you do find that a story is simply not ready to work on, the best course of action is to create a “Spec Story”. A Spec Story’s acceptance criteria describes the missing pieces of the original story that still need to be defined. When a Spec Story is complete the original story will be ready to begin working on. The reason this is so effective is that it follows the exact same pattern as all the rest of the stories in this framework, you plan, budget, and estimate a story’s requirements. Planning and budgeting your spec stories earlier will ensure that the rest of your stories are still on track with the appropriate velocity.
Now we can focus on the details of estimating effectively and practically. The following framework has proven extremely effective in providing realistic estimates that work for both clients and developers in my experience.
The three pillars of estimating ensure estimates are accurate, realistic, and always improving. If your team is consistently providing poor estimates then it’s likely they are not following one of these pillars.
Do not allow estimates to come from non-technical people for technical projects. Understanding the intricacies of how a system works is crucial to providing any type of realistic estimate.
Compare your estimates to similar estimates from colleagues and competing companies. Nothing will tell you better if you are in line with market expectations than comparing against competitive bids.
Keep track of your estimates over time to gauge future estimates.
The gap between what a developer thinks is done and what the stakeholder thinks is done is one of the biggest sources of conflict and misunderstanding between the two. I call this the “Done vs Done” problem. When this gap is large, developers will not only underestimate how long tasks will take but turn in unsatisfactory work that does not meet the stakeholder’s expectations.
Developer Done: The story is done when they get a minimally viable working example of the feature or story.
Stakeholder Done: The story is done when the feature implemented, acceptance criteria met, edge cases addressed, error states fully handled, states when there is no data are handled, code has been reviewed, feature fully tested, approved by QA, approved by UAT, Continuous Integration passes, and the feature has been deployed and is actively used in production without bugs.
The disconnect between these two versions of “Done” can often be this large especially for developers new to estimation. Let’s look at a few of the reasons why developers will underestimate the requirements of a story so severely.
The best strategy to mitigate reason two and three is to use an estimation checklist to ensure everything is accounted for in the estimate. Pressure from stakeholders, on the other hand, can be managed by setting appropriate expectations early on in the process.
Pilots, even highly experienced pilots who have been flying the same planes for 30 years, use checklists to ensure that they have done all the pre-flight procedures because human memory is fallible and it is extremely easy to let something slip your mind. Checklists are a great way to mitigate mistakes and misses. The most effective checklist is the checklist that you create yourself because it will be personalized with concerns and aspects that affect your situation directly. This checklist should be continuously updated as you grow as a developer so that each new estimate incorporates aspects you might have missed last time around. Here is an example of where to start:
Depending on the type of work you are doing, this checklist could vary drastically but the important step is that you are adding to it as you encounter missed estimations and reflect on how you can improve. Save this in a text document or sticky note so you can always refer to it when making estimations. Shared estimation checklists between teams can be even more effective since the whole team will be on the same page and updating it as new cases are identified.
Stakeholders often start with unrealistic expectations because they want the project to be cheap, fast and high quality. Naturally, as the saying goes, you can only pick two of those at a time and let the third suffer. The idea that all three can be done at one time on a software project comes from stakeholders who are not familiar with software development. They might hold misconceptions, often contradictory, about what is reasonable, realistic, and practical for estimates and project timelines. Therefore, to avoid conflict upon delivery, you must help stakeholders better understand the software development process and dispel any misconceptions they might have.
Contending with misconceptions like the ones above can be quite difficult, especially the ones that are self-contradicting. The point of setting expectations, and contended misconceptions, is to make sure that stakeholders are properly accounting for the reality of software development. Let’s explore these misconceptions in more depth to help you dispel these myths for stakeholders.
The construction industry spends a massive amount of time and effort planning a project. Architects, structural engineers, civil engineers, draftsmen, designers, and others put an amazing amount of work before even starting a project. Skyscrapers require detailed specifications and drawings down to the type and location of individual screws. Software development is not approached in this way in most cases. The nuts and bolts of software are left for the actual development phase because there are many unknowns like the capability of various third party packages and the errors, limitations, and roadblocks that might arise. In many cases, even business logic is left for the development phase.
Convincing stakeholders to take the planning phase of a project seriously is difficult. In many cases, you want more mockups, documentation, and specs than they have provided. The best move in this situation is to help stakeholders understand that each release will have a substantial UAT phase where the length of the UAT phase is proportional to the lack of robustness in the design phase. Presenting it as a tradeoff will force stakeholders to choose between spending more time in design/technical-discovery or more time in UAT or Beta phase. The key is to make sure the stakeholders understand and agree to this tradeoff. This sets the expectation that a lack of planning will lead to increased changes, budget, timeline, and UAT phase.
Anyone who has been on a software project for any length of time should recognize this is false. If a stakeholder expresses sentiments along these lines then you need to push back immediately and try to change their mind. Additionally, any deviation from the original plan, during the middle of the project, will almost always affect the timeline and budget. Code is less permanent than bricks, but like bricks, you cannot remove some without affecting others. Changing or removing code almost always requires refactoring, reorganizing, and rebuilding.
Setting expectations with stakeholders does not end once the project moves into the development phase. Adjusting the estimates, timelines, budgets, and releases is essential whenever new changes or requests come down the pipeline. New changes that affect the project must be conveyed to the stakeholders on a continual basis to ensure expectations align with the reality of the project. More specifically, resetting expectations often will do three things.
One, it will ensure that the estimate is always accurate to the current requirements of the project instead of an outdated estimate you made months ago. Avoid the temptation to simply absorb the new changes and hope it all works out because this will almost always fail. Requested changes after the initial estimation will affect the timeline and outcome in most cases.
Second, it will discourage the number of changes coming in before releases. If the stakeholder is seeing how changes are affecting the budgets and timelines, they will be more inclined to push the changes out beyond the scheduled release.
The best method to continually reset expectations is to set up a weekly or biweekly meeting with stakeholders and reset expectations whenever possible.
Breaking large problems into smaller problems is an extremely effective technique for tackling projects. The larger your estimation scope is, the lower the accuracy of the estimate. Imagine a new project will take four developers nine months to complete. This amounts to about 5,700 working hours for the development team alone. Predicting exactly how the project will unfold during those 5,700 hours is unreasonable, there are just too many variables, factors, and unknowns.
Instead, break these 9 months of work into phases and plan deliveries along the way. Perhaps this application has 4 different parts: user management, project workflow, invoicing, and reporting. Aim to release and deliver each part to stakeholders instead of the whole application at the end of the project. This gives you a release about every 2 months, which is a much more manageable amount of time to properly estimate and plan for. Breaking those two months further into four equal two-week sprints will narrow your estimate scope even further. This pattern of breaking down problems into smaller ones is fractal. The following are the common breakpoints in software development from the broadest level to the most specific. Each of these describes a different scope of work with different purposes and timeframe. You should be using all of them to properly plan out work.
A project describes the largest scope of work and often represents an entire application. The timeframe for projects is typically measured in months but can be several years.
Projects are used by the highest level stakeholders to plan and solve large problems.
Sections are large portions of an application or system that consist of numerous epics. They are usually not estimated directly or releasable on their own. Typically they simply group epics together.
A release typically represents a section of a project that makes logical sense to be released together. An example of a release would be the “reporting” section of an application or “authentication” release. Once the work is finished for the reporting section it can be grouped together and released to the stakeholders for user acceptance testing. Releases differ from projects because they usually represent a working section of an application rather than the whole project itself. Releases, as their name suggests, should be in a state that can be “released” to stakeholders. The timeframe for releases ranges pretty widely but typically these range from a few weeks to two months.
Releases are used by Product Owners and Project Managers to plan milestones, budgets, and product release schedules.
Epics typically represent a specific feature of an application. For instance, inside of an authentication release, you might have a Forgot Password epic. The whole authentication release might be made up of several epics: Signup, Signin, Logout, Forgot Password, OAuth, and SSO. Epics should generally take more than a day, but no longer than a week. Anything over forty hours should be broken down into multiple epics.
Epics are used by Project Managers and developers to ensure that they are properly planning out sprints that will lead to on-target releases and budgets.
Unlike projects, releases, and epics, sprints represent a consistent amount of time. Typically, sprints are one or two weeks. Project Managers or developers add tickets, stories, or work that will be completed during the duration of the sprint. Though every project is different, in my experience teams should complete at least one epic per sprint with each developer committing to more than ten stories.
Sprints are used by Project Managers and developers to ensure adequate efficiency and velocity. Additionally, it ensures there is a regular cadence for planning new work, executing against a deadline, and reflecting on delivery.
Stories are the smallest scope of work that can be considered complete. Sessions are smaller but are more like sprints in that they represent a fixed amount of time more than a complete scope of work. A story should represent one aspect of a feature. For example, with the Signup epic, you could have a story: “A user will see the validation errors when missing information on the form”, which could represent 6 hours of work. The timescale for stories is usually measured in hours and shouldn’t exceed 8 hours.
Stories are mainly used by developers to plan, estimate, and deliver sprints and epics.
Sessions represent the amount of time that a developer will sit down and focus on a given story or issue. Generally, this is about 3 hours and is the smallest amount of time a developer should plan for. Each day, developers should plan out their sessions according to what needs to be accomplished. For example:
Session 1 - (3 hours) Finish API endpoints for forgot password
Session 2 - (3 hours) Finish UI for forgot password form and tie state management and from submission to API.
Each layer of planning builds on the layer below. Breaking large problems down into smaller and smaller goals is essential for achieving large goals. For each smaller goal: plan, estimate, and execute each so they add up to efficient, well planned, visible, executed work that stakeholders can be happy with.
Although you can mitigate the effects of unexpected events by creating and maintaining your estimation checklist, you can never avoid unexpected events or results. Therefore, instead of agonizing over when the unexpected will come, incorporate the unexpected into your estimation so you can more easily handle the situation when it does come. All projects will typically encounter two types of unexpected issues: small and large.
Almost all stories include at least some unknowns unless the task is trivial. Stakeholders plan their goals around your team’s development velocity. Unknowns, even small ones can affect velocity, therefore it is important to always inform stakeholders of all unknowns on a frequent basis. Following the estimation process rigorously each time you estimate should help produce a consistent amount of unknowns that you can plan for.
Large unknowns will pop up periodically, usually at least one per project. The best way to mitigate a large unknown is to include a buffer in your estimate that your team can use when a feature starts to go south. Another powerful way to plan for large unknowns is to build up trust capital with stakeholders.
One way to conceptualize your relationship with your client is in the form of trust capital. There are two main ways to build trust capital: turn in high-quality work or deliver on time and on budget. Since this series is about estimation and delivery we will focus on the second method of building trust capital. Every time you deliver something for a client on time or on budget, you build up a reserve of trust capital. This capital can be spent to smooth over problems or mistakes in the future. For example, if you have built up a large store of trust, the client will be more forgiving and lenient when something goes wrong on the project. They will trust that you were doing your best to deliver the feature rather than assuming you’re doing a poor job. However, if you miss estimate after estimate, egregiously going over your estimates, you will eventually spend all of your trust capital and the client will end up firing you or dropping the contract. This is why it’s important to get estimation right, you get more leeway to handle large unknowns or unexpected problems down the road.
You understand the problem, you have the toolset to break the problem down into smaller problems, now is the time to provide a real estimate. Once design and technical discovery are completely done, start to break down the project into epics for each section of the application. However, do not estimate the epics directly. Later on, you will estimate the individual stories and add up each story estimate into a full epic estimate. A typical estimate list might look like this:
Once you have all of your epics listed out, now it is time to go through each and create the stories for each epic. For example:
Now that we have five or so stories for the first epic, let’s walk through how we would estimate an individual story.
We will use the first story, API can accept signup form submission and persist User in Database, to illustrate the questions, answers, and thinking that goes into estimating a story.
Does this change affect the UI?
No, this story is only describing the API side.
Does this change affect the API? Have you considered schema changes, migrations, models, controllers, routes, resolvers, endpoints, scoping, relationships, background jobs?
Yes, we do need to account for schema changes, migrations, models, controllers, routes, endpoints, scoping and relationships.
Have you allocated time for robust test coverage?
Have you allocated time for code review, final review and revisions?
Have you considered what refactoring of the current codebase is needed?
Yes, we do need to account for these in this case.
What libraries are needed by this ticket? Are they included yet in the project? Have you worked with them before?
We will need to include an authentication library as well as an encryption library for password storage and I have used them before.
What data is needed for this ticket?
This is a new application so no data is required outside of the seed data that is already there.
Have you accounted for SEO requirements?
There are no SEO requirements for this task.
Have you accounted for security requirements?
Yes, we will include an encryption algorithm for password data and enforce HTTPS to ensure encryption when in-flight.
What internationalization requirements exist for the ticket?
There is no i18n on this ticket.
What risks can you see by looking at the ticket as it's written? Is there anything that you need to do now to mitigate against these? Does this story need a “Spec Story” to further define any of its parts prior to estimating?
There are no obvious risks here and everything described here is standard authentication.
Always give a high and a low estimate for each story. An estimate range will help you convey timelines better by highlighting stories that contain many unknowns. For the story above we might estimate 3 hours on the low end and 7 hours on the high end, which means 5 hours should be your target to get this story through QA and into UAT.
Continue this process with the remaining stories until you have estimated each story within the Auth Signup epic. Once all stories are estimated, sum up the total number of hours and use that total as the estimate for the full epic. For example:
Summing up all the stories for the Auth Signup epic amounts to 16 hours. If 16 hours of development sounds reasonable for the given epic, then the only left to add is the buffer for unaccounted parts of development like project management, communication, and quality assurance. Although this will vary project by project, a good idea is to start with around 10% additional time for each of these (10% * 16h).
This whole package is what you will turn into your stakeholders. A range of 13-30 hours or in working days about 1.5 - 3.5 days with a target of 2.5 days. Remember that 2.5 days is including all of the effort put into getting this into production (PM, comm, QA, dev time, revisions, deployments, etc).
Sending this package sets the expectation for the acceptable range in which to finish the assigned work. However, if you always finish above your average across all of your stories and epics or you will drastically blow your overall budget and timeline. Most of the time you should be hitting close to the average estimate, other times you will come under estimate. Averaging out over the course of the project should allow for the buffer to weather situations where you need to go over the average estimate.
During development, you will often receive an estimate that was not created by you or meant for another developer of differing skills and knowledge. This is the perfect opportunity to use teammates to verify the estimate given. Make sure time is allocated during sprint planning for developers to check if the estimates on their tickets are reasonable and identify any problems that might impede their commitment to the estimate. This is important because all estimates should be signed off on by the engineer that is actually working the story.
At this point, all epics and stories have been estimated and presented to stakeholders for approval. If stakeholders agree to the timelines and budgets proposed, then it’s time to commit to working against the estimates you provided in an effective, productive, and organized way.
Some developers will jump right into working on the work assigned without thinking about how to approach their estimated work to finish it on time. Naturally, this is not a recipe for success. In the first article of this series, we discussed how important mindset is for the ability to deliver consistently on your estimates. Mindset is still extremely important, but it isn’t quite enough. To really succeed in hitting your estimates, you need to commit to the estimate.
In the book, Willpower Doesn’t Work: Discover The Hidden Keys To Success, Benjamin Hardy argues that when you make a decision and commit to it, it becomes part of your identity and removes any issues you have concerning your willpower.
“If you haven't made a decision yet, then you're going to be required to use willpower.”
As much as possible you need to become the type of person who delivers on what they commit. Who you identify as a person should be someone who estimates effectively and delivers on those estimates consistently. Identifying as this type of person, as a core part of who you are, removes willpower from the whole equation. Questions like “should I go grab a snack or procrastinate” will no longer be a problem when your identity is tied to staying productive and delivering on your commitment.
Some developers approach their estimated work wishfully believing they can finish their work within their estimate without planning anything or reflecting on previous mistakes. They do not plan to hit the estimate, they make no pragmatic decisions along the way, and they do not reflect on why estimates were missed. In short, they are not committed to hitting the estimate provided, they are simply blindly wishing that everything will work out in the end. Developers who approach work this way rarely grow or improve their ability to hit estimates, instead, they keep making the same mistakes over and over. The value they provide for their teams can be offset by the wasted time spent creating and tracking an estimate that they ignored during actual development. Rather than being wishful, we should be committed to hitting our estimates and becoming the type of person who can commit.
Before jumping into a ticket ask the following questions:
If you have an 8-hour task and you know that you need to finish the API side in 3 hours, give yourself a timebox from the start of the day to lunchtime to finish it completely. Mentally preparing for working a large chunk of work is a big step to actually getting it done. Additionally, this will help with the urge to multitask or do things that are off task.
Oftentimes during large projects, you will feel the need to allocate time for major refactors, or adding additional test coverage, or open souring a module you’ve been waiting to do for a long time. While all of these can be quite beneficial and worthy of our time, we need to stay pragmatic and weigh the relative importance of tasks against getting the whole project finished. Plan to make pragmatic decisions to ensure you get the work done on time and on budget while still providing the expected quality.
Constantly reevaluate where your team is at on the timeline and how well your team is doing to meet the estimate and budget. Every time you take a break and return to your work is a perfect time to accomplish this. Ask yourself:
These are broad questions that affect the project as a whole, but we also need to reevaluate the granular aspects of each story and epic. Follow the next three steps to make sure you are still on track.
Pragmatism: Dealing with problems realistically, based on practical rather than theoretical considerations
Idealism: Unrealistically aiming for perfection
Understanding the concepts of pragmatism and idealism will help you on a day to day basis with your work. Pragmatism will mean different things in practice because stakeholders usually have unique goals and every project is different. Unfortunately, this means there is no magic formula or checklist to follow to always make pragmatic decisions. However, one of the best ways to understand pragmatism is to see it in action with a few examples.
Let's say you are working on a project that has 70% test coverage but the stakeholders indicate they want to get it to 90%. In this scenario, you might be the testing advocate on the team who just so happens to be working on an epic that touches most of the application. A naive developer might get overly excited at this point and start increasing test coverage right away on the epic they’re working on. But pragmatic decisions require us to take a step back and ask the question: “Is increasing the test coverage right now during this task the pragmatic decision?” We can better answer this question by asking some more specific, related questions.
If you incorporate the increased test coverage while you work through your stories, it might take an extra 4 days for you to increase the coverage
Most likely it will push the finalization of this epic into the next sprint with these 4 extra days.
In this scenario, perhaps the stakeholders are expecting to demo the project and new features to the C-level execs after the completion of this epic, in which case this epic is extremely important for your team.
Given the stakeholder's goals, it is probably more beneficial to wait until after this epic is done to start working on test coverage.
As you can see in this example, the pragmatic decision is to hold off reaching 90% coverage during this epic. That doesn’t mean you should totally ignore testing in this epic, but you should focus on delivering the vital features of the application for the stakeholder demo.
The pragmatic decision in this scenario was to hold off on adding additional test coverage, but that doesn’t mean that this will always be the right decision. The exact same scenario could lead you to the opposite conclusion given what you know about the stakeholders and the specific project expectations. For example, if the stakeholders didn’t have a demo the cost of pushing the epic out farther would be much less detrimental. Another possibility is that the team is expecting to onboard a bunch of new junior developers on the project who would greatly benefit from the increased coverage. In this situation, you could easily see how the pragmatic decision could be to focus on the test coverage instead of finishing the epic.
In this scenario, the project has a very large and complex file that needs to be refactored. Reducing the complexity of this file will undoubtedly result in a better experience for everyone. But, as with the previous example, we need to ask ourselves a few questions to flesh out whether this is a pragmatic decision to make. Below are a few questions you might ask yourself when evaluating this problem.
The answer to these questions will lead us to make more informed and pragmatic decisions. Depending on the answers, the decision to refactor could go either way.
Some QA professionals have a tendency to return tickets with rejections and comments that are better described as opinions rather than objective misses on acceptance criteria. This is a perfect time to evaluate whether addressing QA’s concerns is pragmatic or idealistic. Ask yourself whether you should handle the changes during the story you are working on or create a new story for the next sprint. Some questions you might ask yourself in making this decision are as follows:
Think through these questions in different scenarios and consider what answers here would push you one way or the other?
Sometimes pragmatism will lead us to defer new work until later. In these cases, you must ensure that whatever was cut ends up in a story and is prioritized later in the project. The best time to do this is directly after you have made the decision. Finally, always make sure the stakeholders are aware of the deferment or tradeoff you made.
Open-ended tasks like “Research Stripe recurring payments API” should always be timeboxed, otherwise, you will end up spending way more time than is needed. Choose a specific amount of time, like 1 or 6 hours, and present your estimate to the PM and stakeholders. Not timeboxing open-ended tasks almost always leads to disagreements.
Using a start and stop timer is crucial for getting real data on how long tasks actually take. Get into the habit of starting and stopping your timer every time you begin and end your work because adding records at the end of the day for work done during the morning will rarely be accurate. Additionally, make sure any project management tool you use has the ability to track time spent back to stories that you are working on as well as roll up the total time into epics and projects.
For there to be any value in this whole process, you need to measure your work and track it back to stories and epics. This is a crucial step that many individuals, teams, and organizations ignore. Typically, the time entries from the time tracking app you are using (like Toggl) should be added as work logs to the tickets in the application you are using for project management. At FullStack Labs, we use a combination of Toggl and Jira, where Toggl entries are sent to Jira and logged directly on each ticket. We are then able to see where we are on any given story, epic or sprint. Every day you should check the status of your stories, epics, and sprints to stay on track and adjust expectations early. If a pragmatic decision needs to be made, the earlier the better.
Your estimates will improve dramatically over time by following this framework and deliberately practicing the steps required. Inevitably, however, some stories will end up going far beyond their estimate. Your goal, in these situations, is to find the root cause of the issue. For example maybe one of the following is true:
In scenarios 1-3, let the estimate run over. After completing the story, retrospectively identify any opportunities for improvements or take-aways that you can incorporate for future stories. For example, consider taking notes for why certain stories went way over your estimate and review these notes later on.
When something truly unexpected happens the only takeaway is to figure out how often these unexpected events might occur and plan for the unexpected whenever possible. For instance, throughout your career you might notice that something unexpected comes up about every three months. In this case, you should create a buffer for the unexpected around those timescales.
At the end of each sprint, review the estimates versus the actual logged time per story, especially for stories that went way over or way under the estimate. Ask the following questions to really dig into what could be improved for the sprint.
1. What adjustments should you make in the future?
2. Were your estimates too low because you didn’t account for something?
3. Were your estimates too high and need to be adjusted down in the future?
4. Did you identify any places where you were inefficient?
5. Did you start on some stories before they were fully specced out and ready to work?
These questions should be asked after each sprint, after each epic, and after each release.
If you’ve been following along so far, you now have a framework that you can track and measure progress. Although the road forward can be challenging and without end, you now can approach estimation with the same rigor that you would any other skill. Understand, plan, estimate, commit, track, deliver, review, adjust and repeat. Good luck!
We’d love to learn more about your project.
Engagements start at $75,000.