It’s official: We’re not doing Scrum anymore.

I will take the time to elaborate on why, and I’ll reflect on what we’ve learned from our experiences.

Why are we not scrumming?

The shortlist:

1. Iterations were too long – It would sometimes feel like a waterfall in miniature. Reality for most development teams is that circumstances change constantly, and developing the capacity to adapt to these changes are more important than planning iterations carefully, which is just a form of guesswork.
2. Too much overhead – Sprint planning felt a bit forced. We had to decide what to do for days or weeks to come, and ended up with requirements churn. Beyond a certain point during planning, we went into congestion, and stopped caring about the details.
3. Our sprints are too diverse – Sprint goals ended up  so diverse sometimes that we ended up more often than not with the sprint goal of “Finish all tasks” which was a quite uninspiring goal. (We tried very hard to create good sprint goals)
4. Demos not often enough – We need to show off the feature we just made right now, not in 14 or 30 days at the demo. That is way to late for us. Demoing should be an integral part of each feature or improvement you make. You need to be close to, and intimate, with real users. If we had investors or any other formal committee of sponsors that needed to approve everything we did, we would see the use of such “demo roundups” at regular intervals.
5. Too long release cycles – The scrum process batches a lot of changes, improvements, bugfixes and new features into periodical releases. We are working on continuous deployment with a steady stream of product improvements, which allows for more predictable releases and better fail-tolerance.
6. “Scrum is just the easy parts of XP”  – It’s a quote from Kent Beck, although he does not like to be quoted on that one. This is why most are doing Scrum in combination with XP of course. I think you in many cases are better off with XP without Scrum, but that’s another story. We’re not doing XP in its purest form, either. Even though XP has much stricter coding practices, XP also has its limitations. The lean startup movement (which Kent Beck himself seems to be quite interested in) complements the weaker parts of XP — the customer feedback part, especially in the initial phases of product development.

You might see a pattern here: We’re still doing most, or all, scrum practices, just more often and in smaller chunks. This brings us closer to Lean.

What did we learn from it?

You should have a Product Backlog, and it should be a very simple sortable list. The emphasis should be on keeping it short, however. A long list of features becomes an administrative overhead. We found ourselves sifting through much of the same tasks from sprint planning to sprint planning. Too much got captured in the sprint backlog. We are still using product backlogs actively.

You should hold sprint planning meetings. The problem with the scrum-format of it is that it has more emphasis on creating a plan up front (i.e. the waterfall comparison), than it has on the planning itself. If planning is a monthly activity (or bi-weekly, or whatever your sprint length is in scrum), you end up reading your plan more than you spend time planning. We find that planning is at its best a continuous, on demand, activity. Do more of it, often, but in smaller chunks, and keep less of it lying around. I will have to agree with the Dwight on this one : Plans are nothing; planning is everything.

Daily scrums solves a non-existing problem – I’m sure that it is not hard to find team that would benefit from daily scrums. If you are such a team, chances are you are not communicating enough. We find that by hanging out on the project IRC-channel doing continuous micro-coordination and sharing information, and by hooking up on Skype to discuss challenges and problems on-screen as they arise we have nothing left to coordinate or discuss on the daily scrums. Everyone is in the loop on what is happening.

Retrospects are pure beauty. Again, the problem with scrum for us is not that it is has bad practices (it has the best of practices, actually). The problem is that it is performed to seldom to have the desired effect. Retrospect is not an activity, it is a mindset, and has to permeate everything you do, and should be done on a continuous basis. Ask these questions about every little project, task and experiment you run. What went well? What could be better? What will I focus on improving? It’s key to learning, and learning is essential.

Under what circumstances would we consider scrum?

If we ever find ourselves in a multi-team environment with quite large teams (5-10 per team), with sponsors that needed formal updates and demos, we would be sure to consider both scrums and scrum of scrums.

So, was all our Scrumming just a waste of time?

No way. It was a learning experience.

Never forget this: There is no silver bullet. You should try different stuff, be critical and analytical in the process, ask why to just about everything, and you will learn a lot. Shed each practice that fails to materialize into concrete value for your development process. In the end, no process formalities will save you, your diverse experience will.

What will we do differently?

We will continue to draw inspiration from Scrum, Lean, XP, Lean Startup, Customer Development, Continuous Deployment, etc. But instead of changing ourselves to fit the process (as was much the case with our scrum-efforts), we will look at the core of their ideas, extract their essence and see how they could apply to our existing and intuitive way of doing things;

If being a keen student of processes and development theory has taught us only one thing, it is that we should trust our gut feeling, our sense of what is purposeful and what is common sense, and that most of our current practices are worth keeping, and improving.

I advise you to try to improve the quality of how your team already, and intuitively go about doing the work. Do not change the way you work entirely in an attempt to magically achieve quality and speed. This way you can better preserve what you already got figured out, and at the same time be improving continuously.

In plain words, what are we doing now?

We still do a lot of the scrum practices. We just do them more often, and in smaller chunks. You might even say continuously, to some extent. We have learned to cherish planning over plans, and a retrospective mindset over bi-weekly formal retrospects. We write less plans, we take less notes. Communication flows more freely instead of formally in batches.

We get more done, and with increased quality, and we also have Scrum to thank for it, but definitely not only Scrum.

One of our biggest ongoing project nowadays is our establishment of Continuous Deployment practices, and all the discipline that goes with such a strategy. (It requires some coordination when you have servers deployed all over Europe, as we do) You will need to optimize the entire value stream to achieve this. We’ll keep you posted on how it evolves, and the benefits our customers and we reap from it.

I have been eager to share more from our experiences with Scrum. Some might remember our Scrum-series from back in the days (autumn ’08). Scrum is currently obtaining the characteristics of a fine wine, it’s getting better with time, given the right love and care. I would proceed straight to sharing our sprint analysis, but first I must clarify the concept of burn-down graphs.

What is a Burn-Down Graph?

The image below represents an ideal burn-down.

The Y-axis (point 1) shows how many hours we’re investing in the sprint (150) and the X-axis (point 3) shows how many days into the sprint of the total 30. Ideally we progress towards the end with the same amount of work completed every day. This would yield the ideal graph-line depicted here (point 2). The only complex situation so far is calculating the right velocity.

Velocity is how many hours we’re working on tasks on the project during the sprint. You might think of velocity as “how fast the project will go” (Putting in one million hours every 30 days would yield a very fast-going project, hence the term; velocity).

How to Calculate Velocity

In Utopia you could calculate that every hour of bodily presence of a project member is an hour of being intellectually present and focused at work on actual tasks. This ideal never holds up, and hence the definition of ideal being some hypothetical optimum.

So how do we find the deviation from the ideal? Let’s ponder a little at this question.

Say you are showing up for 37.5 hours a week, as we do up here in the arctic Norway (time spent eating penguins, whales and seals for lunch not included). Let us simplify and assume there are a total of 20 workdays in a month. That leaves us with 150 hours per month per project member.

Now, is that velocity?

– No.

Velocity is : Hours of bodily presence – (your mind wanders off + making coffee + toilet visits + dusting off that old cowboy hat + getting distracted + having a bad day). In short, velocity is the sum of ideal development hours we are able to put in, and thus all other non-ideal activity should be deducted from hours of bodily presence.

Presume the project member, Jackie Trombona, has just joined our project. He is planning on one week of vacation during the sprint. The remaining three weeks he will be available for the project. He is a very focused and deeply engaged developer. He checks email only twice a day, and keeps communications short, simple and conclusive – which makes him spend a maximum of 30 minutes reading/deleting/writing emails daily. He is effective, but being honest with himself he has calculated that he is loosing another 45 minutes from the ideal to “miscellaneous” each day. This is mainly for when his mind wanders off, or he is making coffee or chatting by the water cooler with colleagues about his multi-billion philanthropic hobby efforts. Another 15 minutes is deducted for the daily scrum. In total he “loses” 90 minutes a day.

This leaves him with 6 ideal hours each day. 6*5=30 hours a week. This means he’s 0,8 into the ideal time-spending, which is really good.

For this project, however he’s only available for 3 weeks this sprint, so he has 3*30=90 ideal hours available for this sprint. 90 of 150 is 0,6 of the ideal.

0,6 is his Focus Factor (FF) for this sprint.

Total sprint velocity =  150 * ∑ (FF(member1), …, FF(memberN))

In English, to find total sprint velocity = Find and summarize the focus factor for all participants and multiply it with 150 hours.

What Do We Do Now With All This Velocity?

Notice the yellow dot on the bottom at the far left below (point 4). This is the sum of our added estimates, which still is zero. The power of knowing the velocity is that we can now add tasks in sprint planning until the “itsy bitsy teenie weenie yellow polka dot bikini” reaches the level of our velocity.

When we’ve added 150 hours worth of tasks, the yellow dot has risen to the top, and we’re ready to burn down through this graph during the next 30 days. Upon completion of one task, the yellow dot drops the equal of the estimate of the task completed.  We can now, through the visual feedback of this graph, easily track our progress towards the sprint goal. When every task is completed, we will be at zero. Our carefully crafted parameters: estimates and velocity (as discussed above) makes this a realistic challenge which we expect to achieve, and when the challenge becomes realistic, it gets interesting and we stay focused on our goals.

Here’s a real-world example from an actual and currently ongoing sprint, 10 days in. As you can easily see, it’s currently on track to reach it’s goals even with some margin.

In upcoming posts, I’ll be showing some more of these graphs, and I’ll be going into the details of their nature. Stay tuned for that and more (via RSS).

Summary

Velocity and estimates are powerful tools for realistic planning. Combined into the burn-down graph it provides a clear and persistent visual feedback to the team on current sprint progress. We love the burn-down graph. We love it so much we created our own Scrum-software to automate it’s creation and maintenance.

Edit 10th or March ’09 : Check out the article describing a few of our real life burn-down graphs.

The Daily Scrum Meeting is the continuous inspection and adaptation mechanism of Scrum. The Team inspects its own efforts, and adapts to new information. To put the Daily Scrum into the larger picture, have a look at this brief overview of Scrum. We’ve arrived at #3 in the list.

The Daily Scrum goes down like this. Everyone gets together for a maximum of 15 minutes in the morning to answer the three following questions to each other.

1. What did you do yesterday?
2. What are you planning on doing before the next daily scrum?
3. What impedes you from performing your work as effectively as possible?

This is coordination (1), planning (2) and continuous improvement (3) packed into a quick small talk session in the morning. We have been discussing the nature and art of software engineering earlier in this blog, and we have claimed our support for the assertion that software engineering is far to complex for deterministic up-front planning. The Daily Scrum embraces this fact, and it establishes mechanisms for Empirical Process Control (EPC). EPC relies heavily on inspection and adaptation mechanisms.

Oh my god. I just realized that the above paragraph is as pompous and boring as it is important. Let’s remove all gobbledygook, and try again.

What I am really trying to convey with the above paragraph is that in order to succeed in anything as complex as software development, you’ll have to talk to your excellent team mates often to stay informed and aligned. The Daily Scrum keeps this communications channel and feedback-loop alive, frequent, brief and structured.

Some rules

In order for the Daily Scrum to be a successful mechanism, we should follow some rules.

Only team members are allowed to talk, and no-one should digress from the three questions above. Only one team member speaks at a time. This is to ensure the efficiency and focus of the meeting not allowing it to grow out of proportions and hurting its intended purpose.

There is no reporting whatsoever. The information is high bandwidth day-to-day, face-to-face. All members should attend, always.

If someone reports anything of interest to other team members, these can get together after the Daily Scrum to discuss matters further. Again, we want to keep the Daily Scrum lightweight and focused. You don’t want everyone to listen to the details of everything, particularly not as the team grows. This will only lead into congestion.

The Daily Scrum is at the same time, at the same place, every day. Everyone is required to attend, and those not precise should be penalized in some way. Putting a dollar (or a femmer) into a bucket to buy Flingo (with added bounce) later, is highly recommended.

That’s the most important simple facts to know about the Daily Scrum.

At the start of our Scrum series, I stated that Scrum is a lightweight and simple process framework with minimal load and overhead. While this statement still holds up, it is also true that rationale behind every Scrum mechanism is a bottomless well. You can get started with Scrum, reaping many of its benefits quickly. Anyone can grasp the mechanisms with a minimal of introductory coursing needed. On the other hand, you can spend months or even years to really understand their consequences and power. We for our part are not through analyzing Scrum for rationale and benefits. We’ll keep on doing that until we’re Scrum ninjas. And I mean ninjas with a capital N. Ninjas. Now you see us, now you don’t.

A quick recap of events. (What have we done before starting The Sprint?)

We’ve spent time with the customer constructing the Product Backlog. This establishes an understanding of what to create. We’ve gathered The Team and stakeholders to pick an amount of items from the Product Backlog to commit to for the sprint. We spent 4 hours doing this. Then, the team has broken down the items into tasks and created a Sprint Backlog (another 4 hours). Now, The Sprint has started.

So that’s all there was to it. The team has done 8 hours of work to prepare for The Sprint itself. What has not been planned or resolved during these 8 hours, The Team is responsible elaborating during The Sprint.

There are two main topics about The Sprint

There are two main topics to discuss regarding The Sprint. I’ll address rules in the remainder of this post, and proceed to talk about the Daily Scrum in a post that will follow the day after this one airs.

Sprint Rules

The Sprint is time-boxed to one month. This is the minimal amount of time needed to create a significant increment of potentially shippable software. Also, it is the maximum time that can be allocated without The Team doing so much work that we loose overview and have to start making documentation and artifacts to support the process. We want to avoid that, as only it adds to the load, so we’ll stick with the one-month iterations.

No one can interfere with The Team during the sprint. It is utterly self managing. The Team can seek advice and help as they want, however. The only way to interfere The Sprint is to terminate it. Stop it dead in its tracks. Then we’ll start over with a new round of Sprint Planning. This rule really makes anyone wanting to disturb the flow of the team, think twice about demanding them to change their plans, as its implications are made very obvious and dramatic. This is necessary.

The sprint can be terminated if The Sprint has become non-viable, or of little of no value due to changing requirements.

If The Team feels they have too much, or too little, to do during The Sprint, they will consult with the Product Owner on which items to remove from, or add to, the Sprint.

The Team is responsible to keep the Sprint Backlog up to date. (At Aptoma, we have developed web-based in-house software to support this (and every other) activity in Scrum. Our software increases visibility and simplifies day to day registering and planning. We call it AptoPappa, as it is so kind as to look after us and our Scrum activities. We might just show it to you one of these days.)

These are almost all, and certainly the most important rules of The Sprint in Scrum. Stay tuned for more on the Daily Scrum, an imperative activity in The Sprint, and in Scrum as a whole.

PS! One of these days I’ll do a post with no links to our RSS-feed, but it will be only one post, and it will be years into the future.