In that case you need to build in slack for learning into the system. You want everyone to have time for sharing, improving and learning continuously, and you want to be able to come home from work with energy to pursue your goals for personal skill development even further.

It’s great fun to be good at what you choose to spend your life doing.

Since the primary raw materials in software product development are skills and intellect, rapid learning is the prime goal and the capital for product development. Fail to invest in raw materials for your business, and you’ll soon run out of stock. Don’t allow for time to learn, rather, require it.

Running a system at 100% yields all sorts of problems.

Symptoms include

• You stop talking to each other. “Sorry, can’t help, have to go do this thing that is  really urgent”
• You start building technical debt. “If I just skip to write that test (it’s really hard to write), then I can save 2 hours today to get to the next thing I have to do.”
• Shallow bug-fixing. “Need to fix this bug real fast, in order to get the other 4 bugs done, too! No time for digging further” Root cause analysis and regression testing get skimped on.
• Useful, although sometimes inconclusive and somewhat directionless, discussions plainly stops happening.
• Stress activates your lizard brain, which numbs the rational being. You are in survival mode.
• Work gets less inspiring and fun, which slows things down. Inspiration is high-octane fuel.
• People get exhausted, and energy-level is low during, and after work hours. You need to “relax” on Facebook at work and in front of the TV at home, instead of doing inspirational learning.

(Picture from Wikimedia commons)

If you have trouble grasping the concept, picture a congested highway. If you remove a small percentage of the cars, you get a magnitude of improvement in throughput and speed. This is a fact.

Did you ever try to squeeze a washing machine 100% full? No clothes get cleaned. Remove 10 or 20% of the clothes, and you have a winner.

Don’t maximize utilization.
Build in time for learning, improving and sharing.

If you, as we have been, find yourself in this mess. There are several ways to get out of it.

1. Say no. If you never say no, you don’t have a strategy — say no to the least important stuff. Saying no is hard, scary and extremely useful.
2. If you work for a consultancy firm, give up. Incentives for maximizing utilization is too high in consultancy companies, as they get paid for “productive hours” only. Quit your job, and go to work in a product development company, or a skilled in-house team. Anything but most consultancy firms, they will cannibalize you.
3. Scale up capacity to meet demand. What can I say, we are hiring.

Quantified

This is no science, but if I would be forced to put some numbers on this, I’d say that:

• Build in approximately 20% learning/sharing time during normal work hours. Yes, that’s one whole day a week. We use it for our monday school, impulsive discussions, reading, friday meeting (our arena for general discussions on processes) and blogging, to mention some activities.
• If you want, you can probably “work” 60 hours a week or more without any danger of burning out, but only as long as that time is spent balanced and having fun, learning and improving/being productive — and only if personal situation allows for it without sacrificing time spent on other important values, like your kids.
• Don’t build overtime into your standard operating procedures. Required overtime doing “productive” work should be the rare exception, not the default.

Find a sustainable pace where you are balancing learning, improving, being productive and all of the time having fun. That’s an important ingredient in the recipe for success.

By the way, did I mention that we’re hiring or that we’re hiring?

It’s official: We’re not doing Scrum anymore.

I will take the time to elaborate on why, and I’ll reflect on what we’ve learned from our experiences.

Why are we not scrumming?

The shortlist:

1. Iterations were too long – It would sometimes feel like a waterfall in miniature. Reality for most development teams is that circumstances change constantly, and developing the capacity to adapt to these changes are more important than planning iterations carefully, which is just a form of guesswork.
2. Too much overhead – Sprint planning felt a bit forced. We had to decide what to do for days or weeks to come, and ended up with requirements churn. Beyond a certain point during planning, we went into congestion, and stopped caring about the details.
3. Our sprints are too diverse – Sprint goals ended up  so diverse sometimes that we ended up more often than not with the sprint goal of “Finish all tasks” which was a quite uninspiring goal. (We tried very hard to create good sprint goals)
4. Demos not often enough – We need to show off the feature we just made right now, not in 14 or 30 days at the demo. That is way to late for us. Demoing should be an integral part of each feature or improvement you make. You need to be close to, and intimate, with real users. If we had investors or any other formal committee of sponsors that needed to approve everything we did, we would see the use of such “demo roundups” at regular intervals.
5. Too long release cycles – The scrum process batches a lot of changes, improvements, bugfixes and new features into periodical releases. We are working on continuous deployment with a steady stream of product improvements, which allows for more predictable releases and better fail-tolerance.
6. “Scrum is just the easy parts of XP”  – It’s a quote from Kent Beck, although he does not like to be quoted on that one. This is why most are doing Scrum in combination with XP of course. I think you in many cases are better off with XP without Scrum, but that’s another story. We’re not doing XP in its purest form, either. Even though XP has much stricter coding practices, XP also has its limitations. The lean startup movement (which Kent Beck himself seems to be quite interested in) complements the weaker parts of XP — the customer feedback part, especially in the initial phases of product development.

You might see a pattern here: We’re still doing most, or all, scrum practices, just more often and in smaller chunks. This brings us closer to Lean.

What did we learn from it?

You should have a Product Backlog, and it should be a very simple sortable list. The emphasis should be on keeping it short, however. A long list of features becomes an administrative overhead. We found ourselves sifting through much of the same tasks from sprint planning to sprint planning. Too much got captured in the sprint backlog. We are still using product backlogs actively.

You should hold sprint planning meetings. The problem with the scrum-format of it is that it has more emphasis on creating a plan up front (i.e. the waterfall comparison), than it has on the planning itself. If planning is a monthly activity (or bi-weekly, or whatever your sprint length is in scrum), you end up reading your plan more than you spend time planning. We find that planning is at its best a continuous, on demand, activity. Do more of it, often, but in smaller chunks, and keep less of it lying around. I will have to agree with the Dwight on this one : Plans are nothing; planning is everything.

Daily scrums solves a non-existing problem – I’m sure that it is not hard to find team that would benefit from daily scrums. If you are such a team, chances are you are not communicating enough. We find that by hanging out on the project IRC-channel doing continuous micro-coordination and sharing information, and by hooking up on Skype to discuss challenges and problems on-screen as they arise we have nothing left to coordinate or discuss on the daily scrums. Everyone is in the loop on what is happening.

Retrospects are pure beauty. Again, the problem with scrum for us is not that it is has bad practices (it has the best of practices, actually). The problem is that it is performed to seldom to have the desired effect. Retrospect is not an activity, it is a mindset, and has to permeate everything you do, and should be done on a continuous basis. Ask these questions about every little project, task and experiment you run. What went well? What could be better? What will I focus on improving? It’s key to learning, and learning is essential.

Under what circumstances would we consider scrum?

If we ever find ourselves in a multi-team environment with quite large teams (5-10 per team), with sponsors that needed formal updates and demos, we would be sure to consider both scrums and scrum of scrums.

So, was all our Scrumming just a waste of time?

No way. It was a learning experience.

Never forget this: There is no silver bullet. You should try different stuff, be critical and analytical in the process, ask why to just about everything, and you will learn a lot. Shed each practice that fails to materialize into concrete value for your development process. In the end, no process formalities will save you, your diverse experience will.

What will we do differently?

We will continue to draw inspiration from Scrum, Lean, XP, Lean Startup, Customer Development, Continuous Deployment, etc. But instead of changing ourselves to fit the process (as was much the case with our scrum-efforts), we will look at the core of their ideas, extract their essence and see how they could apply to our existing and intuitive way of doing things;

If being a keen student of processes and development theory has taught us only one thing, it is that we should trust our gut feeling, our sense of what is purposeful and what is common sense, and that most of our current practices are worth keeping, and improving.

I advise you to try to improve the quality of how your team already, and intuitively go about doing the work. Do not change the way you work entirely in an attempt to magically achieve quality and speed. This way you can better preserve what you already got figured out, and at the same time be improving continuously.

In plain words, what are we doing now?

We still do a lot of the scrum practices. We just do them more often, and in smaller chunks. You might even say continuously, to some extent. We have learned to cherish planning over plans, and a retrospective mindset over bi-weekly formal retrospects. We write less plans, we take less notes. Communication flows more freely instead of formally in batches.

We get more done, and with increased quality, and we also have Scrum to thank for it, but definitely not only Scrum.

One of our biggest ongoing project nowadays is our establishment of Continuous Deployment practices, and all the discipline that goes with such a strategy. (It requires some coordination when you have servers deployed all over Europe, as we do) You will need to optimize the entire value stream to achieve this. We’ll keep you posted on how it evolves, and the benefits our customers and we reap from it.

• Plan it
• Code it
• Test it
• Commit it
• Deploy it

Nothing out of the ordinary, it seems. The seemingly new thing about continuous deployment is that we remove all the waiting in between each of these steps. That is, we don’t batch changes before to testing them. You test immediately what you just coded. You commit it. Not to a branch due for merging in a week. No, directly to trunk. And you even deploy it at once to production. Done. Clear your mind of it, enjoy, relax, spend some time doing something proactive.

Develop, test, commit, deploy. One feature, bug-fix or improvement at a time. That is the mantra for continuous deployment.

Some developers, and perhaps particularly managers, will find the idea crazy at first. Release immediately? Omigod. But frequently this sensation of safety in postponing something, is not rooted in any real benefits, and there seems to be no rational answer to the question of: why wait? Did your code ever get better during the night? Some will even argue that big feature releases is a marketing invention, invented to squeeze more money out of customers frustrated with the bugs in the current version. This practice of batching changes in releases in turn got embraced by the industry as the professional way of conducting business. I could easily add this to my list of major problems with the software industry.

Wouldn’t it be nice to deliver a bug-fix the same day it was reported? The same hour, even? It might just be possible if you are continuously deploying product improvements like this, kaizen-style.

Where does this idea come from?

The idea of continuous deployment comes from the lean school of thought. In the mindset of lean kanban pull systems — where you are constantly looking to remove waste and minimize your batch-sizes — grouping a lot of big software changes together in big batches and big releases makes no sense.

Even Flickr is doing continuous deployment these days.

The whole idea is to respond to customer “pull”. The customer needs something now. The clock starts ticking. Is the feature more useful if you wait a bit before you start coding it? No. Then start now. Is the feature better if you wait with testing after coding it? No. Then test now. Wait until tomorrow to integrate into version system? Is that better? No. Then integrate. Does thing get better if we wait to deploy that new feature? No. Then deploy it. Stop your clock. The shorter the time, the better for the customer. And better for quality, it seems. And the better for you, the coder. Unreleased code is stressful. Released, working code, is stress free. The code works. It’s proven.

After a few iterations, our fear level was actually lower than how we used to feel after a staged release. Because we were committing less code per release, we could correlate issues to a release with certainty. (From Case Study: Continuous deployment makes releases non-events)

Bugs that are not discoverable in testing are easier to find and fix if you released only one feature, than if you batched a months worth of changes together in one big bang release. Or so the lean school of thought states. I agree. You would have written the code hours ago, and probably say ‘a-ha!’ the second you hear about it. 20 minutes later, a bug-fix is released. Why wait?

A few resources on lean from Amazon:

• Software Development: An Agile Toolkit
• Implementing Lean Software Development: From Concept to Cash
• Lean Thinking: Banish Waste and Create Wealth in Your Corporation
• The Toyota Way

What does it take?

Not surprisingly, continuous deployment requires a high level of team discipline. No goofing off and no more fixing everything mañana in testing. Practices such as 5 Why root cause analysis, extensive testing and automation will be required. But it will all lead you to better quality and higher iteration speed. Speed and quality, it is all related.

Eric Ries writes on O’Reilly Radar about “Continuous Deployment in 5 easy steps“. Eric Ries has gotten a lot of traction with his Lean Startup movement lately. And you can follow his thoughts and reports on personal experience on continuous deployment on his blog http://www.startuplessonslearned.com/

Look ahead

Will we pursue this issue of continuous deployment any further? In the name of delivering a high quality experience to our customers and users now, when they still need it: yes we will.

If you will join us in this, please follow this blog on RSS or follow the company or the author on twitter. If you digg it, then digg it.

Speed

The ultimate ideal for speed is a one-man show.

There’s this one guy doing everything in the project. He is competent in engineering practices such as software design, scaling and testing, and he excels in design, user experience and what not.

When he has an idea, he considers it. He weighs it back and forth. Then a decision is made, right there on the spot. No waiting, no straying, no nothing — just an idea, wham bam, a decision. All while taking a shower or having a cup of coffee. Soon he’s back at the keyboard implementing it, testing it, committing it, deploying it.

The next thing you know users are using it, providing their feedback and insight.

Did you notice the lack of all-day design committees (probably reaching no consensus, scheduling follow-up meetings the next week). Notice the lack of a board of directors that takes weeks to assemble into one room. Notice the lack of calling, facilitating and planning endless project-meetings. No extensive change processes. No writing frequent status reports. No large specifications documents needed to keep in sync for review and approval, there is no handing over documentation between the design phase and the implementation phase.

With the one-man show, it’s just wham bam thank you ma’m. It’s like a ninja — the ultimate warrior mastering all needed disciplines needed to win the war, and he’s calling the shots faster than you can say cheeseburger.

– Cheese (I chopped your head of) burger.

Quality

But say the ultimate goal is quality, not speed.

The ultimate ideal for quality is hordes of different specialists working the project. They work in teams, or they might even separated into entire departments or companies dedicated to one profession each. One of these teams consists of the best designers the world has to offer. Another team has the best programmers and there’s a team of the best usability experts in the field.

At the start of the project all sit together until everyone agrees what problem we are solving, and how we want to solve it. If new challenges arise underway, they will gather to discuss and to agree how to adapt.

We start the project, and we are passing the project from team to team, giving each of them the time they want, and do not interrupt them until they’re done. They will ask for your input when they need it.

When they are finished, you should have the best quality money can buy, shouldn’t you? After all they had all the time in the world, and they were not interrupted.

Speed or Quality?

Now, all you have to do is choose, will you need speed or will you need quality in your particular project? Choose your strategy from the two outlined above thereafter. Unfortunately, it’s not that easy. Or, rather: luckily there’s more to it.

The problem with speed

With the one man show, you are very exposed to obtaining one or more crappy qualities on your software product. Your guy will have his strenghts, and there is bound to be weaknesses. So you end up with the best user interface, but a crappy test-suite making the software unmaintainable. Or perhaps you have a delightful backend, but a user interface that sucks. Or maybe your ninja is so anal about every quality that he is failing to build up speed in the first place.

It just does not scale, and you are very exposed to this individual’s priorities.

Ultimately the lack of tests, the poor scalability or the increasing complexity will stop the project dead in its tracks. So much for the ultimate speed ideal. It was fun while it lasted, but it did not last. Project stops. Try again.

The problem with quality

OK, so you chose the smart path and aimed for quality. You landed the big budget and told everyone to do their very best, and gave them all the time in the world.

I’m sorry to report that problems seems to pile up for these kind of teams as well.

So your user experience department spends a week, a month, two months, or whatever time it takes to craft the best user experience possible for the problem they are solving. Then your design department spends a week, a month, two months, or whatever time it takes to create the best design of their life. They hand it over to engineering, and, as they go off to win a design award, the engineers have a go at fulfilling all the promising features that the UX-team wire-framed now fitted with the award winning design. They spend a week, a month, two months or whatever it takes to create the very best implementation. Their algorithms are featured in next years text-books for the university students.

Only now the solution solves yesterdays, yestermonths or even yesteryears problems.

The core problem with this quality approach: While each of your teams were busy creating the best user experience, the best design, the best algorithms, someone else were busy making the most accurate solution for the problem.

And as if that wasn’t enough, they were adapting to the problem as it changed in character. The problem always does that. Change, that is. At least in the field of software product development.

No problem, you’re thinking! Let’s just pass the ball back to UX, on to design and then back to engineering. That will show them!, in just 1 month or 2months or n months time, that is.

Oops, and there’s that change of problem to solve again. Darn it. Your project is behind, and your award winning design and algorithm are not solving the right problem. Quality wasted.

Turns out that time available and uninterrupted focus in each discipline is far from the only factors which affect the total quality of the project.

Solution accuracy is also a quality. Too often forgotten, but probably among the most important ones.

The solution lies in team design

In order to realize the solution, you will first have to buy into the assertion that:

You need quality to achieve speed, and you need speed to achieve quality.

This means that you should not choose either speed or quality. You should aim for both, as they tend reinforce one another. It might feel counter intuitive or even paradoxical, but it holds out.

The team design solution you are looking for draws inspiration from both worlds: the overly optimistic one man show (the ninja) and the bureaucratic silo process with a lot of specialists in their own departments passing the project from silo to silo in total isolation from one another.

You need speed and you need quality.

It’s really quite simple to explain (and very hard to execute)

You need one guy calling the shots, and you need specialists backing up and implementing in quality what he dictates. They all need to interact in a cross functional environment and they also need time to dedicate themselves to their specialties, creating price-worthy work within their profession.

You need to strike the fine balance in your team design to get the speed of cross functional teams, combined with the qualities of the isolated teams. Your continued efforts to ensure both will ensure that these two disciplines reinforce one another to achieve the best quality software you can produce.

“Accept and embrace the risks, even plan how to mitigate them, but don’t avoid them.”

There are, of course, loads of challenges.

1. Why should the teams of specialists trust the chief engineer’s decisions, and thus back his ideas with a 100% commitment and enthusiasm?
   • The centralized decision-maker will have to be a respected individual, an individual in which all team members place their trust. (Challenge: These individuals are hard to come by, and they have to invest a lot of energy into understanding everything about all parts of the project)
2. How will the chief engineer be able to know all about everything in the project?
   • Do not underestimate the capacity of man, given the opportunity to focus. The chief engineer will focus on two things : knowing everything about the project, and developing a deep relationship and understanding of the problem to solve (i.e. getting intimate with the customer group). He makes it happen through this focus.
   • Also, he must be wise and humble. He must be so humble as to pull information from specialists whenever it is needed to reach the best possible decision for the project. Just as much as he knows where to get all the information he needs — he does not himself possess it. He is trusted to do this, and his trust among his peers gets strengthened all the more when actually doing it.
3. What if the chief engineer is hit by a bus?
   • A very common question from concerned prospects. The truth is that unless the chief engineer has had a brilliant protege at his side all this time that can step in immediately and take over, your development process will grind to a halt. At least for some time. Consider the alternative for a moment, before gasping in repulsion. Your alternative is a process similar to design by committee (a lot of people involved in every project decision), in which case your development process will grind to a halt from day one. Surely and you will be safeguarded from loosing momentum, but it’s not worth is when the solution is to never really building any in the first place. And that’s just not a smart strategy. Accept and embrace the risks, even plan how to mitigate them, but don’t avoid them.

There are surely more challenges with this approach than my shortlist above, and you’ll have to develop the capacity to deal with them. Nobody said this would be easy. I just asserted that it will be the right thing to do. And it will. Master this, and it will make you strong, fast and of high quality.

Summary: The solution in a simple list

You know you love lists. I know you love lists. That’s why I’m willing to oversimplify the answer to this complex problem, and state it in a few simple bullet points below. Be sure, however, that you set out to absorb these ideas beyond this list, and follow up on the recommended reading, in order to make your team design better and better with time. Kaizen.

1. Keep decision making centralized at a wise, trusted, experienced and respected source.
2. Ensure a steady flow of cross-functional communication between specialist teams to keep everyone aligned towards the ultimate goal of creating the best overall solution for the problem — not just the best design or algorithm in isolation.
3. Ensure that each professional discipline has enough isolated time so they are able to produce work they are proud of.

Am I making this up? (AKA: References)

The concept has been tried and tested not only by us, but in larger scale by Lean inventor Toyota (their chief engineer role), by Gore (the Gore-tex manufacturer) who refuse to scale beyond 150 people per factory in order to keep it small enough for one guy to pull all the threads and have direct contact with everyone involved.

Toyota relies on the chief engineer role in every new product development process. Their future existence relies (more and more) on their ability to come up with new designs that accurately address the needs of the marketplace, and just for this reason they have the chief engineering role, which also makes them one of the best in this field. The others are following in their tracks.

Both of these companies has enjoyed great success as a result of their strategies.

• The Toyota Way by Jeffrey Liker address the chief engineering role.
• The Poppendieck series in Lean comes highly recommended if you find these theories intriguing. (Wisdom seldom come wrapped as a 20 year old kid in a suit)
• Also you can read about “the genius of the and” in the much quoted book Built To Last.

There is no silver bullet. You’ll need to experiment with team design to master it. My best advice would be, that you give it a shot in the small, and in the process you should read the references provided. See how it goes, revise, celebrate your ability to spot your failures, acknowledge your victories and try again.

Repeat indefinitely.

If you like this post and want more, enter the fourth dimension. You can subscribe by email to the right.

Disclaimer

Never mistake team design to be the silver bullet. There is no silver bullet. Good team design will not help if you have team members that lack the skills or the motivation to excel within their respective fields. You need excellent individual skills in each team member to pull this one of, and you need to put full trust in them.

It’s not a development strategy for the faint of heart, or for the micro managing control freak.