In Toyota, they have this role of Chief Engineer. For those who don’t know, Toyota has been the fastest growing car company most of the time since the 1950′s, and apart from having technical excellence in their product, they credit a lot of this to the role of the Chief Engineers. The Chief Engineer is a guy that would embed himself with potential customers of the new design. If he were to design a car for carpenters, he’d put on his carpenter pants and get all carpenty. Usually he’d discover a lot of design details needed a carpenter. Stuff like double (or triple, or quadruple) coffee holders, a flat dash-board for throwing documents on and to write upon, doors that swing up really wide, to expose the sound system speakers to the outside workers, and so forth. Basically a lot of stuff an office rat, or a technical engineer wouldn’t dream of.

This approach, combined with the excellent technical quality and their unprecedented production effectiveness, created a lot of success for Toyota.

We’ve chosen the little more cheesy title “Program manager” for our Chief Engineers. We like cheesy, because it’s so cheesy. (Actually, people don’t like it).

With 7 people and 3 products, (Aptoma anno 2010) everybody is a PM. And everybody is a developer. And everybody is a janitor. And so on. The lines between developers, visionaries and managers are as blurry as they should and have to be. Everybody is taking turns doing anything, and this is part of the fun of being in a small company. Great stuff.

It doesn’t scale.

We’re now 13 people in Aptoma, 10 of which are developers (and we’re looking for more talent). With more customers and products, development and maintenance gets increasingly complex, both in coding and in managing. And so we need more focus time on development and on program management as two different, but mutually dependent, exercises.

We now have established formal program management for two of our products (front-page editing, and article- production and layout) and one for our elastic cloud services (streaming, hosting and video transcoding for the media industry).

What Does a Program Manager do?

In short, there are five main responsibilities to the PM.

1. Serve as the customer advocate
2. Write functional specs
3. Keep track of progress
4. Hold team reflections.
5. Maintain road-map.
6. Remove obstacles for the team.

How?

Seems so simple from the list above, but there is a lot to this role.

1. Serve as the customer advocate. In order to serve as the customer advocate, you need a lot of information about what to advocate. You need to talk to users, you need to empathically understand them, and you need to dear to expose yourself to irritated users having problems. And you get to talk to those that are really, really happy and has had the time to reflect and envision new stuff. We aim to do this a lot for the same reasons that the Chief Engineer do in Toyota. This can be done by checking in at the customer, and saying “what’s up!”. Sounds easy, but the tricky part of this is that if you have a crappy product, they’ll only tell you about how crappy your product is. So get your bug-list to zero first. As such, the PM has to coordinate input from various sources, and streamline this information to use it to advocate the customer needs to the rest of the dev-team.

2. … and that’s where the functional spec comes in. Based on the deep understanding of customer needs, the feedback from support, from developers, from just about anyone with a stake in your product, the PM will propose changes, often through a written functional spec. The functional spec will describe an idea for improvement or extension to the product. If the functional spec contains wire-frames (in HTML, or just drawings) it can even be tested on potential users for early feedback.

3/4. Keep track on progress, and hold reflections. Once you say GO to a great development team, they will go into ‘get-shit-done-mode’, and won’t stand for any interruptions. That’s why the PM is keeping noise away from the team by being the customer proxy, gathering input, writing the next functional spec in parallel. And if anyone is to know anything about what is going on, someone must have the job of saying “Hey! How are things going?” and “Hey! Do you have what you need?”. Developers in ‘get-shit-done-mode’, will first bite your hand off, but everybody appreciates to pause and reflect from time to time, and to be able to report achievements.

5. Maintain roadmap. This is the wet dream for every bureaucrat. The internal road-map needs to contain information about technical challenges, resource availability, timing, deadlines, suggested and affirmed projects and so forth. Someone needs to coordinate all this. Presto!, the PM is there to help.

So how did anyone in Aptoma find the time for all of this when we didn’t have formal PMs? They didn’t. But these things just kind of works, to a point, when the company is small. We’ve exceeded this limit now, and this, our first small step towards dev-teams’ separation of concern is here.

6. Remove obstacles. The PM will, by nature of the role, gain overview of the work flow, and during reflections get insight into obstacles that are holding the team back. This can be recurring problems, communication challenges, and so forth. The PM will work to remove such obstacles, to make work flow smoothly.

Possible challenges

Boring maturity. We have to fine-tune this role for it not to become a boring formal meeting process. And we need to acknowledge that we’re not Toyota. Yet. We’re still a very small company, and a lot of people do a lot of different stuff. A job description any cleaner than your average pile of laundry still does not exist at 13 employees.

Conflict of interest. There’s been internal developer outcry for PMs for the last few months. And as expectations are high, it might be natural to think of a PM as a silver bullet “Ah, nice, here comes someone to do all stuff that is boring to me, so I can do only the fun parts. Woho!”. Chances are, however, that some work is fun for both PMs and developers. As such, fun vs. boring is not a well defined separation of concern. For separation of roles to work, one has to consider intellectually (what works, and what does not), as opposed to just consider emotionally what belongs where. The goal is better software and services for our customers, we must keep in mind.

Loss of agility. If the PM (the bureaucrat that he is) goes all-in on on functional specs, and basically tries to solve the entire problem of creating new functionality/products there, this might lead to over-planning causing unnecessary delays and loss of agility, innovation and flexibility. Knowing programmers quite well, I’m quite sure he’ll get a friendly (or even hostile) kick in the butt if this occurs.

Expected Benefits

1. More time for developers to focus on development.
2. Deeper empathical understanding of our customers needs.
3. Better and more holistic planning; one guy gets access to all the input and sees the whole picture.
4. Better coordination to strategy; PMs will report to the director, which makes it easier to provide feedback on direction, and to fine tune strategy.

What We Aim to Preserve

1. The autonomy of teams; As the director gets higher quality reports, and as planning gets well managed, it might be tempting for top management to mingle more and leave less decisions to the team. This can be harmful.
2. Developer ownership of ideas; The point of program management is not to shield the dev-team completely from information and customer concerns, but to structure and wisely involve at the right time. It’s a balancing act.
3. A peer team-structure; PMs don’t need ties. They’re embedded in the team. Authority comes from professional and personal qualities, not from formalities.

PS! I say “he” not because we don’t think women can be great developers or PMs, but because this is a note about us. All PMs and developers are still men in this hot dog shop. Don’t let that scare you from contacting us, if you’re a woman.

Identifying our own weaknesses

Reflections were very open hearted, which made it a very useful exercise.

Not enough “slack” in the system

This surfaced as a root cause for many of the negative caracteristics we were able to find about ourselves. We’re always pressured for progress on all fronts, by customers, by our own ambitions and by the need for more revenue to keep going. And even though this tension between where we are and where we’d like to be drives us forward at breakneck speeds, it does have some negative side-effects.

a) “Sharpening the saw” Not enough time is spend educating ourselves.
b) Sharing: It feels like disturbing each other when we need to get help or want to share.
c) At times, lack of slack limits our capacity to jump straight onto new important challenges.

Solutions to this problem are many. In many ways it’s a management problem. But solutions rarly have only one single solution. As the management problem might have schrunk in the past years, the culture for pushing the limits of the machinery is there still.

We need to

• Take the time needed to sharpen the saw. You don’t get more time, you take it.
• Drop projects that are not core business.
• Be clear on when we are available, and when we’re not.
• Insist stronger to hand over projects to customers’ internal development teams.
• Hire more people.

And please note that the problem is one of ambition, not of sloppiness — we’re not working hard due to excessive amounts of bugs. We’re out there pushing forward.

Not enough sharing between projects

Lack of sharing is not always a problem. (Creating shared dependencies between products can be very limiting to innovation. Google’s Eric Scmhidt knows this, as he recently was quoted “I learned a long time ago, don’t try to force technology to be merged.” And thus independency is a strength, too) However, there are ways to increase sharing and still keep the benefit higher than the cost.

We’re always on the lookout for ways to share code and technology without creating limiting dependencies. We’ve had several unsuccesful attempts at this in the past, but we’re getting better. We’re a bit concerned that we’ve been scared by our earlier failures, but we’ve now made a concious decision to keep up the dialogue of a possible “loosely coupled” sharing architecture.

We have a lot to benefit from more knowledge sharing, though. Learning more about “who knows what about what” we believe will enable us to improve our ability to pull knowledge when needed. Combined with fixing the “lack of slack”-problem above, we hope that this will trigger many an interesting conversation in the time to come.

We’re forcing ourselves to share more information such as “I’m going to be working on caching technologies for the next two weeks. Let me know if you have anything to share.” and then to be sharing the results findings in each projects regularly (i.e. every two months in an interactive whiteboard-session)

Our web-site stinks

Being quite proud of our individual and collective capabilities, we like to think of ourselves as a Ferrari-engine inside a Volkswagen. Thus it’s time to do something serious about the way we present ourselves. That does not mean that you’ll be seeing us coding in suits any time soon, but we’ll definitely have to do something about our web-site. We’re currently on the lookout for graphical designers that can understand our business and help with design and profiling. Let us know if you have any tips. Pimp my http://aptoma.com. We need it.

Other caracteristics

As you can see we’re more interested in what we can improve, than what we’re good at already, and we do think we’re a bit hard on ourselves. Some caracteristics have negative side-effects, and some have positive, but most have both positive and negative aspects. Below are some caracteristics with good side-effects, with a few ones with negative side-effects thrown in for good measure.

• We’re sometimes a bit hard on ourselves.
• We’re very agile – Customers not accustomed to our style sometimes find that this makes us seem unpredictable — we’re not commiting for distant detailed roadmaps, as we’re adapting to the ever changing future as it unfolds. We see this as a strenght, and customers tend to do so, too, when they’ve seen us in action for a while.
• We like doing everything ourselves – You don’t have to look further than our web-site to understand that this is not always good. We learn a lot from coding our own sites, frameworks and tools, but sometimes the time has come to admit that we have learned enough and should move on to build on other’s work.
• We’re very focussed – Focussing means saying no to something. Customers don’t always love this. But we didn’t forget or ignore you, we just focussed on something more important for the time being. It’s our job to care for customer needs in the long-haul, also. When we get to your problem, it will also get our undivided attention.
• We don’t communicate enough internally.
• We think that exposing ourselves to customer feedback is a strenght, but sometimes it becomes too much and it ends up limiting our work.
• We’re very clear on what we value in our software: maintainability, usability/productivity and performance.
• We’re good at creating cheap proof of concepts before we solidify into production code, but sometimes we move on too fast which creates a bit more technical debt than necessary. But we’re very concious about our technical debt, and we have a track record for paying it down before it is too late.

During these strategy sessions we also spent a great deal of time talking about the future of large scale new media publishing, and our role in it. We’re very grateful and excited about our increasing opportunity to play an important role in the way ahead for innovative new media publishers.

And, oh, there’s one solution to all the problems here that we have yet to discuss. We could hire you and put your great skills to good use. Join us in our relentless quest for personal and collective productivity.

It got me thinking (once again) about all that is wrong with the software industry. I now have even more to report. Everywhere I look, I see a mess. It is important to acknowledge that the industry is in its infancy. Acknowledging this, you can confidently and profitably dedicate a lot of time to work on the system, not just in the system. Because the system is broken by default, and will remain so for many decades to come, I believe.

3 Industry Problems

1. Requirements are not well defined. The main problem, as I see it, is that we’re mixing means and ends, as Gilb puts it. We’re talking too much about what we want, and not so much the background for that wish, which is what we should be specifying. If we know why you want something (i.e. business strategy, broken into objectives), we can use our experience to devise better solutions. Gilb proceeds to explain how to quantify and measure requirements. He’s right to say that we should, but he’s just way ahead of us, and we need to fix the way we specify requirements before we can start measuring anything. His point remains valid.

2. Factory organization – Separated and specialized functional units passing work between them in a large, efficient factory organization is doomed to fail (planning & specification, design, coding, QA, bugfixing). Such disregard for tacit knowledge and basic motivational psychology should not exist. Economies of scale in development (and services) is a myth. This has been tested and has failed horribly (think waterfall). It still happens all the time, and in many industries, not just the software industry. Probably also where you are right now. The closest metaphor I can think of is a kindergarden separated into “baby-feeders”, “diaper changers” and “comforters” etc. Think holistically — the real value delivered in in a kindergarden are not merely the practical operations. By contrast it is the safety and relationship developed with a few adults that the child will value.

3. Budgeting – Budgeting is usually a waterfall process in disguise. Budgeting was a tool developed for manufacturing, and should have remained so. But, to a carpenter, every problem looks like a nail, and so budgets are for manufacturing, service organizations and even product development companies. We’re not in manufacturing. Don’t adapt their tools. And, yes, this includes the Toyota Production System. Watch out, they’ll make you decide in your budgets what to do for the next year, and hold you to your promises. They’ll even scientifically measure this. Budgets forces you to spend the best part of each autumn guessing and assuming while you really should be out there gaining knowledge about emergent challenges, and adapting to them. Time is scarce — don’t waste it budgeting.

What does the Industry Think are the Solutions?

1. Start XP/Scrum/Lean-teams
2. Start XP/Scrum/Lean-teams
3. Start XP/Scrum/Lean-teams

John Seddon says it well when he states that everything needs to be projects and tools for us westeners. “Toolheads”, he calls us. I believe he’s got a point. Even Lean, originally The Toyota System, has been subsumed into tools and projects. Every organization has got a lean manager, or a lean team these days — fixing problems functional department by functional department (oh, the irony) by introducing some tools and running some projects, and then moving on. An A3-reporting there, a daily scrum here, a pull system there, 80% code coverage over there, and so it goes. Basically they are piling new tools and projects onto a broken framework, instead of building the capacity to change the system from the inside by getting managers out there, getting knowledge, working on the system. Get away from budgets, spreadsheets, resource plans and other guesswork – get knowledge and let’s use it to improve the way we work.

The current industry solutions only have us doing the wrong thing better. There’s nothing more wasteful than doing well what should not be done at all.

What are the Real Solutions?

We need to acknowledge that the way work is organized is broken and start addressing it from the core. What makes this challenging, is that the entire organization needs to be in on this, not just the lean tools team or the new empowered Scrum team. These teams are usually not empowered to change the system. Start by getting that power. Talk to your management. Talk to your CEO. Be patient. Discuss. Elaborate. Start small, but do it every day.

You need to design your organization towards challenges (think in terms of what you are solving for your customers, and design against that), rather than just defaulting to functional departments with their meaningless budgets and policies.

This means organizing in cross-functional teams empowered to change the way work is done, with management as a support function embedded into the work. Everyone’s goal becomes to establish good flow in how we solve real problems for real customers in the most effective way possible.

This is when magic starts to happen, and the fun kicks in. Solving a real problem for a real customer, and actually being there to witness it, is motivating and stimulating. Communicating directly and related to real challenges, not just via plans, sprint-/project backlogs, meetings, reports, Gantt charts and budgets.

Freed from this burden we can spend time improving the way we work, just a little, but every single day. It’s really quite simple. It’s fun. It’s motivating. And not the least, it is very, very profitable. And it is sustainable. It scales.

No more factories with human cogs, no more economies of scale. We all want to be a linchpin. We need economies of flow.

Anyway, this is how we’re designing our organization, and it seems to work very well. Come to us and take part of this journey. We’re hiring in Oslo and Gothenburg. (Follow the author @geirber or the company @aptoma on twitter, or through RSS)

References

1. Why Do We Believe in Economy of Scale? Article on Bryce Harrison inc.
2. Linchpin – Book by Seth Godin, inspired by the same factory problem
3. 3 Major Problems with the Software Industry (Select *)
4. The surprising truth about what motivates us (YouTube)
5. What’s Wrong with Requirements Specifications (PDF)
6. Cultural Change is Free (John Seddon on Vimeo video)
7. Systems Thinking (Dry, but good, Wikipedia stuff)
8. Rethinking Lean service (Entertaining Podcast)
9. Economies of scale – It’s a myth (Entertaining Podcast)

In that case you need to build in slack for learning into the system. You want everyone to have time for sharing, improving and learning continuously, and you want to be able to come home from work with energy to pursue your goals for personal skill development even further.

It’s great fun to be good at what you choose to spend your life doing.

Since the primary raw materials in software product development are skills and intellect, rapid learning is the prime goal and the capital for product development. Fail to invest in raw materials for your business, and you’ll soon run out of stock. Don’t allow for time to learn, rather, require it.

Running a system at 100% yields all sorts of problems.

Symptoms include

• You stop talking to each other. “Sorry, can’t help, have to go do this thing that is  really urgent”
• You start building technical debt. “If I just skip to write that test (it’s really hard to write), then I can save 2 hours today to get to the next thing I have to do.”
• Shallow bug-fixing. “Need to fix this bug real fast, in order to get the other 4 bugs done, too! No time for digging further” Root cause analysis and regression testing get skimped on.
• Useful, although sometimes inconclusive and somewhat directionless, discussions plainly stops happening.
• Stress activates your lizard brain, which numbs the rational being. You are in survival mode.
• Work gets less inspiring and fun, which slows things down. Inspiration is high-octane fuel.
• People get exhausted, and energy-level is low during, and after work hours. You need to “relax” on Facebook at work and in front of the TV at home, instead of doing inspirational learning.

(Picture from Wikimedia commons)

If you have trouble grasping the concept, picture a congested highway. If you remove a small percentage of the cars, you get a magnitude of improvement in throughput and speed. This is a fact.

Did you ever try to squeeze a washing machine 100% full? No clothes get cleaned. Remove 10 or 20% of the clothes, and you have a winner.

Don’t maximize utilization.
Build in time for learning, improving and sharing.

If you, as we have been, find yourself in this mess. There are several ways to get out of it.

1. Say no. If you never say no, you don’t have a strategy — say no to the least important stuff. Saying no is hard, scary and extremely useful.
2. If you work for a consultancy firm, give up. Incentives for maximizing utilization is too high in consultancy companies, as they get paid for “productive hours” only. Quit your job, and go to work in a product development company, or a skilled in-house team. Anything but most consultancy firms, they will cannibalize you.
3. Scale up capacity to meet demand. What can I say, we are hiring.

Quantified

This is no science, but if I would be forced to put some numbers on this, I’d say that:

• Build in approximately 20% learning/sharing time during normal work hours. Yes, that’s one whole day a week. We use it for our monday school, impulsive discussions, reading, friday meeting (our arena for general discussions on processes) and blogging, to mention some activities.
• If you want, you can probably “work” 60 hours a week or more without any danger of burning out, but only as long as that time is spent balanced and having fun, learning and improving/being productive — and only if personal situation allows for it without sacrificing time spent on other important values, like your kids.
• Don’t build overtime into your standard operating procedures. Required overtime doing “productive” work should be the rare exception, not the default.

Find a sustainable pace where you are balancing learning, improving, being productive and all of the time having fun. That’s an important ingredient in the recipe for success.

By the way, did I mention that we’re hiring or that we’re hiring?

It’s official: We’re not doing Scrum anymore.

I will take the time to elaborate on why, and I’ll reflect on what we’ve learned from our experiences.

Why are we not scrumming?

The shortlist:

1. Iterations were too long – It would sometimes feel like a waterfall in miniature. Reality for most development teams is that circumstances change constantly, and developing the capacity to adapt to these changes are more important than planning iterations carefully, which is just a form of guesswork.
2. Too much overhead – Sprint planning felt a bit forced. We had to decide what to do for days or weeks to come, and ended up with requirements churn. Beyond a certain point during planning, we went into congestion, and stopped caring about the details.
3. Our sprints are too diverse – Sprint goals ended up  so diverse sometimes that we ended up more often than not with the sprint goal of “Finish all tasks” which was a quite uninspiring goal. (We tried very hard to create good sprint goals)
4. Demos not often enough – We need to show off the feature we just made right now, not in 14 or 30 days at the demo. That is way to late for us. Demoing should be an integral part of each feature or improvement you make. You need to be close to, and intimate, with real users. If we had investors or any other formal committee of sponsors that needed to approve everything we did, we would see the use of such “demo roundups” at regular intervals.
5. Too long release cycles – The scrum process batches a lot of changes, improvements, bugfixes and new features into periodical releases. We are working on continuous deployment with a steady stream of product improvements, which allows for more predictable releases and better fail-tolerance.
6. “Scrum is just the easy parts of XP”  – It’s a quote from Kent Beck, although he does not like to be quoted on that one. This is why most are doing Scrum in combination with XP of course. I think you in many cases are better off with XP without Scrum, but that’s another story. We’re not doing XP in its purest form, either. Even though XP has much stricter coding practices, XP also has its limitations. The lean startup movement (which Kent Beck himself seems to be quite interested in) complements the weaker parts of XP — the customer feedback part, especially in the initial phases of product development.

You might see a pattern here: We’re still doing most, or all, scrum practices, just more often and in smaller chunks. This brings us closer to Lean.

What did we learn from it?

You should have a Product Backlog, and it should be a very simple sortable list. The emphasis should be on keeping it short, however. A long list of features becomes an administrative overhead. We found ourselves sifting through much of the same tasks from sprint planning to sprint planning. Too much got captured in the sprint backlog. We are still using product backlogs actively.

You should hold sprint planning meetings. The problem with the scrum-format of it is that it has more emphasis on creating a plan up front (i.e. the waterfall comparison), than it has on the planning itself. If planning is a monthly activity (or bi-weekly, or whatever your sprint length is in scrum), you end up reading your plan more than you spend time planning. We find that planning is at its best a continuous, on demand, activity. Do more of it, often, but in smaller chunks, and keep less of it lying around. I will have to agree with the Dwight on this one : Plans are nothing; planning is everything.

Daily scrums solves a non-existing problem – I’m sure that it is not hard to find team that would benefit from daily scrums. If you are such a team, chances are you are not communicating enough. We find that by hanging out on the project IRC-channel doing continuous micro-coordination and sharing information, and by hooking up on Skype to discuss challenges and problems on-screen as they arise we have nothing left to coordinate or discuss on the daily scrums. Everyone is in the loop on what is happening.

Retrospects are pure beauty. Again, the problem with scrum for us is not that it is has bad practices (it has the best of practices, actually). The problem is that it is performed to seldom to have the desired effect. Retrospect is not an activity, it is a mindset, and has to permeate everything you do, and should be done on a continuous basis. Ask these questions about every little project, task and experiment you run. What went well? What could be better? What will I focus on improving? It’s key to learning, and learning is essential.

Under what circumstances would we consider scrum?

If we ever find ourselves in a multi-team environment with quite large teams (5-10 per team), with sponsors that needed formal updates and demos, we would be sure to consider both scrums and scrum of scrums.

So, was all our Scrumming just a waste of time?

No way. It was a learning experience.

Never forget this: There is no silver bullet. You should try different stuff, be critical and analytical in the process, ask why to just about everything, and you will learn a lot. Shed each practice that fails to materialize into concrete value for your development process. In the end, no process formalities will save you, your diverse experience will.

What will we do differently?

We will continue to draw inspiration from Scrum, Lean, XP, Lean Startup, Customer Development, Continuous Deployment, etc. But instead of changing ourselves to fit the process (as was much the case with our scrum-efforts), we will look at the core of their ideas, extract their essence and see how they could apply to our existing and intuitive way of doing things;

If being a keen student of processes and development theory has taught us only one thing, it is that we should trust our gut feeling, our sense of what is purposeful and what is common sense, and that most of our current practices are worth keeping, and improving.

I advise you to try to improve the quality of how your team already, and intuitively go about doing the work. Do not change the way you work entirely in an attempt to magically achieve quality and speed. This way you can better preserve what you already got figured out, and at the same time be improving continuously.

In plain words, what are we doing now?

We still do a lot of the scrum practices. We just do them more often, and in smaller chunks. You might even say continuously, to some extent. We have learned to cherish planning over plans, and a retrospective mindset over bi-weekly formal retrospects. We write less plans, we take less notes. Communication flows more freely instead of formally in batches.

We get more done, and with increased quality, and we also have Scrum to thank for it, but definitely not only Scrum.

One of our biggest ongoing project nowadays is our establishment of Continuous Deployment practices, and all the discipline that goes with such a strategy. (It requires some coordination when you have servers deployed all over Europe, as we do) You will need to optimize the entire value stream to achieve this. We’ll keep you posted on how it evolves, and the benefits our customers and we reap from it.

• Plan it
• Code it
• Test it
• Commit it
• Deploy it

Nothing out of the ordinary, it seems. The seemingly new thing about continuous deployment is that we remove all the waiting in between each of these steps. That is, we don’t batch changes before to testing them. You test immediately what you just coded. You commit it. Not to a branch due for merging in a week. No, directly to trunk. And you even deploy it at once to production. Done. Clear your mind of it, enjoy, relax, spend some time doing something proactive.

Develop, test, commit, deploy. One feature, bug-fix or improvement at a time. That is the mantra for continuous deployment.

Some developers, and perhaps particularly managers, will find the idea crazy at first. Release immediately? Omigod. But frequently this sensation of safety in postponing something, is not rooted in any real benefits, and there seems to be no rational answer to the question of: why wait? Did your code ever get better during the night? Some will even argue that big feature releases is a marketing invention, invented to squeeze more money out of customers frustrated with the bugs in the current version. This practice of batching changes in releases in turn got embraced by the industry as the professional way of conducting business. I could easily add this to my list of major problems with the software industry.

Wouldn’t it be nice to deliver a bug-fix the same day it was reported? The same hour, even? It might just be possible if you are continuously deploying product improvements like this, kaizen-style.

Where does this idea come from?

The idea of continuous deployment comes from the lean school of thought. In the mindset of lean kanban pull systems — where you are constantly looking to remove waste and minimize your batch-sizes — grouping a lot of big software changes together in big batches and big releases makes no sense.

Even Flickr is doing continuous deployment these days.

The whole idea is to respond to customer “pull”. The customer needs something now. The clock starts ticking. Is the feature more useful if you wait a bit before you start coding it? No. Then start now. Is the feature better if you wait with testing after coding it? No. Then test now. Wait until tomorrow to integrate into version system? Is that better? No. Then integrate. Does thing get better if we wait to deploy that new feature? No. Then deploy it. Stop your clock. The shorter the time, the better for the customer. And better for quality, it seems. And the better for you, the coder. Unreleased code is stressful. Released, working code, is stress free. The code works. It’s proven.

After a few iterations, our fear level was actually lower than how we used to feel after a staged release. Because we were committing less code per release, we could correlate issues to a release with certainty. (From Case Study: Continuous deployment makes releases non-events)

Bugs that are not discoverable in testing are easier to find and fix if you released only one feature, than if you batched a months worth of changes together in one big bang release. Or so the lean school of thought states. I agree. You would have written the code hours ago, and probably say ‘a-ha!’ the second you hear about it. 20 minutes later, a bug-fix is released. Why wait?

A few resources on lean from Amazon:

• Software Development: An Agile Toolkit
• Implementing Lean Software Development: From Concept to Cash
• Lean Thinking: Banish Waste and Create Wealth in Your Corporation
• The Toyota Way

What does it take?

Not surprisingly, continuous deployment requires a high level of team discipline. No goofing off and no more fixing everything mañana in testing. Practices such as 5 Why root cause analysis, extensive testing and automation will be required. But it will all lead you to better quality and higher iteration speed. Speed and quality, it is all related.

Eric Ries writes on O’Reilly Radar about “Continuous Deployment in 5 easy steps“. Eric Ries has gotten a lot of traction with his Lean Startup movement lately. And you can follow his thoughts and reports on personal experience on continuous deployment on his blog http://www.startuplessonslearned.com/

Look ahead

Will we pursue this issue of continuous deployment any further? In the name of delivering a high quality experience to our customers and users now, when they still need it: yes we will.

If you will join us in this, please follow this blog on RSS or follow the company or the author on twitter. If you digg it, then digg it.

a) Ease of Use

The software solution walks you gently through the process of solving your problems as intended. No distractions, no unnecessary decisions to make, no confusions, always heading towards the goal. In short : A great user experience.

b) Quality of Craft

Quality of Craft means that the software solution is easy to maintain, it’s easy to modify so that it’s always accurate in solving the ever changing problems it’s going after (and by that I don’t just mean in configuration, but easy to change the code of), it’s easy to move about (i.e. from server to server), it performs great, it responds immediately when interacted with and it scales, or can be modified to scale, in order to handle the traffic it will attract.

Photo by toffiloff

Isn’t that an intuitive definition?
The definition is common sense in software product development.
Perhaps a “Software Product 101” should start out with such a definition.

Thus, we should assume that everyone get’s this nailed down from the start, or through a little experience, no?

I’m sorry to report that surprisingly few players in the industry is able to see this.
And even fewer is able to execute it.

It’s always either or, never both. The tendency is quite prominent.

It seems as start-ups create great solution accuracy and a good user experience using customer oriented development methods, and in general a lean start-up methodology. Growth, however, turns them in the direction of stability and the need for predictability. Sadly this usually leads to a loss of the qualities in a) in favor of a few of the more stabilizing qualities in b).

As much as I agree that as a start-up turns into an enterprise, the need for operations stability and predictable performance increase. I do not, however, believe that this is achieved best by sacrificing the ability to adapt and to keep the software usable and accurate.

I believe that while sacrificing qualities in a) might seem to be the easiest path, it is not the only path, not to mention that it in no way is the right path.

I am part of a small, growing, start-up company, and we’re focusing our resources at keeping on the right path, maintaining the agility of a start-up, and the stability and predictability of an enterprise. It is possible — this path exists.

It’s hard to find. But we’ll find it, or die trying.

Related posts

• A post about buildings (and software quality).

Disadvantages of Pair Programming

Don’t four hands produce more code than two?

If you are a carpenter, you can hit more nails with four hands than with two. If you are a brick-layer you can lay more bricks with four hands than with two. If you are a developer you can write more production code with four hands than with two. Or wait. This is based on the assumption that the bottleneck in software development is a mechanical one ; the speed of which you can type on the keyboard. My 10 years as a developer and 5 years as a program manager tells me otherwise. The bottlenecks in somewhat innovative projects (non-repetitive) rarely or never include typing speed. Bottlenecks do include hesitation (doubting your own solution), introducing bugs and having to go back to fix it, suboptimal design that deserves a refactoring, lack of needed knowledge for the task, loss of focus and so on.

A small disclaimer before I go on : We have not been experimenting with Pair Programming for more than a few months, and my above observations are based on observing teams develop from a managerial point of view and from reflecting upon my own coding habits as a team member.

Benefits of Pair Programming

Amplify learning

The speed of learning in pair programming is overwhelming. Once you enter a topic one party has more experience in, you are in for a ride. I have never experienced learning at such a pace as when this occurs.

Prevent bugs

Bugs are cheapest when they are prevented  from being introduced with measures that does not put restraints on your risk-taking and innovation. Pairing keeps you innovating, and will still help you spot errors more frequently. You will be more critical of your own code, and you will have someone constantly looking for possible improvements in your code, such as stopping a bug in the making, or having an idea for an even better test.

Continuous improvement

Pair programming is a dialogue between two people trying to simultaneously program, and understand how to program better.

Articulating ideas

Pair programming forces you to put words to your thoughts. This will through constant practice improve your skills in articulating your ideas.

Share frustrations

The responsibility on one single programmer can be huge at times. It is relieving to carry the load together. A teams strength and robustness is greater than the sum of strengths.

Avoid hesitation

When exhausted for ideas or new angles, it can lead to hesitation to implement what you suspect is a sub-optimal solution. The union of ideas and angles between the two developers makes this a rarer incident, and the resulting discussions will usually help remove the knots causing hesitation.

Distributed resources

From a managers point of view, a migration of people between project modules, and even between projects, becomes vastly easier. This is a dream come true. Nobody is the sole “owner” of code anymore, and developers can move more freely between code sections and projects, simplifying the complex task of resource scheduling. Starting out on unfamiliar code means that you should be pair programming with someone with a lot of domain knowledge in the start.

How to do pair programming

Pair programming is two people using one keyboard to write code. Here are some important best-practices to go along with that description.

1. Choose a suitable partner for pair programming the given task during the daily scrum.
2. Sit side by side with only one keyboard and one mouse. Both should have a good view of the monitor.
3. It must be easy to slide the keyboard from person to person. Switch roles from typer to observer every so often.
4. Discuss and have a good time.

So, these are my reflections on the subject so far. I will be sure to update you (RSS) on any changes of opiniton as we continue to improve our development practices.

The Software Management Manifesto *)

You have the right to an overall plan. The team should tell you what they could accomplish in the next year or two, and tell you how much that would cost.

You have the right to see progress. From the very beginning of the project, the team should be producing functionality that you care about. The functionality should be in the form of a running system, proven to work by passing repeatable tests that you specify.

You have the right to change your mind. As software development proceeds, you should be able to substitute new functionality for old. You should be able to change the relative priorities of the features of the system, dictating what should be done first and what should be done later.

You have the right to be informed of schedule changes. Since some things will turn out to be easier than expected and some things harder, the schedule is going to change. You have the right be informed of such changes as soon as the programmers know about them. You have the right to exercise your business judgment by choosing among options for reducing scope so as to restore the original date. You have the right to cancel further development at any time and be left with a useful, working system that reflects your investment up to that date.

*) From Software Management Manifesto

Does your process ensure your customers these rights?

One of the more useful activities of Scrum involves a bit of self-reflection — namely the retrospect. Admit that what you did yesterday could be done better today, and you are on the way to continuous improvement — a goal in and of itself.

Scrum tells you to ask “What could be improved for the next sprint?” at each retrospect. It is usually hard to think of good answers to this question. In this post I make an effort to compile a list of the most usual suspects, and I also suggest some remedies we have seen to work.

If you are anything like us, your usual suspects would be:

Presented in no particular order, these are suggestions for improvement that we have discovered.

1. Complex and / or undocumented code

Did you stumble upon any code that were hard to understand and even the comments did not help you out? If you did, you know how time consuming and frustrating this is fixing such code. It is a giant waste of time — and you have to spend a lot of time understanding the whole picture in order to refactor it to simpler, documented code. Andidote : Refactor mercilessly. Stop writing legacy code.

2. Untested code

Did you ever struggle to track down a bug in code, only to find that it could have easily have been pinpointed in milliseconds if a test had existed? Did you ever fear improving a piece of code because you did not trust the tests to cover enough to tell you if you broke anything? This and similar kinds of project mistrust is a real killer for productivity. Antidote : Stop writing legacy code. If you have to use legacy code, write tests for the code you are about to use, first.

3. 90% done

Did you ever start coding on some functionality, only to find that it depends on something that is almost finished, but not quite? Obviously you’ll have to finish the 90% done item. And it takes a lot of time, you have to dig into it (losing the context of the current functionality you are working on) in order to complete it properly. Antidote : Schedule work one user-story at a time,  and bring all code to 100% completion — i.e make it pass all unit-tests and acceptance tests (customer tests) — before proceeding to the next feature.

4. “Iterating quality in”

Thinking of iterations in form of “write incomplete code now” and the next iteration (whenever that is) “I’ll make it right(er)”, will bring you into all sorts of trouble. This kind of mentality leads to both 1, 2 and 3 above. Antidote : Add more discipline to your iterations. All iterations should end in all commited code being 100% done. Iterations are lovely and necessary, and you will improve existing code during each of them, but you should do your best the first time, as well.

5. You tested what?!

Did you just receive feedback on stuff you did a month / two months / three months ago? Delaying acceptance-testing (customer testing) until a developer has forgotten all about the code is very waterfallish. Antidote : Keep cycle time from checked in code until testing, deploying and feedback as low as possible (hours or days, not weeks or months).

6. Setting architectural goals, instead of functional goals

Setting out to “Create world-saving fancy-schmancy plug-in-environment” instead of “Allow programmer to paste text-data through a plugin-api” (much smaller and better defined) will usually lead us into creating a lot of code that will not be used any time soon. Also, it is much harder to measure “done” in “Create fancy plugin-environment” and the goal in itself is likely to bee too big for one sprint. It leads us into the “90% done”-problem. Antidote : Deliver value continuously and in increments by splitting the problem into smaller user stories, each with a value for the customer. The user stories will be much smaller, allowing feedback and quality into the process.

7. Big-bang integrations

Branching for a while is not a problem when you keep your branch synchronized to trunk all the time. Merging to trunk becomes a breeze. The problem is created for the other guy(s) that have another active branch (god forbid). When you merge to trunk, they are (when synchronizing) smacked in the face with your last week of changes. Antidote : Branch only when you must in order to keep commiting daily changesets which would otherwise make the trunk unstable. Merge to trunk as soon as you have a state of passing all tests. Aim for worst-case scenarios of day(s) between merging to trunk, not weeks.

8. The future is strangling the present

If you are spending a lot of time in requirements (user-stories) the future is killing the present. You are probably going into details too early.  Antidote : Add just-in-time discipline to your process.

9. New functionality, but no users

Have you ever launched a new functionality that no-one were delighted to have immediately? This is a waste of time, and best-case scenario someone will start using it in 3 months and lead you into what resembles #5 above. Antidote : Your planning process is broken, involve real users in planning, make sure you are not planning to far ahead.

The “Standish Group Report – Chaos” (1995) points out a key difference between bridge building projects and software development projects (a difference besides the 3000 years of experience in building bridges). The key difference is that when a bridge falls down, it is investigated and a report is written on the cause of the failure. This is not so in the computer industry, where failures are covered up, ignored, and/or rationalized. As a result, we keep making the same mistakes over and over again.

Consequently, do not judge anyone by their mistakes, but by their ability to recover and learn from them. Ultimately this will determine your success.

Seeking to continuously improve materializes in admitting that the last time you did it, you did not do it as well as you could be doing it today. Finding out why, is the key. Today you can be better than yesterday, tomorrow you can be better than today, and so on. This is true if you are willing to take a honest and sincere look at your mistakes.

What could you have been doing better in your previous projects?
Do you recognize any of our usual suspects?

Stay tuned with RSS.