For so long there have been so many advocating the benefits of the various Agile, Lean, Iterative, or whatever-you-want-to-call-it-these-days methodologies. We, as software developers, seem to understand the benefits of these methods. So why, then, do we still have customers that insist on big design up front, waterfall based methodologies, and other practices that we know are wrong?

Some Ad-hoc Consulting

During my Christmas vacation, I had the opportunity to do some consulting / coaching for my dad’s current company. The employee list that I spoke to consisted of my dad, my aunt, and my cousin. It was a nice little family gathering, I got paid with some amazingly good barbeque, and it was a very enjoyable experience.

I’m a software developer working for a company that provides software development and related services. My dad’s company is a customer of a software development organization (not the one I work for), receiving some custom built software for a project. They have been running into some issues with quality and timeliness of delivery for the last year or so and were wanting to ask some very pointed questions about the state of the software development industry. Essentially, they wanted me to tell them whether or not the issues they were having are “normal” and / or “acceptable” when receiving custom software development services. My answer was an unfortunate “yes, this is normal”, but “no, this is not acceptable”.

Over the next few hours of our discussion, we touched on many different issues that they are having and how some of those issues can be addressed from their perspective and from the perspective of the software development supplier. In the end, I had successfully educated my dad’s company on a few key points, confirming their suspicions and observations, and giving some new directions to start heading with the company that is providing the development services. I can’t go into all of the detail, but the summary bullet points included the following:

• The customer won’t know what they want until they see something they don’t want
• The software provider should deliver a working, stable version of the system on a regular basis, allowing the customer to use it in it’s current form
• The customer sets the priority of features and functionality to be implementing, in coordination with the technical needs and pre-requisites of the software development organization
• The customer is allowed to say something is wrong, even when they receive what they asked for
• The relationship between the customer and service provider must be a whole-team view. That is, “we are one team. we succeed together and we fail together”

On top of these bullet points, I laid out a simplified Scrum process for them, based on the already stated 3 week development cycle that the provider prefers.

In the end, my dad’s company was shocked, astounded, amazed, and energized by everything they heard. The light bulbs were clicking in their eyes, and I could see the value that they received form the conversations. They have dramatically changed their understanding of what it means to be a good customer and what it means to be a good supplier.

A New Perspective

As much as I believe I helped my dad’s company in their understanding of what they should expect and how they should approach their issues and needs, I also learned a great deal from this experience. It was very enlightening for me to hear the issues that we have known for so long as software providers, but from the customer’s perspective. Nothing that they spoke of was unknown or unfamiliar to me. I have lived through all of the same pains and issues that they are going through – from both a software provider and a customer of software services.

What really stuck with me from this experience, though, is the idea that our perspective on how to improve our industry is likely wrong. Yes, we need to continue educating our own ranks on the benefits of better software development techniques and processes. However, it does no good for us to have the perfect pneumatic toolset when the customer is asking us to use a rusted hammer and a broken hacksaw.

What I’m getting at is that its not us, the software providers, that need to change so greatly. Rather, our customer base needs to be educated and change. We need to stop being so self-centered and introspective for a while so that we can educate our customer base on what it means to be a good customer; how they can accomplish their goal of receiving quality software that provides true business value for them, at an affordable price. Until we have a customer base that expects software to be high quality, the first time, every time; expects to be given working software on a regular basis; and expects to be able to recognize that what they thought they wanted is not what they really needed; we will have no chance of our inward looking self-improvements taking hold.

I have to say, the Agile Manifesto got it right from a values perspective. The values espoused there are what we need to be looking toward, but I think our perspective needs to be changed on how we enable those values to be reached.

Of course, if we do succeed in educating our customers, we need to be able to respond appropriately. So please, continue the education of our own industry. Create better development standards that increase our quality and productivity. If our customer expects a quality delivery every 2 weeks, we better be able to deliver a quality product every 2 weeks. Just don’t stop here. Educate your customers. Help them understand the financial impact of poor development standards and processes. Change the norms and improve our industry by creating a customer base that requires perfect. The manufacturing industry did it 20 to 30 years ago. It’s time for us to step up to that level.

Where Do We Go From Here?

So where do we go from here? How do we start educating our customers? How do we change the social norms of software development so that our customers expect us to behave more like professional craftsmen and engineers, delivering quality faster?

I don’t claim to have the answers and I don’t claim to be the first to encourage this change. I only hope that we as an industry will begin to understand that the customer is usually the one that sets the expectation for how often software is delivered, and that we need the customer to expect something better.

Imagine that there are three group of people going out for a hike. Within each group of people, we have a goal of everyone reaching a picnic table at the end of the hike (and no one can eat until everyone has arrived). One of the people in each group has a map and a compass, knows how to use them, and will be directing the group on where they need to be going (we’ll call that person “the Pathfinder”). Everyone else is just along for the hike and the picnic.

The Pathfinder in each group is given a specific set of instructions on how they are going to direct everyone else.

Group 1: Observe, Plan, Walk

1. Stand on top of the large rock at the beginning of the hiking trail.
2. Once you have a clear view of the area, plot a course in one hundred foot increments, with your map and compass.
3. Using any means necessary (pencil and paper, verbal communication, etc), describe the direction that the group is supposed to be walking. Give an updated direction to walk every one hundred feet, enduring the group can go around any obstacles such as large rocks, impassable tree groupings, and small creeks.
4. Once you have plotted the correct course, store the compass and map in your backpack with the understanding that you are not allowed to use them again, until you have reached the picnic area.
5. Have the entire group read the instructions, then set out on your hike.
6. You are only allowed to check that everyone is on course by re-reading the instructions that you wrote down, every 100 feet – when the team needs to change direction to avoid an obstacle.
7. Once your group has reached the picnic table, record the time that it took for the hike to be completed by everyone – yourself included.

Group 2: Observe, Plan, Walk, Start Over, Verify; Repeat;

1. Stand on top of the large rock at the beginning of the hiking trail.
2. Once you have a clear view of the area, plot a course of one hundred feet using your map and compass.
3. Using any means necessary (pencil and paper, verbal communication, etc), describe the direction that the group is supposed to be walking for the next one hundred feet, ensuring everyone in the group understands the directions.
4. Place your compass and map on the rock, then hop down from the rock. Re-join your group and lead them in the right direction for one hundred feet
5. Tie one end of a rope to a tree or post at the beginning of the one hundred foot walk.
6. Travel the one hundred foot distance with your team, ensuring that they end up in the correct location.
7. At the end of the one hundred foot walk, have the group stop where they are.
8. Tie the other end of the rope to a tree or post at the end of the one hundred foot walk.
9. Have you and your group observe the new surroundings and take note of any obstacles that were note previously seen, and offer suggestions on where to go next.
10. With the help of the ropes to guide you, walk back to the rock at the beginning of the trails, by yourself, and climb back on to it.
11. Plot the next one hundred foot leg of the journey from where the group currently is, using your map and compass.
12. Repeat this process from step 3 through 11, until your entire group has reached the picnic area.
13. Once your group has reached the picnic table, record the time that it took for the hike to be completed by everyone – yourself included.

Group 3: Observe, Plan, Walk, Verify; Repeat;

1. Stand on top of the large rock at the beginning of the hiking trail.
2. Once you have a clear view of the area and general understanding of how to get where you are going, plot a course of one hundred feet using your map and compass.
3. Using any means necessary (pencil and paper, verbal communication, etc), describe the direction that the group is supposed to be walking for the next one hundred feet.
4. Place your compass and map in your backpack, then join your group.
5. Let the group know where they are heading (the end goal) and where they need to be in the next one hundred feet, ensuring everyone in the group understands the directions that you outlined.
6. Tie one end of a rope to a tree or post at the beginning of the one hundred foot walk.
7. Lead the group in the right direction for one hundred feet, ensuring that everyone is able to safely navigate any obstacles.
8. Verify that that your group has stopped in the correct location, by using your map and compass and any other landmarks or means needed.
9. Tie the other end of the rope to a tree or post at the end of the one hundred foot walk.
10. Have you and your group observe the new surroundings and take note of any obstacles that were note previously seen, and offer suggestions on where to go next.
11. Repeat this process from step 3 through 10, until your entire group has reached the picnic area.
12. Once your group has reached the picnic table, record the time that it took for the hike to be completed by everyone – yourself included.

Predictions on Outcome?

• Which group do you think will make it to the picnic site first?
• Which group do you think will be the most happy with the way the group was organized and run?
• Which group of people do you think would be most likely to do the same process on another hike?

Observations of the Groups

Group 1 didn’t get to the picnic site very fast and wasn’t very happy about the experience. While everyone in the group was standing around waiting, the Pathfinder was sitting on top of a rock, writing things down in a notebook, trying to describe – from a very high level, where people were supposed to be walking. When the Pathfinder finally came down from the rock – possibly an hour or more after he climbed up there – the initial directions were good and people could start finding their way. As time went on, though, the detail of where they should be going became more and more unclear, because the Pathfinder could not nee the actual pathways below the tree tops and behind the large rocks. This caused Group 1 to constantly stumble and struggle, creating a situation where the team had to find paths around obstacles that they couldn’t previously see. Every now and then, they got lost because the description of the landmark was insufficient from the distance that it was recorded. They eventually got to the picnic site – well after everyone was hungry and tired. In the end, no one was happy with the experience and just wanted to go home.

Group 2 had an easier time finding their way. Since the Pathfinder and the group were able to see all of the obstacles within one hundred feet, the directions that the Pathfinder described was much more enjoyable. No one stumbled over anything or got lost. But it was a difficult journey, at times. The group had to veer off the original course many time, in order to avoid some previously unknown obstacles. In the end, the Pathfinder did a great job of keeping everyone focused on the goal – the picnic table and food. The problem, though, is that the group took forever to get to the picnic area. At first, having the Pathfinder re-check the direction they were heading was great. Everyone knew that he was keeping the group on target. However, every time the group reached their next stopping point, the Pathfinder would gather information about the next hundred feet, across a wide arc since he didn’t know which direction to go next, with great certainty. Only then could he travel all the way back to the original rock to verify that everyone was on the right path, heading toward the picnic table. After a few iterations of hiking, the process of verifying the course and location took more time than the process of hiking. The Pathfinder was simply repeating the same hike over and over again, to verify that they were on the original path laid out. No one in this group was disappointed in the path that they took to get to the picnic table, but no one appreciated the standing around and waiting for hours on end – waiting for the Pathfinder to get the next one hundred feet plotted. When they finally reached the picnic table, group 2 was full of hungry, grumpy people who just wanted to go home.

Group 3 got to the picnic site in record time. It was a difficult journey, at times – the group had to veer off the original course many time, in order to avoid some previously unknown obstacles. In the end, the Pathfinder did a great job of keeping everyone focused on the goal – the picnic table and food. With the ability to see all of the obstacles in the surrounding area, and the ability for the group to make decisions that would get them around obstacles while still making progress toward the picnic table, the group was successful in getting to the picnic table before lunchtime. Everyone on the group loved the trip and appreciated the ability have input in how they got to the picnic table. As long as the direction that they headed for the next one hundred feet did not cause a serious delay in reaching the picnic table, or was justifiable by avoid an obstacle, all input was considered before setting the new direction. Several of the people in group 3 decided that the would take a few more hours and head off into the next set of hiking trails, trying to reach another picnic table before dinner time. Other decided to bring their family out to the trail the next day, and re-walk the well defined path (noted by the ropes tied to the starting and ending points of each leg of the journey). Overall, everyone in group 3 enjoyed the experience.

The Real World

Now imagine that the Pathfinder is represented by some aspect of your organization. It may be the company or project leadership; it may be the customer; it may be an individual person or group of people within a project team. Your Pathfinder may not be a person or other sentient being, though. Perhaps its the Methodology (capital M, as defined in Peopleware) or political environment in your company.

Which Pathfinder(s) do you have in your company? Which Pathfinder would you rather work with? More importantly – if you are not working for / with the Pathfinder that you want, what steps are you taking to correct that? Be a catalyst for change – appropriately and professionally.

Continuously improve – always within the context of the goal.

My coworkers really do love me. This was their Christmas present for me, complete with shiny bow.

I was told to use it only for good, not for evil.

And yes, that is me preaching from my soap box.

Let's assume that we are doing the appropriate amount of testing during our development process. If we include TDD, test automation, test engineers and customer acceptance testing, we should find the majority of the bugs in our system before they are released. However, not every bug will be found. There will be some situation that no one thought about before. There will be some special circumstance on someone's computer that hasn't been accounted for. There will be some client data that does fit the expected variance, despite the data being valid. The point is, there will be something that breaks after we deliver the software. Worse yet - it doesn't even take delivery to find bugs. What happens when the software gets to customer acceptance and the customer says that something is wrong, broken or whatever? We simply must account for the inevitable bug fixes and emergency patches in our system.

Our current kanban board, with all it's pipelines and queues, hasn't addressed the need to address problems. But before we get into any changes to the board, we have to define and create the culture of quality that we need in our team.

A Culture Of Quality

There are many different aspects of quality and many different ways to view and interpret quality. The individual specifics, as always, come down to your team, the project and the needs of both. Even with these variances, though, there are a few key concepts that should be found in any culture of quality, including Whole Team / Collective Ownership and Zero Defects.

Collective Ownership

I've talked about this concept before (see above link) and would recommend reading those prior posts. One thing I would like to add, though is that collective ownership can only success when the team takes their individual egos out of the equation. We must be able to accept criticism as a way to improve ourselves and our team. When we let go of our ego, we can be honest with ourselves and others, enabling everyone to own every part of the system.

Zero Defects

This is a subject that I have not talked about before, but is prevalent throughout the continuous improvement philosophies that I subscribe to. Wikipedia lists four principles of Zero Defect methodologies, all of which are paramount to our culture of quality.

1. Quality is conformance to requirements
2. Defect prevention is preferable to quality inspection and correction
3. Zero Defects is the quality standard
4. Quality is measured in monetary terms – the Price of Nonconformance

Don't be confused by "requirements" in item number one, though. In the world of software development, the word "requirements" has many meanings - not simply the feature list or functional descriptions set forth by our customers. Our teams and processes have certain requirements that are imposed on top of, and into each of the business requirements for the software. These often include the use of source control systems, test driven development, principles like S.O.L.I.D., etc.

Quality Checks in Kanban

Collective ownership and zero defects are only the tip of the iceberg. There is so much more to a culture of quality and so many different aspects of quality to account for. However we define quality, though, our ability to create a culture of quality is necessary for our kanban system to work. Without it, we lose the continuous improvement and elimination of waste that defines Lean. With it, though, we can introduce some specific tools to our process and improve our kanban system by having it handle the errors that are found in our systems.

Andon

From Wikipedia:

"a system to notify management, maintenance, and other workers of a quality or process problem"

When an issue is found in our system, we need to notify the team immediately so that the problem can be taken care of. The idea in a manufacturing line is to allow a worker to stop the line when a problem is found. In our grocery store example, andon could be invoked by a shelf stocker noticing a loose nut or bolt on the shelf and then notifying a maintenance person. In software development, andon can take many different forms. We report our issues list during our daily standup. We create issue tickets in our issue management system. If we're lucky and work in a company that supports the idea of a team room, we may just need to look up or turn around and let the team know that we have problems. We could even place an actual red flag on the desk of our developers, testers, customer representatives, etc., and have them raise that flag when they see a problem. Any of these ideas, plus many many more, can all be our andon system.

Andon itself is not intended to be an all encompassing electronic system with metrics and reports and yadda yadda yadda. It needs to be simple. It needs to be easily employed by anyone on the team. And it needs to mean something to the team. If a team member throws their andon card out on the table, the culture of that team needs to be ingrained with the knowledge that work may stop until the problem is addressed.

There's so little to andon, yet so much more than what I've described. However andon is enabled, it is critical to our zero defect policy in software development.

Jidoka

AKA "Autonomation", AKA "Automation with a human touch", AKA "intelligent automation", AKA ...

From Wikepedia:

"Autonomation prevents the production of defective products, eliminates overproduction and focuses attention on understanding the problem and ensuring that it never recurs."

I've talked about Jidoka in previous posts and would recommend reading them, at this point.

• Jidoka - If it's broke, fix it now
• Jidoka - More than just 'if it's broke, fix it now" - fix it and inform the team

I don't have much else to add, other than to say that Jidoka and Andon go hand in hand. Automated build servers (such as CCNet, TeamCity and many others) can often combine Andon and Jidoka for us by giving us instant visual feedback when something is broken. I've also recently set up BigVisibleCruise in my team area, giving even the casual observer the knowledge of whether our builds are broken or not.

Applying Andon and Jidoka to Our Kanban Board

Once we have the concepts of Andon and Jidoka in our team and culture, we can use these tools to generate issue cards and then look at a few possible changes to our kanban board to account for them. The three basic methods that I have seen used include:

1. Creating an Emergency Fixes pipeline
2. Tacking a smaller bug notice onto an existing card
3. Putting a Bug card in the backlog

I'm sure there are other alternatives, too. In my current team, we use the first and third method and are considering the second one as well. 

Options two and three essentially require no change to our kanban board. Implementing option two is a distinctive way of visually attaching a bug notice to one of the cards in our system. This could be done with little bug stickers, little red cards, or any other visual indicator that the team agrees on. Option three also needs something distinctive. Since we are creating an entire card for the bug, though, the entire card should be distinctive. I would recommend using a card that is colored red to signify issues. I would also recommend prioritizing the bug to the top of the backlog queue, when using option three. This will ensure that the bug gets worked as quickly as possible.

Option number one can also make use of number two and/or three. When we move a card into the Emergency Fixes pipeline, we may want it to be distinguishable as an issue by tacking on our bug symbol or by using a colored card.

Emergency Fixes Pipeline

Depending on the complexity or severity of the bug, we may want to include Analysis and Customer Acceptance in our Emergency Fixes. With andon and jidoka in mind, we will want to ensure that we fix any emergency issue immediately. This is not always possible, however. The customer may decide to delay the fixing of an issue for whatever reason. This leads us to only needing a single pipeline for Emergency Fixes, letting us set our limit to one.

We can easily add an Emergency Fixes pipeline to our kanban system, placing it directly underneath our existing WIP pipeline. This special pipeline can be designated with a name, color code, or other marks as needed.

If you are supporting production releases, you may also need to include a Delivery queue specifically for emergency fixes. This would allow multiple fixes to be compiled into a single patch release. There are other configurations to this, of course. As always, you will need to find what your specific team needs and create your process to suit.

Where Do We Go From Here?

With an Emergency Fixes pipeline in place, our kanban system is now set up to handle just about every situation that we will encounter. However, this does not mean that our system is perfect or truly complete. No process, no matter how well defined it is, is worth anything if the people running the process do not believe in it. I've said it before and I'll continue to say it - never stop improving your process. Always be mindful of waste, friction, smells, problems or whatever you want to call it. Inspect, adapt and continuously improve your process. Perfection is a journey, not an end-goal.

Stay tuned to my Adventures in Lean series, as I continue to explore the various aspects of lean software development in my own team and company culture.

In part 2 of my Kanban in Software Development series, I talked about completing a kanban board with queues, order points and limits. We saw how to take a complete development pipeline and work with a team, its processes and its bottlenecks. In the end, we had a kanban board that could easily represent the processes of the fictional team that we outlined.

One of the questions that I've often asked about a kanban board is how anyone would know when work in one column is done and ready to be pulled into the next column. For example - if a kanban card is sitting in the Analysis column, how does a developer know when that card is done so that they can pull it into Development and start coding it? I found the answer to this question when I was at the Kaizen Conference in October. Jef Newsom did a workshop on kanban and we ended up with this same question, and a solution.

A Pipeline for Analysis - WIP and Done

To facilitate the visualization of the difference between work in progress and work that is ready to be pulled to the next column, we can use the concept of a pipeline and split our existing queues into a WIP and Done step. For example, we want developers to pull work from the Analysis queue into the Development pipeline. We can show which cards are ready to move by splitting Analysis into sub-columns of WIP and Done.

When an Analyst is ready for work, they would pull from the Backlog into the Analysis / WIP column. When the analysis work is done and the card is ready to go into development, the card would be moved into the Analysis / Done column. Since we are wanting to maintain the concept of a queue for the overall Analysis column, we have create the WIP and Done subdivisions as a pipeline (noted by the dashed line). This allows us to keep our order point (3) and limit (5) in place for Analysis, and know what work is ready to be pulled into Development.

Applying Pipeline per Queue Across The Board

Not every queue needs to be a pipeline.

Consider the Backlog - the customer is placing the prioritized list of features in this column. The cards in this column exist so that the analysts will know what work needs to be done - not because any work needs to be done in this column, explicitly.

The Delivery column may not need a pipeline, either. If the delivery process is composed of sending an installer package to the customer, then there is no real work to be done in this column aside from sending that package. However, if there is some specific integration work (say, changing a web.config file) that needs to be done, we could include a WIP and Done pipeline for delivery. For this simple example, we'll say that there is no configuration change needed. Let's assume that the installation package changes all the needed configuration files based on user input.

With all of this in mind, we can apply our WIP and Done pipeline to the Analysis and Customer Acceptance columns.

Alternative Visualizations

There are other ways of visualizing WIP vs Done cards for a given column, of course. For example, you could forgo the pipeline idea and simply add a queue between Analysis and Development. This may require you to adjust your order points and limits between analysis and development - at least, where those order points and limits apply (Analysis vs. the queue between Analysis and Development). You could also split the column horizontally instead of vertically, creating either the top or bottom half as the WIP with the other being Done. Or you could just note the status on the kanban card itself. I'm sure there are a number of other ways to show this information, as well. The point is that the pipeline visualization that I'm showing here is not the only way. Find what works for you.

Where Do We Go From Here?

We have another tool in our belt, now. If need be, we can visualize the WIP vs work Done for a given queue, allowing us to see when we can pull work forward. But, like all tools in our belt, it is not *the* answer and this additional visualization may not be necessary for your team.

If your team is small or has great communication, you may not need any special distinctions to show work in progress vs work ready to be pulled. You could rely on the daily standup for people to report when they are done, or do it ad-hoc - when someone finished a card, they announce it to the team immediately. In a larger team, though, it may become necessary to have some visual distinction between work in progress and work completed for a given column.

Small teams might get away without it; larger teams might need it; its up to your team to decide. The key is not to apply these ideas as blanket rules that must be followed, but to allow the individual project and team to decide what is right for them. And as always, allow the process to change when it needs to. Always inspect, adapt and continuously improve.

In Part 1 of Kanban in Software Development, I introduced the concepts of kanban boards and pipelines. I also showed a very simple example of creating a pipeline for our development process. However, there were some obvious limitations in what I showed. The reality of software development is much more complicated than three simple steps (Analysis, Development and Testing). A software development team or company is going to have more to do than just these things, and there is usually a need for some team members to be cross-functional. Some team members will have to do development and testing, or analysis and development, or documentation and delivery, or any other combination of steps involved in creating software. In this post, I'll address these issues, creating a more complete pipeline and finishing our kanban board.

Working In A More Complete Pipeline

To create a more complete kanban board, we need more than just a three step pipeline. We need to allow for a fully functional team - developers, analysts, testers, technical writers, and others. We also need to allow the different team members to work on different parts of the system, as work is needed. The end goal is to enable the system to flow through the process and to ensure the work is "done done" before it goes to the customers.

For a more complete software development pipeline, let's use the following columns: Backlog, Analysis, Development, Documentation, Testing, Customer Acceptance, and Delivery. We can put together a pipeline diagram that follows these steps.

Now, let's assume that we have the following team (ignoring project managers, customer representatives, etc, for now): 6 developers, 2 business analysts, 3 test automation engineers, and 3 technical writers. Given this team, we would not be able to make a feature flow through this pipeline. We would have developers sitting around, waiting for work from the analysts. And, our documentation writers and test lab people would probably be pulling his hair out from boredom then pulling their hair out from too much work, in an unbreakable cycle.

Fortunately, we can account for the team makeup and the potential bottlenecks by introducing the concept of order points and limits that we saw in the grocery store, and by creating queues and multiple pipelines to be worked.

The Real World - Multiple Pipelines and Limits

In most software development efforts, it's unreasonable to expect an entire team to work on only one feature at a time. Most development managers want to maximize the throughput of development by working on tasks in parallel. Given this desire, we'll divide up our developers into three teams of two developers each. This development team division should allow three features to be developed at the same time. Since we want to work on three features at a time, we will need three pipelines for work.

These three pipelines constitute our first limit - we can have a maximum of three features in development at any given time. This limit is noted by putting a "3" in the upper right hand corner of the development column header, as shown above.

Having three development pipelines puts us in a tricky situation, though. We have very few business analysts compared to developers - even with our developer teams, we have less than one analyst per developer team. Fortunately, development work often takes longer than analysis. We can use this knowledge to our advantage and let the analysts work slightly ahead of the developers, creating a queue of work to be done.

Handling Bottlenecks - Queues, Limits and Order Points

If all three development teams happen to finish at the same time and need more work, we would need a minimum of three features that are ready to be worked. However, since we know that requirements change regularly, we don't want to get too far ahead of the developers. With this in mind, we can introduce an Analysis order point of three (the number of development teams) and limit of five. This gives the analysts the ability to work ahead of the developers and also makes them responsible for keeping work available for developers.

For an order point we'll introduce the number into the upper left hand corner of a column header, and we'll continue using the upper right hand corner to specify our limits.

Since we are now dealing with potentially more work in analysis than development, it seems we would have to increase our pipelines to five. This is not desirable, though, since we only have three development teams. What we will do, instead, is restructure the pipeline and turn the analysis column into a queue.

Since Analysis is now a queue, there does not seems to be a need for Backlog to be a step in a pipeline, either. The backlog is simply the list of features that the customer is expecting to be done next. Therefore, we will also turn the backlog into a queue - this time, with a priority (top of the board is highest priority) allowing the customer to tell us which specific feature should be worked on next. We will also want to keep the backlog column limited, to prevent the team from having too much information to think about. Since we only have two analysts on our team, it seems appropriate to keep at least two features in the backlog column at all times. With this in mind, we can safely set a backlog order point of two and limit of five (enough to keep the analysis column full).

The changing of Backlog into a queue has no made our Development column the first step in our actual pipeline.

Completing The Pipeline - Aggregate Limits and Work Cells

The next few column - Documentation and Testing - both have an easy amount of team members to deal with. There are three technical writers on our team, and three testing personnel. This distribution lets us keep the pipeline in tact between Development, Documentation and Testing, allowing us to set the same limit as Development (three).

With all three of these columns having the same limits, and with each column being properly staffed so that three pipelines could run at the same time, it makes sense to combine the team members from these three columns into a single workcell. If we consider all four team members (2 developers, 1 tech writer, 1 tester) as a team and allow them to work together as a team, we can aggregate the limits of the columns in the pipeline. We can also create a consolidated name for this pipeline - work in progress.

This consolidation can be shown by creating a parent / child header with the limit shown in the parent. We will also use a dashed lines between the columns of the pipelines. The combination of these two details will show us that we are dealing with a pipeline, and how many pipelines are allowed to flow, simultaneously.

By aggregating the pipeline limits, we allow the workcell for a given feature to focus exclusively on that feature. The development staff, technical writers, and test lab personnel will all be working on the same feature, allowing them to more easily share information between themselves. This will prevent the team members from having to switch back and forth between subjects, reducing cognitive load and allowing for greater quality to be attained in the individual feature.

Completing The Kanban Board - Customer Acceptance and Delivery

If the customer is on-site (co-located with the team or close enough to meet every day), then it may be possible to aggregate the Customer Acceptance process into the Work In Progress pipeline. If you can do this, you should. Getting immediate feedback from the customer, while work is still fresh in you mind, is critical to the quality and success of that work. What's more - you may not need this explicit column if you have the customer working with you every day.

Unfortunately, we don't always have the luxury of on-site customers. We may only have the customer around at specific timer intervals or only on request. In either of these cases, we have to account for the bottleneck of Customer Acceptance.  Additionally, the customer may not want to get delivery of individual features. They may want to wait for a quarterly release, or some other regularly scheduled release. Once again, we have to account for this bottleneck. We can apply the same principles that we used for Backlog and Analysis here, and create some appropriate order points limits.

Let's say that our customer wants to do Customer Acceptance testing once a week and wants a Delivery once every calendar quarter. To accommodate this, we can set the Customer Acceptance column as a queue with a 1 week limit. We can also set the Delivery column as a queue with a 1 quarter limit. Setting these limits puts a direct responsibility on the customer. If they cannot pull the features through these columns on or before the specified limits, then the entire system could grind to a halt. It's important to educate the customer in what they are committing to, and ensure that they can fulfill their obligations. This fulfillment may mean that they assign more than one person or group of people to the task of Customer Acceptance and accepting Delivery. Whatever the solution is, the customer has responsibilities to meet.

With Customer Acceptance and Delivery specified, our completed kanban board would look like this:

Put It In Action - A Kanban Process Is Never Truly "Complete"

Now that we have clearly defined our Kanban process and visualized it through our Kanban board, the next steps is to start using it! Start working through this process, pull value from the end of the process and allow work to flow through the pipeline. Respect the order points and limits, and above all, look for problems in the process and fix the process when you need to.

Perhaps you have too few or too many items in one of your queues - adjust the limits and order points. Perhaps the test lab is now a bottleneck - take them out of the pipeline and change them to a queue. There are many possible problems and many possible solutions. The key is to be aware - inspect and adapt and hold regular retrospectives - kaizen your process and continuously improve.

Where Do We Go From Here?

No process is perfect. Anyone that tells you otherwise is selling something. A customer may find a bug during Customer Acceptance; an issue makes it past all of our QA processes and is found out in production; or there's a feature that has a sudden priority over any other feature currently being worked. But our current kanban process doesn't address the natural problems that occur during a software package's lifetime. To address this properly, we'll need to employ the lean tools of Andon and Jidoka, and we'll also need to decide how to visualize this on our kanban board. These issues, and possibly more, will be addressed in another installment of Kanban in Software Development.

In the world of Scrum, XP and other forms of Agile software development, many teams use visual control systems to outline the various steps that software goes through during development. These boards are known by various names -  Scrum boards, card-boards, task-boards, swim lanes, and probably a few other names that I'm not aware of at the moment. Whatever you call it and whatever agile creed you live by, the primary use of these boards is the same all around - information radiation. Anyone can look at the board at any given moment in time, and know where in the development process a given card is. If you're doing iterations or sprints, the board also tells you where you are in that time box - if you're ahead, behind, or on time. The far left column - the iteration backlog - will be filled with cards at the beginning of the iteration. Over the coarse of the iteration, cards will be pushed through each of the columns until they are in the done column. The goal is for every card to be "done done" by the end of the iteration, with a set of features that is now deliverable to the customer.

Kanban Boards

Visual management tools are very common in agile shops, and for good reason - they work very well. It should come as no surprise, then, that kanban software development also employs various visual management tools, including a kanban board.

On the surface, there isn't much difference between an average task board and a kanban board. Each of these boards has various columns that represent the stages that a card needs to go through before it is considered done. The real difference in a kanban board, is not the board itself. The board is just a visual indicator, the same as any task board, and the intention is still to get the cards to the "done done" state - that is, delivered to the customer so that they can use the features from that card. The real difference is how the process is approached - by pulling value through the system.

Kanban Pipelines

When dealing with a kanban process and visualizing it into a kanban board, the various steps that a card goes through is often called a pipeline. A single card starts at one end of the pipe and flows through to the other end. This flow is enabled via the pull system that happens at the end of the pipe.

In the grocery store example from my previous post on pulling value, the pipeline would would likely include the store shelf, the store back room, the warehouse, the supplier and the product creator.

A Simple Software Development Pipeline

A software development pipeline works the same way as the grocery store pipeline. In this case, though, the product flowing through the pipe is likely to be a feature of the software package.

Consider the following three columns in a simple pipeline for software development: Analysis, Development and Testing. When a customer requests a given feature for a software product, they want to pull that feature out of testing so that they can start using it.

Once that feature has been moved out of Testing and the customer is ready to pull the next feature out, there isn't anything to pull. At this point, the Testing people would then try to pull the next feature out of Development.

And the same pull happens from Analysis to Development.

In the end, we have created a pipeline for how our development process works. The work that is done flows through that pipeline based on how often the customer wants to pull features out. As one feature exits the pipeline, another feature can be added into the pipeline.

The key to all of this is, again, pulling the features through the system.

Where Do We Go From Here?

I've briefly shown how we can take three steps and produce a simple pipeline for work to flow through. There is more to software development than just Analysis, Development and Testing, though. We also have to consider team size and makeup, parallel work, and other constraints. In the next entry of Kanban in Software Development, we'll flesh out a more complete pipeline and take the next steps to show how to complete our Kanban board, enabling our customers to pull features out of our development process.

For the last month or so, I've been having a serious problem with the apparent lack of professionalism in the software development industry as a whole. There are many factors contributing to this and many different aspects of the problem. I'm not going to go into any detail on the issues I'm seeing, at this point, though. It will be sufficient, for the moment, to simply state that there is a serious lack of professionalism in our industry.

I've been discussing this issue with my dad, recently, and this morning he sent me an email that I thought was worth sharing with the world.

Today’s thought is to remind the team that life in general is determined by the 2^nd Law of Thermal Dynamics. All matter (systems) go from a state of order to disorder with out any outside input.

[From Wikipedia]:

"The second law of thermodynamics is an expression of the universal law of increasing entropy, stating that the entropy of an isolated system which is not in equilibrium will tend to increase over time, approaching a maximum value at equilibrium.

The second law traces its origin to French physicist Sadi Carnot's 1824 paper Reflections on the Motive Power of Fire, which presented the view that motive power (work) is due to the fall of caloric (heat) from a hot to cold body (working substance). In simple terms, the second law is an expression of the fact that over time, ignoring the effects of self-gravity, differences in temperature, pressure, and density tend to even out in a physical system that is isolated from the outside world. Entropy is a measure of how far along this evening-out process has progressed.

There are many versions of the second law, but they all have the same effect, which is to explain the phenomenon of irreversibility in nature."

 

Bottom line is that the point of natural human equilibrium is far below the level of professionalism demanded by our responsibilities to work, self, family and others.

The Point … doing what’s natural or going with the flow points life in the downward direction every time.  Let’s tweak the line upward… put some thought and energy into it.

Robert D. Bailey   12-04-2008  

In the comments of my previous post - Descriptive State Enumeration - Maxim Tihobrazov asked me to show how to map a state pattern with NHibernate; and I am more than happy to oblige!

NHibernate Mapping Options

I certainly don't claim to be an expert on NHibernate, but I do use it on a daily basis and I've solved my fair share of problems - including how to map a State pattern with NHibernate. According the NHibernate inheritance documentation, there are three core ways of mapping this pattern:

1. Table per class hierarchy (I've always called it "record per class") - each subclass (state instance, in this case) is associated with a specific record in a table. All of the subclasses are found in the same table.
2. Table per subclass (also called a "joined sub class") - each sub class has it's own table and is joined to the super class by a one to one relationship with it's primary key.
3. Table per concrete class - each sub class has it's own table and is mapped by the specific class type, not by the abstraction.

My Choice - Table Per Class Hierarchy

In my current set of applications, I've always used the table per class hierarchy option so that is what I will describe here. My reasons for choosing this option are simplicity in the mapping and more importantly - not having a need to store the records in their own tables. Each of my states can be very cleanly represented as a record in my state table.

Setting Up Shop

Let's consider the same coffee shop that I've used in both of my PTOM posts - in this case, we're dealing specifically with the Order class and it's OrderStatus state. It's necessary to note that when I am working with NHibernate and the state pattern, I actually do set up my state models a little different. First off - we need an Id for NHibernate to map to the table's primary key. Secondly, I like to keep the core information in the abstract base class and provide all the varying values via a constructor. Third, each of the inheriting classes needs to provide a default (no args) constructor. Forth, we need a way for NHibernate to know which class it should actually instantiate, when loading - a "descriminator". I like to use a simple string Name property for this. And lastly - I think (though I am likely wrong on this aspect) that the abstract base class needs to provide a default constructor. Fortunately, the constructors we need don't need to be public - they can be private.

With all of that being said, here is realistic code base that I would use for my Order and OrderStatus model:

public class Order

{

   public int Id { get; set; }

   public OrderStatus Status { get; set; }

}

public abstract class OrderStatus

{

   public static InProcess = new InProcessStatus();

   public static Totaled = new TotaledStatus();

   public static Tendered = new TenderedStatus();

   public static Delivered = new DeliveredStatus();

   public int Id { get; set; }

   public bool DisplayForFullfillment { get; }

   public string Name { get; set; }

   private OrderStatus() { }

   private OrderStatus(int id, bool display, string name)

   {

       Id = id;

       DisplayForFullfillment = display;

       Name = name;

   }

}

public class InProcessStatus: OrderStatus

{

   private InProcessStatus(): base(1, false, "InProcess");

}

public class TotaledStatus: OrderStatus

{

   private TotaledStatus(): base(2, true, "Totaled");

}

public class TenderedStatus: OrderStatus

{

   private TenderedStatus(): base(3, true, "Tendered");

}

public class DeliveredStatus: OrderStatus

{

   private DeliveredStatus(): base(4, false, "Delivered");

}

The Fluent NHibernate Maps

I'm a huge fan of Fluent NHibernate. I've been using it since just after it was branched from the original ShadeTree code. And at one point, I had submitted so many patches that I was made a comitter on the project. So, it should be no surprise to anyone that I'm going to advocate using FluentNHibernate over the .hbm.xml mapping files. Truth be told, I don't even remember how to do the needed descriminators in hbm.xml files. There was some trick to the descriminator being the first specified items after the Id or something... it's just easier to do in FluentNHibernate, so I don't bother with hbm.xml files anymore.

The Order map is going to be as standard as any other map. You don't need to do anything special here. Just map the Id of the Order and the referenced OrderStatus. (I like to add a "CreateMap()" method that is called from the constructor, so that I can avoid the "virtual method call from a constructor" warning. But I also use the "treat warnings as errors" option for my C# projects).

public class OrderMap: ClassMap<Order>

{

   public OrderMap()

   {

       CreateMap();

   }

   public void CreateMap()

   {

       DefaultAccess.AsProperty();

       WithTable("Orders");

       Id(o => o.Id).GeneratedBy.Assigned();

   }

}

It's really the OrderStatus map that is special in this case. This is where we get into the details of the descriminator - telling NHibernate which specific instance to create, when loading the data from the database. In our case, we have added a "Name" field to our OrderStatus object and we will be explicitly using this property as the descriminator.

public class OrderStatusMap: ClassMap<OrderStatus>

{

   public OrderStatusMap()

   {

       CreateMap();

   }

   public void CreateMap()

   {

       DefaultAccess.AsProperty();

       WithTable("OrderStates");

       Id(s => s.Id).GeneratedBy.Assigned();

       DiscriminateSubClassesOnColumn<string>("Name")

           .SubClass<InProcessStatus>()

               .IsIdentifiedBy(OrderStatus.InProcess.Name)

               .MapSubClassColumns(x => { })

           .SubClass<TotaledStatus>()

               .IsIdentifiedBy(OrderStatus.Totaled.Name)

               .MapSubClassColumns(x => { })

           .SubClass<TenderedStatus>()

               .IsIdentifiedBy(OrderStatus.Tendered.Name)

               .MapSubClassColumns(x => { })

           .SubClass<DeliveredStatus>()

               .IsIdentifiedBy(OrderStatus.Delivered.Name)

               .MapSubClassColumns(x => { })

       Map(s => s.Name);    

   }

}

The key to all of this is the DescriminateSubClassesOnColumn method. The generics <string> tells us what .NET Type is being used to identify the specific class instances that we are dealing with and the ("Name") parameter is the column name in the database that represents the specific instance.

Then, we have the specific class instances referenced by the ".Subclass<type>" calls. This is a fluent interface, so ".Subclass" is a method that gets called directly off the DescriminatoeSubClassesOnColumn method. The specific type we want NHibernate to know about is specified in the generics parameter: ".SubClass<InProcessStatus>". The "IsIdentifiedBy" is where we give NHibernate the knowledge of what value in the "Name" column maps to what specific class in our code. To prevent magic string syndrome from setting in, I like to use the actual enumeration pattern of my OrderStatus to specify the string name of the class.

And finally, we have to provide an ugly workaround for our maps via the "MapSubclassColumns" method. There is an implementation issue in FluentNHibernate currently, and because of this issue we are forced to call the "MapSubclassColumns" method, with an empty lambda expression. If we don't call this method, the sub class will not get registered and NHibernate will not know how to handle the data in question. (I am hoping to fix this issue at some time, and make it so we don't have to call that method. I just haven't had time recently.)

Wrapping Up The Map

The rest of the map (the one remaining "name" property being mapped, in this case) is a regular NHibernate map. You'll notice, though, that we never mapped the "DispalyForFullfillment" property. I explicitly choose to leave any and all "volatile" properties out of my state maps. By doing this, I am able to add, edit, and remove any properties or methods on the state objects that I need, without having to change the NHibernate mappings. Since the state objects I am dealing with don't change without recompiling the code anyway, I don't need the ability to define them in the database. However, if you do need or want the flexibility of defining your states in the database, you can map each individual property of the state. Just remember that you will have to modify both the code and the database, to make the changes complete.

I hope this quick look at mapping a state pattern with Fluent NHibernate will help to shed some light on the subject, for someone. I realize that I have only provided the FluentNHibernate mapping, though. I specifically chose to do this because it would be a serious chore for myself to create a code project with NHibernate set up so that I can actually verify my mapping xml is correct. However, the translation from FluentNHibernate back to standard .hbm.xml files should be fairly straightforward, with the help of the NHibernate Documentation. If anyone out there is willing to help out and send me the correct .hbm.xml mapping, I would be more than happy to add it to this post and credit you with the work.

This post is part of the November 2008 Pablo's Topic Of The Month (PTOM) - Design Patterns and will primarily outline the State pattern, with an Enumeration or Descriptor pattern thrown in for good measure.

Switch statements and if-then statements are not object oriented code. They are conditional logic for procedural control and processing. This doesn't mean that they are bad - by no means. It only means that they are not object oriented. There are many legitimate reasons for using if-then statements and a few legitimate reasons for using switch statements. However, there are also many bad reasons for using these statements.

Setting Up Shop

Before we get into the guts of the State pattern, let's consider the Point of Sale system of our coffee shop again. When a customer is placing an order, we need to track what they are ordering. This would likely be done through an Order object. While the customer is placing their order, the order is considered "in process". Once the customer is done with their order, the cashier can "total" the order. Then, the order is "tendered" (paid for). And finally, the order is "delivered" to the customer. Each of these quoted items is a state that the order goes through during it's brief run through the coffee shop. In C#, it would be very easy to represent these states with a simple enum:

public enum OrderStatus

{

   InProcess = 0,

   Totaled = 1,

   Tendered = 2,

   Delivered = 3

}

With this enum in hand, we can have our point of sale system set the correct order status according to what is going on at the moment.

//when a coffee is ordered

Product regularCoffee = new Product("RegularCoffee", 2.99);

Order order = new Order();

order.Add(regularCoffee);

order.Status = OrderStatus.InProcess;

//when an order is totaled

double total = order.GetTotal();

order.Status = OrderStatus.Totaled;

//when the customer pays for the order

order.AmmountPaid = ammountPaidByCustomer;

order.Status = OrderStatus.Tendered;

//and finally, the order is delivered

order.Status = OrderStatus.Delivered;

Now this code we have so far is technically correct - it works and it gets the job done. Unfortunately, though, this code will likely lead us down a bunch of dark paths of code duplication, ugly switch statements, and an unmaintainable mess. For example, what happens when we start integration our Order object into the rest of the coffee shop and we need to display orders to fulfill on a screen for the order runners to fill. In such a case, we only want to show orders that have been tendered but not delivered, and to try and get orders out the door faster, we'll also show orders that are totaled.  If we were going to examine a single order to determine whether or not we wanted to show it, we would likely have one of the two following pieces of code in place (yes, I know this could be done with a database query. The point is that somewhere in the code you will likely end up with switch statements or if-then statements like this):

//one way to know

bool shouldShowOrder = false;

if (order.Status == OrderStatus.Totaled || order.Status == OrderStatus.Tendered)

     shouldShowOrder = true;

//another way to know

bool shouldShowOrder;

switch (order.Status)

{

   case OrderStatus.Totaled:

   {

       shouldShowOrder = true;

       break;

   }

   case OrderStatus.Tendered:

   {

       shouldShowOrder = true;

       break;

   }

   default:

   {

       shouldShowOrder = false;

   }

}

//after we know, we show it

if (shouldShowOrder)

     ShowTheOrderForFulfillment(order);

So - what happens when the order status list change? Or if we now have a need to show a different order status on our display? We would have to find all of the locations that used this status enum and check to see if it needs to be changed, to handle the new status or change in how the status is handled. This is where the problems really start - hoping that you got all of the uses changed and all of the right states handled in the right ways. Fortunately, there's a design pattern that we can use to simplify this situation and eliminate many of the associated switch and complex if-then statements.

The State Pattern

Wikipedia describes the State Pattern as "a clean way for an object to partially change its type at runtime." The C2 Wiki gives a little more insight into the State Pattern, as well: "Allow an object to alter its behavior when its internal state changes. The object will appear to change its class."

What both of these descriptions are really getting at, is that a single concept in the real world is likely going to be modeled as more than one class in code - composed of many different pieces. When any one part of a concepts model changes, the state of that concept is changed. The relationship between our Order class and OrderStatus enum are a prime example of this composition. What the state pattern allows us to do is change the behavior of our modeled concept by partially changing the composition of our model.

Modeling Our State As State Model

To model the state of our Order with a state pattern instead of a state enum, and begin introducing the ability to change our systems behavior without changing the code that relies on the state, we need to introduce an abstraction that can represent any state we care about. To start with, we need to know what orders should be shown on our display for the order runners.

public interface IOrderStatus

{

   bool DisplayForFulfillment { get; }

}

After introducing this abstraction, we can introduce any state implementation that we need.

public class InProcessStatus: IOrderStatus

{

   public bool DisplayForFullfillment { get { return false; } }

}

public class TotaledStatus: IOrderStatus

{

   public bool DisplayForFullfillment { get { return true; } }

}

public class TenderedStatus: IOrderStatus

{

   public bool DisplayForFullfillment { get { return true; } }

}

public class DeliveredStatus: IOrderStatus

{

   public bool DisplayForFullfillment { get { return false; } }

}

Once we have these states in place, we can replace our code that sets the state with these objects.

double total = order.GetTotal();

order.Status = new TotaledStatus();

And we can also significantly reduce the code that needs to know if we should display this order in our order display system. Now, instead of having to do those if-then or switch statements, we can simply check the order status:

//the order class now relies on the IOrderStatus for it's composition

public class Order

{

   public IOrderStatus Status { get; set; }

}

Yes, we are still using an if-then statement. However, this one simple statement is no longer an issue in terms of maintainability. We don't have to change this code as we add, remove, or change the states of our system. Since we only depend on the IOrderStatus interface for our composition now, we can freely change the implementation of the states as we need - change the composition of the Order's model.

But Wait! There's More!

We don't have to stop at simple boolean values on our state abstraction - we can provide actual behavior via methods in our state. And the best part is - when we can add and remove any behavior or boolean values, or whatever else we need to change with the state of our Order, we don't have to change any of the existing code that uses the existing DisplayForFullfillment value. This property is defined on the interface, letting us use it where we need it and ignore it where we don't need it.

By using a state pattern, as we've seen here, we have introduced the ability to change the behavior of our system without having to change the places that need to know about the state of the order. We've gained a great level of encapsulation. We've gained flexibility in changing states when we need to - and we have an abstraction that hides the details when we don't need them. Seems like a winning situation to me.

But Wait! There's Still More!

Unfortunately, one thing we lose is the friendly syntax of our enumeration. For example, "OrderStatus.Totaled." Personally - I like the simplicity that enumerations offer us in our code. Something about saying "OrderStatus.Totaled" seems to be right to me. So how do we marry the state pattern to the enumeration syntax? The answer is simple - an Enumeration pattern (also called a Descriptor pattern).

We can make a simple change to our order status abstraction - make it an abstract base class. Then we can provide some static fields to provide access to the class instances that we want. To enforce the enumeration like syntax, we can also make the constructor private.

public abstract class OrderStatus

{

   public static InProcess = new InProcessStatus();

   public static Totaled = new TotaledStatus();

   public static Tendered = new TenderedStatus();

   public static Delivered = new DeliveredStatus();

   public abstract DisplayForFullfillment { get; }

   private OrderStatus() { }

}

public class InProcessStatus: OrderStatus

{

   public bool DisplayForFullfillment { get { return false; } }

}

public class TotaledStatus: OrderStatus

{

   public bool DisplayForFullfillment { get { return true; } }

}

public class TenderedStatus: OrderStatus

{

   public bool DisplayForFullfillment { get { return true; } }

}

public class DeliveredStatus: OrderStatus

{

   public bool DisplayForFullfillment { get { return false; } }

}

//the order class now relies on the OrderStatus base class for it's composition

public class Order

{

   public OrderStatus Status { get; set; }

}

And now we have our nice friendly enumeration syntax back!

//when an order is totaled

double total = order.GetTotal();

order.Status = OrderStatus.Totaled;