Scott Bellware (who finally started blogging again), is posting a series on Sustaining Capacity in Maturing Agile Software Teams. It's worth a read, and may be familiar territory to anyone who has done Agile and been reading about Lean.

In the 3rd Post - Recognizing Entropy - he talks specifically about synchronizing the development testing and QA testing efforts:

Development and QA Test Synchronization. Without greater synchronization between development and QA testing, valuable input often comes too late in a development or release cycle to be effective. Test design and test architecture are valuable inputs to development, and software design and architecture are valuable inputs to QA testing. Teams often loose unrealized capacity by not pursuing means to do more development and testing in parallel.

This one really hit home for me, and is a shining beacon to the problems that my current team is going to face, soon. We had our first iteration planning and kickoff meeting yesterday, and the whole team was involved - except the testers from the QA department. For reasons that I have no control over (and honestly - they are legitimate business reasons), the QA department is not coming on to the project for another 2 months (at least).  Today, the team leadership was discussing how the QA personnel would create their test scenarios, and I brought up the point that creating the test scenarios after-the-fact (after the iterations are done) is going to be very problematic.

If a dev team is working from User Stories, and those User Stories are divided along various aspects of behavior, it is very likely that a single screen, workflow, or 'function' of a system is going to be divided into multiple user stories. When the QA team is involved with testing in parallel with the development team, this is not an issue because the QA team will be updating their test scenarios in each iteration. This allows them to keep the scenarios for a given screen, workflow, or 'function' of the system, growing over time. However, since our QA team is going to come into the project at least 2 months after the development team has started coding and testing, the QA team would have to sift through all of the current and previous iteration stories, try and aggregate them down into the screen, workflow, or 'function' of the system, and build test scenarios from there. If there is even a modest number of stories to sift through, this would be a daunting task for the QA team - they would be in a 'catch-up' mode from the beginning of their QA cycle, likely through the end of the project.

As a result of the QA team being unable to join our project immediately, we are forced to modify our entire development process and account for the needs of the QA team, after the fact. Essentially, we are creating additional swim-lanes for our stories to travel through, during an iteration:

1. Backlog
2. In Development
3. Developer Testing
4. Ad-Hoc Testing (BA's, Developers, etc. testing the app)
5. QA Functional Spec Documentation
6. Done

Swim-lane #4 and #5 are directly affected by the QA team not being able to enter the project immediately, and are causing the rest of the team to lower it's capacity. We (developers and BA's) are now required to do our own ad-hoc testing during the iterations, instead of letting the QA team take care of that; and we (the developers and BA's) are now required to create functional specifications, based on the user stories that are completed, to hand to the QA team once they are on the project. The functional specification documents will need to be updated for any given User Story, before the ticket is considered complete (the functional spec updates have become an acceptance criteria on every User Story).

End result - because we cannot get the QA team on our project immediately, and ensure that they are testing in parallel to the developers, we as the development team are losing capacity and building entropy from the very first day of the very first iteration.

Yesterday, I posted a quick thought on code generation, and one of the statements I made is worth re-stating to stand on it's own.

In Agile/Lean software development, a single User Story is one piece flow when implemented via TDD, in a Workcell

In Lean Manufacturing, one piece flow is the idea that you do not produce anything in batches, but that you produce one product from start to finish per a customer's order. In Lean / Agile software development, this is analogous to an iteration with user stories. An iteration backlog is a customer order - a set of behaviors or features that are requested to be delivered in the current iteration. A user story is the single piece that we want to flow from beginning to end - from the start of coding all the way to acceptance via user testing.

In manufacturing, a Workcell is used to facilitate one piece flow through the manufacturing process. In software development, a Workcell is also used to facilitate the one piece flow. The Workcell may be 1 person, pair programming or a three person team. In order for it to truly be one piece flow, though, strict standards such as TDD and automated acceptance tests must be followed.

I'm not going to detail the benefits of one piece flow, here (not yet, anyway... another post for another time). I highly recommend that you read The Toyota Way and Lean Thinking for a good understanding of this concept and the benefits it provides.

A coworker asked this question about our automated integration test suite, recently:

"with the RowTest feature of NUnit, does the test data being used need to be hard coded into my code or can it be a variable that gets defined based on some rule?"

Here is my response, outlining the need to know the state of the database (know every value of every field, in every table) before and after the test suite is run:

The short answer is that the tests will have data hard coded into them.

This may be a change from how you have looked at integration testing, previously – where the data is changed on each run, so as not to duplicate the data in the system. Rather than changing the data that is used during each test, we need to ensure that the data is exactly what we expect, before and after each test. If the data is not what we expect before the test is run, the test cannot produce the data that we expect to have after it is run. If the data is not what we expect after a test is run, the test fails.

At a very high level, automated tests against the database require three things:

1. A known, state of ALL data in the database
2. A known, suite of unit tests that manipulate said data in predictable ways
3. An expected, verifiable suite of data returned by the software and manipulated in the database

The data is not necessarily "perfect" in that it has no flaws in it – rather, every last character in every last field of every last table is known, and all of the ramifications of that data is known with the explicit purpose of that data being there to support the automated tests. The automated tests expect this "perfect" data to produce the desired results and verify that the system works as expected. Any data that is manipulated in the system is expected to be in a predictable state, so the tests can verify that the system manipulated the data correctly.

When it comes to implementation of this, there are some fairly strict requirements for it to work:

1. Before the first test is run from the test suite, the data must be exactly what the tests expect
2. After the tests have run, the data must be exactly what the tests expect
3. Repeat for each run of the test suite

What this really means is that we cannot add or modify data between two test runs, without changing the tests that work with the data. In order for the tests to run multiple times throughout the day and/or on-demand, we have to ensure that the data being manipulated is what we expect it to be. Therefore, the first step of any run of the test suite is to revert the database to the known state, through sql scripts or code that are automatically run before the tests are run.

The Toyota Way mentions the concept of Jidoka in chapter 1 (and probably other places that I haven't read yet). On page 6, in the "4 P" diagram, jidoka is described as

"Stop when there is a quality problem"

Wikipedia calls the same concept "Autonomation" and says it may be described as

"intelligent automation" or "automation with a human touch"

and

"At Toyota this usually means that if an abnormal situation arises the machine stops and the worker will stop the production line."

We can apply the same principles in software development in many different ways. One of the more common implementations is the use of Continuous Integration and Automated Testing.

According to the notes I took during A Day Of Bellware, if our CI server says the build is broken, we need to immediately stop working and fix the problem. I've heard this before, and will likely hear it again. However, I never really understood why people would say this. After all, as long as the problem is fixed eventually, isn't that ok? In the training that day, Scott provided a excellent visualization of why we should fix it immediately.

Waste and Rework

To steal Scott's illustrations, consider the following image to be an example of "perfect" software. All of the edges of each module (block) are well defined and it's easy stack new blocks next to existing blocks.

Figure 1. "Perfect" software

Now let's assume that somehwere in the coding process, someone accidentally causes a defect in the software. That defect could be represented by a buldge in one of the lines - as if the module was doing more than it should.

Figure 2. A defect

The individual block that has the defect may not be that bad, at first glance - or when examined on it's own. And on it's own, the defect could be fixed. Jidoka would say that you need to stop immediately and fix the problem.

So, what happens when you don't fix the problem right and and you try to stack another block to the right of the defect? Suddenly you find yourself re-shaping the next block in order to account for the problems in the previous one. Eventually, you may be able to smooth out the issues. Depending on the size of the issue, though, it may take several new modules to completely normalize things.

Figure 3. The effect of a defect

Now, 3 day, 3 weeks, 3 months, or whatever period of time later, you have a significantly larger problem to deal with. Let's go back and fix the original cause of the problem, to start with. What happens to the rest of the code that was warped around the original defect? The warped code in the rest of the system also has to be fixed. Simply removing the original bulge does not mean that the rest of the warps will magically disappear as well.

Figure 4. Removing the original bulge

How much re-work will it take to fix the rest of the system that was warped around the original problem? How much time and effort will be wasted because the original defect was not addressed immediately? These questions can only be answered in the context of your problems. If this defect is one line of code in one module, maybe it's not so bad. But if this defect is an entire module used by many other modules, the time and cost could be huge.

Conclusion: Fix It Now

If the continuous integration server says that our build is broken; or if the customer says "the software does not break, but piece XYZ isn't correct"; or ... pick a problem in your system; we need to respond as quickly as possible, in order to prevent the rest of the system from being warped by the original problem.

Under the assumption that we want to fix a defect as soon as possible - how do we ensure that we know about the problem as soon as possible? Whatever your implementation of this solution is, the solution to this problem comes down to shortening the feedback loop. If you are notified of the problem in 20 minutes vs. 20 days, there will be significantly less damage to the overall system and significantly less re-work and waste caused by the defect.

Myself and 10 other developers in my company went through a day of BDD / TDD training, with Scott Bellware, yesterday. It was a lot of fun, very challenging at times, and covered a lot of topics including an overview of Agile and Behavior Driven Development, all the way down to writing Specification Tests, doing Test Driven Development  and refactoring the model to improve readability, maintainability, flexibility, etc. I took notes via index cards (love that cool-aid) and wanted to share. I don't expect these notes to make sense to everyone. Hopefully it will spark some dialog in someone's mind and cause them to dig further.

First off - the quote of the day.

"Can I be honest with you and say that I've been wanting to touch your keyboard, all day?"

Now for my notes.

User Stories

[Role], [Goal], [Motiviation]

• As a [role], I want to [goal], so that [motivation]
• Example: As a nurse, I want to record a patients vital signs, so that I can determine their medication and care needs
• Motivation is critical - it determines how the development team understands and implements the story. It determines the user experience, how things are integrated, how the software is designed, etc.

Acceptance Criteria

• Acceptance Criteria is used to drive code, not the story, directly
• may change at any point, up to implementation
• is used to drive code design, test design, implementations, etc.
• should be spoken in domain language
• may include non-functional, technical details such as database tables, infrastructure, performance, etc
• All acceptance criteria must be met and tested / verified before a story is considered done

Specification Tests

• Test Fixture per Class is an anti-pattern (on a personal note, this problem bothered me for months before I discovered BDD)
• Context Specification or Behavior Specification testing
• When [verb] then [verb]
  • "When [verb]" is the context
  • "Then [verb]" is an observation of the behavior
• Based on Acceptance Criteria, but not "code-gen'd" from acceptance criteria

Story Estimation

• Agile Poker: uses generalized Fibonacci sequence as order of complexity
  • "?", 0, 1/2, 1, 2, 3, 5, 8, 13, 20, 40, 100, infinite
• everyone throws their estimate at same time
• if estimates have significant outliers, discussion occurs to understand why, get more detail, etc. and re-throw may happen

Entity Data vs. Aggregate Data

• Entities should never contain aggregate data
• Aggregate data is for reporting and other aggregate needs
• If you need aggregate data to process something, write an SQL query, stored proc, etc. - don't use an ORM like NHibernate
• We don't want a "Customer" entity to need 10,000 "Order" entities, to aggregate data for processing; write a query to aggregate instead
• We don't want to persist data that can be calculated / aggregated, generally (performance issues may override this)

Domain Services

• Can have dependencies on external systems
• are part of domain logic, therefore are in domain model / assembly
• are "Doers" of process that don't fit into entity and entity logic, directly
• coordination of entity logic
• can include calls to data access, logging, etc.

Continuous Integration

• Not just continuous compilation of code
• Full end to end integration of all code, components, databases, services, etc
• Full suite of integration testing including database testing
• Do not allow commits if build is currently broken
• do not allow defects to live - fix immediately, to fix build
• "Defect" is broken software, "Bug" is functional but wrong

Daily Scrum

• 3 Questions everyone answers:
  • What did I do yesterday?
  • What am I doing today?
  • What issues am I having?
• Each person should answer quickly - 1 or 2 minutes, max
• further discussion happens outside of the Scrum meeting

Productivity of Dev Team

• RAD and other non-review, non-iterative based management causes problems and loss of productivity
• we need constant review of the design to ensure good design
• shorten the feedback loop and get constant review of the design, to always improve the design, via pair programming, work cells, retrospectives, etc.
• good design will cause productivity gains in the development team beyond the capabilities of any tools

Whiteboard Diagramming vs. Details Specs

• White board diagramming and human interaction is always better than detailed documents and specs in UML
• Human interaction leads to knowledge crunching and learning, not just reading a repeating
• Take pictures, don't re-draw in UML; don't waste time with it
• Video the entire conversation is even better, so others can learn from the knowledge crunching that occurs; capture the human interaction, body language, etc.

I'm reading "Lean Software Development: An Agile Toolkit", and the first paragraph under "Tool 11: Queuing Theory" talks about the bottleneck that often occurs in the test lab - not enough testers, too much work for the number of testers, etc.

"We have often heard the lament 'My biggest problem is the testing department.' Now, testing people are very nice people: dedicated, hard working, and very important to the development effort. But there never seems to be enough of them to go around. And although the developers might write their own unit tests, testers frequently do acceptance testing. So, without enough testers, the whole development process bogs down."

The rest of the chapter talks about the queuing theory that can be applied to help alleviate the issue. It's a great chapter with lots of good information. I have a problem with the idea of applying this type of queuing theory to the the test lab, as a bottleneck, though.

First, let me state some assumptions about the primary responsibility of the testers in the test lab:

• Testers are writing automated acceptance tests, for automated regression testing and integration testing
• Testers are also doing human interaction testing, for the "human touch" of usability, etc.

If the testers are the bottleneck, and the two primary functions of the testers are as I have listed, then I think the there is a much more simple solution to the problem, from a lean perspective:

let the developers write the automated acceptance tests.

Assuming that the developers are already writing unit tests, and are therefore capable of writing code to test code, it makes a lot of sense in my mind that the developers should be writing the majority of the automated acceptance tests. It all goes back to the idea of flow - ensuring that the entire system (or process) has a smooth flow from beginning to end. This means that we may need to sub-optimize one area for the benefit of the whole, but the end result is that we will have a better system or process by making the entire flow as smooth as possible.

Counteracting Mura - or "Let's make it smooooooth"

If we are looking at the test lab as a bottleneck - a rough spot in the flow of our software development cycles - then let's take the most simple course of action possible, to reduce that rough spot as much as possible. Rather than spending so much time and effort on queuing theory and implementation, let's find a way to remove the bottleneck.

Assume that the software developers are experts at writing code - and writing code to test their code. Doesn't it make sense, then, that the software developers should be writing the acceptance tests, even if the acceptance tests are being specified by the customer and test lab personnel? If we allow the developers to take a little more responsibility, we may be sub-optimizing the development department a little. But, by doing so we are freeing up the much more scarce resources of the test lab and we can then make adjustments to the test lab's queue and workload, if needed. The idea of leveling out the flow of the system like this can be traced back to the Japanese term, Mura. The Wikipedia entry says it all:

"The fact that there is one operator will force a smoothness across the operations because the workpiece flows with the operator."

In this case, we are calling the combination of production code, unit tests and acceptance tests, the "workpiece". I believe this is a fair assesment, since the code and tests are all going to be based on a feature, use case, or user story. In fact, I would say that the workpeice actually is the feature, use case or user story that is being worked on. The code, unit tests and acceptance tests could be considered the artifacts products by the workpeice flowing through the system. ... but that's just splitting hairs, really.

It's all about Occam's razor, Parsimony, KISS, or whatever you want to call it - the simple solution is often the correct solution (simple, however, doesn't always mean easy).

The Need for the Test Lab

I'm certainly not saying we don't need a test lab. The testers are (supposed to be) experts in interaction testing, usability testing, and adding that "human touch". We absolutely need that perspective on those aspects of software testing that can't reasonably be automated. I am advocating that we find a better way to smooth the flow of the system - rather than apply complex theories and equations to the situation, find a solution that doesn't require anything complex.

Conclusions

In the end, the problem of the test lab bottleneck can be solved many different ways. You might level the system via Pair +1 Programming or some other form of involving the developers in writing the automated acceptance tests. Perhaps you make the testers part of the team and have them writing the automated tests at the same time as the developers writing code. You might still need to employ queuing theory. Either way, try to find the solution that works best to smooth out the process for your team.

A while back, I posted a stream of thoughts concerning programming in triples. The basic idea is that we have 2 developers doing pair programming and a third developer doing automated acceptance test development, on the same story at the same time.

Triples Programming Terms

I've been discussing this idea with various people since then, and I'm finding myself referring to this team organization with a specific term, now:

Pair+1 Programming

I don't know if this term has been used previously, or if there is already a term for this. However, I find this to be a good name for the organization that I am talking about.  It really is pair programming, as defined by many common sources, plus one acceptance test developer. The really fun part is that all three of the developers in this unit will switch roles throughout the day. The end result is that we have a very well cross-trained team - each member of each unit will be responsible for writing unit tests, writing production code, writing acceptance tests, and thinking ahead of the current code, throughout any given day.

An Agile Team includes more than just the developers - the customers, the test lead, the technical writers, the project manager, etc., are all part of the team. Pair+1 Programming is not into teams. A "team" implies that they the persons involved are fairly static - they don't move between teams, etc. In order to distinguish the Pair+1 organization from the Team as a whole, I am also going to refer to this group with another name:

Development Unit (Update: based on further discussion, renaming this)

WorkCell

Just as a developer will rotate rolls within the Unit, I believe Pair+1 programmers should move between WorkCells on a regular basis. This will further the cross-training that occurs and help to pollinate the knowledge of one WorkCell into the thoughts and goals of other WorkCells. The migration is likely to be less volatile than the role switching that happens within a WorkCell. For example, developers rotate positions every 2 or 3 hours, within the WorkCell, whereas a developer may only rotate between WorkCells once a day or once a week.

Development Systems

Within each Development Unit, there will be two development systems.

1. The primary system of that Unit will be the pair programming system. This system should have a very large monitor - 30" is preferable - to allow the Pair to easy see everything on the screen.
2. The secondary system will be used for Acceptance Tests and is likely to only need a 22" or 24" monitor, as it will be primarily used by a single developer.

Each of these systems, aside from monitor size, should be a mirror of each other - the same system specs with the same software development packages installed. Every machine used for development should be able to execute any of the tests - unit tests or acceptance tests - no matter the primary use of that development machine. This is required so that the Pair working on the Development machine can execute the Acceptance Tests being created on the Acceptance Test machine; thus ensuring that they are coding appropriately, and passing the expected tests. The Acceptance Test developer should also be able to execute the code being written by the Pair, to ensure that the Acceptance Tests are being written against the API correctly.

Following standard Agile practices, these machines should not be concerned with Email, IM, or other non-development tasks. They should be dedicated entirely to development work. If a person needs Email, IM, etc. then the office space should provide access via shared systems and/or a developer should have their own system (likely a laptop) that they can take with them wherever they go. The point is, though, to remove distraction from the Development Unit. Keep the non-development software off the Unit's systems, as much as possible, to ensure that the Unit can focus on development and not be distracted by 50 different flashing icons and windows on the machine.

Disclaimer

A lot of my thoughts are purely academic, at this point. I am hopefully going to have some actual experience with this environment, soon. As such, I am trying to organize my thoughts so that I can have some direction to start with. I reserve the right to be wrong and change my mind whenever I want, likely re-defining the terms and processes involved in Pair+1 Programming.