Sunday, December 14, 2008

A Look Back at Our Scrum implementation

I've been working at my current project for one year now and we are 'nearly done' with our first major release. I know that saying 'nearly done' has a little bit of a bad smell in the agile community but we couldn't do everything perfectly. But generally speaking, we did it pretty well and I'm happy how we could eliminate and master the difficulties of the project with scrum.

We had the following situation at the beginning of the project:
  • A list of requirements in an excel sheet. The list summarized all the thoughts, requirements and promises in an informal way.
  • There was a file containing drawings, print-outs from the former legacy system, or other artefacts for each requirement.
  • The project was embedded in a much larger project, which was driven by Hermes. Hermes is a heavy weight project management process, commonly used by governmental organisations in Switzerland.
  • Our team consisting out of 6 software developers was hired to implement the software. We proposed to implement scrum and the customer agreed on that.
  • A contract with a fixed price that was signed a long time before the developers got involved

It was not easy to push scrum from the development into the larger project organisation. There was a good support for our ideas as long there was no organizational change required. Fortunately we could find a pretty good product owner, but we never have had a scrum master. The role of the scrum master was poorly fulfilled by me as a developer.

What worked well?

A lot of things worked great and I'm sure that the project would have failed without an agile approach. Here is what has worked well (see glossary for details on scrum terminology):

  • User-Stories: After we had transformed the informal requirement list into user stories it was relatively easy to estimate it without knowing the details of each story. Even they were written by us developers it was better than working with the initial list.
  • Productbacklog: It gave us a very important planning tool, helping us to focus on the near future and driving the development sprint by sprint.
  • Sprints: The sprints helped us a lot to get the team and the customer focused on the tasks that needs to be done. Without focusing on the sprints and forgetting a little bit about the whole, we could never have done it.
  • Sprintplanning: Without this meeting we wouldn't have any chance to know what to implement in detail. The customer was surprised at the beginning that he had do discuss every detail of its initial requirement list again in the sprint planning.
  • Sprintreview: It was often the only opportunity to present our achievements and we always have had valuable feedback from the customer and the future users. Customer and users could get used to our development velocity and that took a lot of pressure away.
  • Adoption: The product owner discovered during the first two sprints that he needs to be better prepared for the sprint meetings. He started to consolidate the different opinions in additional meetings prior to the sprint planning.
  • Team forming: Event just two out of six developers have worked together before we never had major problems.

What were the problems?

  • Scrummaster / Productowner: The roles were never officially assigned to persons, because the customer was confused in relation to the existing roles and responsibilities (project manager, solution architect and so on). There was also no budget available for 'non-programming' tasks.
  • Importance of the Productbacklog: The Productbacklog was built by the developers and was never owned by the product owner! Once it was initially built up it was used by the management to plan the whole year front up! Development was driven by the Productbacklog, but not testing, acceptance and fulfilment of the contract.
  • Testing: We couldn’t get things DONE. As the customer didn't test we started to do more functional and system testing. But after one year we do not have any story officially accepted by the customer. But due to the sprint-reviews I'm pretty confident that the customers will accept it after some minor changes.
  • Sprint-Retrospectives: the meeting was always held by the team but not attended by the product owner, nor attended by another representative of the management. Therefore just team internal impediments could be addressed and resolved. No organizational impediments were resolved.
  • Meetings: Due to the fact that the whole process was not coached by a scrum master, the meetings usually were ineffective and lasted too long. I, as developer and interim scrum master, was too involved to act as a coach for both the team and the customer.
  • Responsibility: There were few developers in our team that couldn't follow the process at all. They were not able to catch up the information at the sprint planning nor did they take the responsibility to fulfil the tasks they assigned themselves. We lost a lot of our performance here as senior team members had to coach them. Those weaker developers could not catch up as every thing went far too fast for them.

Conclusion
I'm still saying that agile development is the best practice I know. In my career as a software developer I sat too many times in my office trying to imagine how I could fulfil some requirements that I didn't understand. Today we have a simple process that brings customer and team together and that's much better for everybody. I'm thinking agile development is a process for people that want to take responsibility. If you don't have people that take responsibility you might be better to follow more heavy weight processes were every thing is predetermined and fixed. But everything you can't foresee will cost you a lot of money! It will cost you much more than the salaries of a good scrum master and good developers.

Thursday, October 16, 2008

Reflecting about how we implement aggregates

How we implement aggregates

Since the beginning of my current project we tried to apply the ideas of structuring our business-logic into aggregates. We started to implement the aggregates using the following guidelines:

  • define an aggregate root and implement it as a public class
  • define the internal classes of an aggregate and implement it as a public class
  • implement a repository that can query and return aggregate roots of this class
  • try to avoid accessing the internals (like traversing from aggregate root to an internal class) from outside the aggregate
  • put as much as possible of the business-logic into the aggregate root

Following this rules we had some sort of guidelines about where to locate the business-logic and when to implement a repository. I think it was worth to have the aggregates. It was a valuable concept when we had to change our application from a 2-tier to a 3-tier architecture.

Nevertheless we always had following problems with aggregates:

  • Most of the time we had code in the presentation-layer that accessed the internals of an aggregate. It appeared often in case where there was an one-to-one data binding. For instance, in a master-detail view, where the master was the aggregate root an the detail an internal entity of the aggregate. (e.g. editing invoice and invoice position)
  • There were relations on the data model that do not comply the aggregate rules. In such a case we often had a nice definition of an aggregate, however there was sometimes a nasty dependency. It happend that an internal entity was depending on another entity in another aggregate. We agreed that this is not bad and that it can be abstracted to a dependency between the two involved aggregates.
And what happened:

  • Since the internals could be accessed, the business-logic begun to spread out into the presentation layer. It's impossible always to have the discipline to comply the aggregate rules and all of us violated it from time to time.
  • We didn't centralize the creational-logic. The test code is often a mess and it deals with setting-up the aggregates as well the internals of an aggregate.
  • A NDepend-analyse revealed that our business-logic assembly will be a serious maintenance problem ('Zone of pain')

Why is our business-logic in the zone of pain?

We generally get closer to the 'zone of pain' (see link for details), when a high number of types in an assembly:

  1. are more concrete than abstract (see Abstractness in the formula)
  2. are used a lot by other assemblies (see Ca in the formula)
  3. do not depend on other types (see Ce in the formula)

Since I can't see any value in implementing business-logic abstract (1.) and our business-logic already depends on a small number of types in other assemblies (2.) the only variable would be to reduce the number of types that are used by other assemblies (3.).

How could we do it different?

We could try to better encapsulate the internals of an aggregate avoiding the problems described above. That means, not allowing to access any internals by another type than the aggregate root. This increases the afferent coupling of the aggregate root, on the other hand eliminates the afferent coupling of the internals. The afferent coupling of the business logic assembly itself would then decrease.

Therefore I transformed one of our aggregates from something like that

// File Invoice.cs
public class Invoice
{
public EntitySet<InvoicePosition> Positions { get; set; }
}
public class InvoicePosition
{
}

to

// File Invoice.cs
public partial class Invoice
{
private EntitySet<Invoice.InvoicePosition> Positions { get; set; }
}
// File InvoicePosition.cs
public partial class Invoice
{
private class InvoicePosition
{
}
}
After completing the refactoring and changing some logic in the presentation layer I got the following code metrics:

  1. The assembly containing the business logic moved a little bit away from the zone of pain (from D=0.577 to 0.575). This is still bad, however the refactoring had a positive effect.
  2. The type-rank for the refactored aggregate root increased from 10.7 to 12. Ca (Afferent coupling) increased from 62 to 66. That means, it got a more important part of the business logic.
  3. The CC (cyclomatic complexity) for the aggregate root increased from 38 to 77, while CC decreased generally for the presentation layer (e.g. in one case from 26 to 16). That means that we put complexity into the business logic. It's a good thing since the complexity generally is easier to test and therefore to maintain in the business logic. It was also possible to eliminate duplicated business logic found in the presentation layer.

Testability and Unittests

As the internals are completely hidden from outside now I test the refactored aggregate with state-based black-box tests. I think that's a reasonable way to do it since the test setup do not depend anymore on the implementation details of an aggregate. I'm not sure if this should be a general testing strategy for aggregates. I think there are more complex cases, where white-box tests would still be needed. The following pictures shows on the left, the test-code before the refactoring and on the right, the test-code after:

Monday, July 14, 2008

TDD, Design and Testability

After being on holiday for a while and being pretty busy at work I want to talk about an upcoming issue at my current project. The issues are
  1. Presenter-Logic (Passiv View) is hard to test. It costs a lot of effort and it's difficult to be done right.
  2. There is code that is not worth being tested. We shouldn't test that code just to have the test coverage increased.

As a TDD advocate I'm trying to find out what's about those two issues above:

1.Passiv View is hard to test.
Let see an example:
public void EditDienstAbwesenheit()
{
var abwesenheit = View.SelectedDienstAbwesenheit; // get selected item from view
if (abwesenheit != null) // if user as selected something
{
IModalDialog dialog;
if(abwesenheit is SpitalAufenthalt) // decide what type of item is selected
{ // create a modal dialog for this item type
var presenter = _viewBuilder.Create(_unitOfWork,typeof(ISpitalaufenthaltbearbeitenDialog));
dialog = presenter.View;
}
else
{ // creat a modal dialog for the other item type
var presenter = _viewBuilder.Create(_unitOfWork,typeof(IAbwesenheitbearbeitenDialog));
dialog = presenter.View;
}
if (Shell.ShowInModalWorkspace(View, dialog) != DialogResult.OK) // display the modal dialog
{
_unitOfWork.Refresh(abwesenheit); // reload data the might have been modified
}
UpdateView(); // show the modification on the view
}
}


We all know that the more dependencies and responsibilities a method has the more is it hard to test. This is like that because each dependency needs a mock or stub and each responsibility needs a test-assertion to verify it. The example above has following responsibilities:
  1. Get the users selection
  2. Decide about the next dialog to be shown
  3. Create the dialog
  4. Open the dialog in the shell
  5. Reload to rollback modifications of the dialog
  6. Present the changes to the users

To accomplish that tasks we need:

  • a View to get the users input
  • a Model (called 'abwesenheit') that represent the users input
  • a ViewBuilder to create a Dialog
  • a Shell to display the Dialog
  • a UnitOfWork to rollback the modifications

So what? Is it a test-problem or a design-problem? I strongly believe that code that can't be tested is poorly designed. The most obvious is that we violate the Single Responsiblity Principle (SRP). Could we split the responsibilities to find methods or even classes to separate the different concerns?

Lets try:

  1. Get the users selection -> Lets keep it where it is
  2. Decide about the next dialog to be shown -> Lets put it into a separate method
  3. Create the dialog -> Lets delegate it to another object as creation and displaying a dialog is often used in an application (e.g Application controller)
  4. Open the dialog in the shell -> move it to the Application controller
  5. Reload to undo modifications -> That's wrong here. The dialogs are responsible not to modify data in case they are cancelled!
  6. Present the changes to the users -> Lets keep it where it is

And after that minor refactoring:

public void EditDienstAbwesenheit()
{
var abwesenheit = View.SelectedDienstAbwesenheit;
if (abwesenheit != null)
{
CreateEditDialog(abwesenheit);
UpdateView();
}
}
public void CreateEditDialog(DienstAbwesenheit abwesenheit)
{
if(abwesenheit is SpitalAufenthalt)
{
_applicationController.ShowInModalWorkSpace(_unitOfWork,typeof(ISpitalAufenthaltBearbeitenDialog));
}
else
{
_applicationController.ShowInModalWorkSpace(_unitOfWork,typeof(IAbwesenheitBearbeitenDialog));
}
}

So is that better? I think yes. In terms of Unit-Testing we reduced responsibility and dependency per unit (method) and we got a cleaner more expressive design.

2. There is code that is not worth being tested
Lets see following example:
public class Person
{
public string FamilyName { get; set; }
public string Name { get; set; }
public string Age { get; set; }
}
public class Address
{
public string Code { get; set; }
public string Country { get; set; }
public string Street { get; set; }
}
/// <summary>
/// Model
/// </summary>
public class PersonRow
{
public PersonRow(Address address, Person person)
{
_address = address;
_person = person;
}

private readonly Person _person;
private readonly Address _address;

public string FamilyName
{
get { return _person.FamilyName; }
}

public string Name
{
get { return _person.Name; }
}

public string Age
{
get { return _person.Age; }
}

public string Code
{
get { return _address.Code; }
}

public string Country
{
get { return _address.Country; }
}

public string Street
{
get { return _address.Street; }
}
}


Is it worth to test that each getter returns the right value? When is it worth to test a piece of code? I could think of following answers:

  1. It's worth if something can go wrong
  2. It's worth if I don't need to put a lot of effort to test it
  3. It's worth if it's likely to be refactored in future

The problem with the example above is that little can go wrong (except the mapping) and that the effort is too high to test it. On the other hand as a TDD advocate I'm a little bit annoyed that this little stupid class should lower our test coverage. So what is the way out?

  • If this class were not necessary there shouldn't go anything wrong
  • If we could reduce the lines of code, less could go wrong and less needs to be tested

So let's try to introduce a mapper to externalize the mapping:

public class Person
{
public string FamilyName { get; set; }
public string Name { get; set; }
public string Age { get; set; }
}
public class Address
{
public string Code { get; set; }
public string Country { get; set; }
public string Street { get; set; }
}
/// <summary>
/// Model
/// </summary>
public class PersonRow
{
public string PersonFamilyName { get; set; }
public string PersonName { get; set; }
public string PersonAge { get; set; }
public string AddressCode { get; set; }
public string AddressCountry { get; set; }
public string AssressStreet { get; set; }
}

public void DoTheMapping()
{
var person = new Person();
var address = new Address();
var row = new PersonRow();
PropertyMapper.To(row).From(person).From(address).Execute();
}

The idea is to eliminate all the boring mappings to a specialized class PropertyMapper. The PropertyMapper would map the values from person and address to the row combining Classname and Propertyname. Furthermore the method Execute() would throw an exception in case any property of the PersonRow was not initialized with a value. My observations are:
  1. The PropertyMapper is difficult to implement, but I think after 10 mappings i have already my return on investment
  2. Execute() is self-validating, therefore this code can also be run under test
  3. I split the responsibilities into two classes (PersonRow and PropertyMapper) and I got testability
  4. The example should be extended to support two way mappings
  5. Is that too much?

Friday, March 21, 2008

Some thoughts about C# 3.0

We have been using the new C#3.0-features for two months and its quite interesting that our code looks in some parts like Jeremy describes in C#3.0 is like a moped. I wouldn't like to miss the new features and I won't stop using them but I'm thinking it's getting too much.

It makes me feel that C# already had it's best time and there should be something new. Where is the new microsoft-ruby-language? C# was a better Java (I mean the language not the community) but where is now the better ruby?

Sunday, January 13, 2008

Behind the scences of LINQ

I always knew that LINQ was founded on the idea of monads but even trying hard I couldn't understand how LINQ and monads are related to each other.

But fortunately there is Wes Deyer and he describes in The Marvels of Monads what I always wanted to know. It's a pity that LINQ was always described and documented as query language rather than as a monad extensions. When I look to the msdn-documentation there is no hint that I could implement query operators on other types than IEnumerable and IQueryable.

But that's exactly what Wes is doing. He sees a more general concept and he shows a way to implement the query operators on any type. In fact this is for me a huge step and leads us to a new level of abstraction where LINQ (as a list-monad) seems to be a specific case of a monad.

My preferred way to use LINQ was the method syntax as I thought this is the more general case. But now, after knowing that the query syntax is something like a monad extension I start to like it.