In my previous entry about Morts, Nick Malik left an interesting response. What caught my eye the most was his comment that:

"If you find yourself having to maintain a program that was not written to be maintained, don't blame the maker of the tools!"

I agree with this sentiment to a large degree. How people use the tools, is critical. But, that's only part of the story. See, a lot of tools don't lend themselves towards wiring maintainable code. Some of Microsoft's current tools fall in that category, unfortunately. For example, some don't lend themselves towards creating testable code. That's a big problem all by itself.

So I'd like to take Nick's comment and turn it an opportunity. Sure, it's the people using the tool that take most of the blame, however, the maker of the tool may (should!) have a significant responsibility in creating tools that encourage the creation of maintainable code. (Notice I use the word encourage here).

There's a thin, but significant line between tools that allow you to create crappy code quickly and then leave you high and dry, versus tools that encourage you towards creating good code. The key here is that the tools should make it as painless and natural as possible to do the right thing. This is not about wizards. It's not about visual designers. It's about code.

Microsoft is in a unique position to provide this, but so far some teams within it insist on making the default way of creating applications to be the one that favors quick and dirty over the one that favors maintainable code. The latter does not need to take longer, be more complicated, or be harder than the former, but some tools insist on making it that way. Choosing the right abstractions, by the way, is a key step here.

Microsoft, Morts

There are some discussions going on at this time on the community around the topic of
writing maintainable code and the different trade-offs a given "project reality"
might force on the developers involved in it. Very much worth reading and highly
recommended everyone reads and participates. Jeremy Miller beat me to it with some
good comments, as well.

My own personal opinion on the matter is a bit of a mixed bag (and so at times I might
seem to contradict myself).  On one hand, I find myself in total agreement with Ayende and Sam (just to name some of the most vocal proponents) that raising the level and knowledge of the developers that are going to be receiving the application is a critical step. I am a strong believer, learned from my own personal experience, that there's no amount of tooling and supervision/control that can "fix" the deficiencies of the team members themselves. It just simply doesn't cut it.

I believe says it best when he says that "Because my view is that having the developers learn a better way to build software is much less costly than continuing to produce software that is hard to maintain". I couldn't have said it better.

I am, however, sensitive to the dilemmas brought up by JDN and Frans Bouma, both of which have articulated their concerns and point of view in very provocative and thoughtful ways. I'm sensitive because I've found myself in their position a few times, and I've made the same choice they've argued in some of those occasions. However, my experience also shows that making that trade off, at those times I've made it, has eventually shown to have been a less than optimal decision.

Given this, I'm firmly in the camp that a) you should strive to avoid making that choice if at all possible, and b) you've got to be pragmatic about it. Sometimes, yes, organizational dynamics and politics play a role here, even if we disagree with it. So sometimes that might be the right choice to make, but rarely (if ever) for purely
technical reasons. And the whole "other developers might not get it" is, in the end, a red herring and tends to obscure the real reasons and motives behind it.

Why is it a red herring? Because it is usually a sign of the "we want to have a few highly skilled developers and lots of unskilled, cheap developers for the 'easy' stuff" mentality that pervades a lot of the companies in our profession, and it's just one I can't agree with. It's not that everyone should have 10 years of experience building extremely complex applications, but rather that everyone should have the opportunity to do their work to their best of their abilities. Inevitably, I find that this sort of mentality tends to set bounds that developers can't later cross easily without a lot of push back from management and the organization, leaving unskilled developers without a path they can collaborate with their more skilled, more experienced peers to grow their own skill set.

In many ways, this is a topic that strikes close to home for me, for several reasons. First of all, this kind of discussion is one of the reasons I decided to quit my last job and start my own company. In my point of view, the whole concept that the people are the key ingredient that make/break a project is absolute key. It's not about how much experience and knowledge the developers have. This is a significant component, for sure, but one that can be acquired with time. What is absolute key is the attitude: 

• A willingness to constantly learn and look for new opportunities for personal/team/product improvement. 
• A commitment to satisfying the client/costumer/user.
• A commitment to being ethical, honest and professional about your work and your own capabilities and limitations.
• Awareness of your context and that of your project/product.
• Caring about what your doing and being enthusiastic about it.
• Being critical about your own work and those of others, even if your believe others are above you or have different responsibilities (say, project manager or architect). Note this is not about constantly criticizing other people's work; it's about thinking critically about what the work that's being done by the team as a whole and the results.

I've seen many developers that lack one of this. Some are just 9-5 guys [1] that just want to get through the day without caring about the end result. Others have no desire for learning new things or getting out of their comfort zone. Others simply don't care about what other people do or what they do, usually manifested by them saying they're done implementing something and when you go look it doesn't even compile. Others will just do what their told to do and never even stop considering if it makes sense or if indeed it is the best way to do it.

Like Ayende, I refuse to be just a replaceable cog in a big machine; I want to make a difference and continuously learn and grow my skill set. I wouldn't have it any other way.

So it's about lots of different things. I love working with people that are critical about my work; they make me be a far better developer, and by listening to their opinions and sometimes different alternatives they propose I myself learn a lot. Sure, it's sometimes hard to listen to it all, but it's much worth it.

Arguing about the right things
I think one aspect that has tended to mud a bit the discussion is that we put a lot of different things into the same bag, which makes everything more confusing. For example, some people have mentioned MVP/MVC, ORM and TDD as important aspects in achieving maintainability, which some people have reacted a bit against.

From my point of view, the problem here is that we're mixing practices with tools. For example, TDD is a practice, and one that indeed is (imho) very important in facilitating the creation of maintainable code. But by itself it is no silver bullet (my friend Sam, for example, always talks about using the full set of 12 practices proposed by eXtreme Programming, not just TDD).

On the other hand, ORM and MVP/MVC are tools (yes, patterns are just design/coding tools, in the end), and sure, they can significantly help in producing maintainable code. But they are just tools, and you have to evaluate their impact and usefulness in each case. Some projects might simply not benefit from using ORM
(for example, this is rarely a useful component in the integration project's I've worked on), while it might reduce the amount of hand written code in your typical business application (and less code will always mean more maintainable code). But others might prefer for certain applications the use of code generation instead of an ORM and, you know what? That might be a perfectly valid choice in the context they're in. What's key here is
that they've done the analysis and choose what they believe it's best for them given their context.

Note: It's perfectly possible that the choice still turns out to have been the wrong one in the end, given that many times you start out with limited information from which to make your decision; but that's life.

It's about the future

One key issue I've seen again and again is that many developers don't even care about writing maintainable software, simply because they're not the ones that will have to maintain it. It's a very pervasive mentality in some circles that has been brought, in part, by the outsourcing of software development (something I make my living off, by the way). For example, in Colombia, a lot of companies have no internal software developers anymore, so external providers come in, create the initial release of an application and go away, many times never to see that code again except for one or two fixes during stabilization.

This means many developers don't really learn about the value of writing maintainable software because they it's easier to justify to yourself the writing of poor code when you think "I'm not going to maintain this later on, so why should I care?". This is not only a very egotistical point of view (and easy to fall on) but also ethically challenged. But what's worse is that it is a complete fallacy: you're still maintaining your code, and every day you write poor code in a project it's more code you have to maintain. It's just a lot of people seem to think otherwise and lie to themselves as a defense mechanism against the pain their going though every day on their project.

[1] there's a difference between wanting to have a life outside of work (good) and not giving a shit (bad).

Software Development, Maintainablity

My good friend Sam Gentile pointed to me Nick Malik's post on "Tools for Mort", in which Nick responds to some comments made by Sam a few days ago. I don't particularly agree with Nick's point of view. Several things bothered me about this post.

Why are we talking about Mort again?

First of all, I've said in the past that I don't particularly care in the Mort/Elvis/Einstein personas. Besides the fact that they tend to split users/developers around what I believe are the wrong areas, they are way too misinterpreted all around. Just bringing up the whole Mort topic is a sure way to turn a discussion in an unproductive direction.

What's worse, Nick apparently thinks Morts are the great Agile developers fighting the corporate world. Frankly, I have no clue where he got that idea from, but what do I know.

Look, it's great that there are people leveraging office and other platforms to solve business problems and get work done. There's a lot of really good use cases for that technology. However, it doesn't mean all software projects should be solved that way. In some scenarios, these kind of tools are great for solving immediate issues, but many times they are not suitable for building long-lasting, scalable solutions. Both sides of the software spectrum have different needs and deserve different solutions.

However, let's be clear about why the term Mort got a bad reputation: It was typically applied to people who didn't quite understand the tools/environment they were working on, creating half-assed applications that while did the trick initially, were a living hell to maintain because they didn't respect the most basic principles of good software development (in many cases, were not even aware they existed). Hey Nick, let's be honest here: your "Mort" friend was definitely no Mort if he was doing continuous integration and automated unit testing [1].

Sam doesn't get Agile?

Nick goes on to say:

"Do you really mean that Microsoft should make a priority of serving top-down project managers who believe in BDUF by including big modeling tools in Visual Studio, because the MDD people are more uber-geeky than most of us will ever be?  I hate to point this out, Sam, but Alpha Geeks are not the ones using TDD.  It's the Morts of the programming world.  Alpha geeks are using Domain Specific Languages."

From this sentence alone it's clear to me that Nick is obviously not aware of Sam's extensive and continuous efforts talking about "agile architects" and railing against BDFU. Really, you might want to look a bit around before making those kind of comments.

Now, Morts are the ones doing TDD? Wow. color me surprised. Alpha Geeks use DSLs? Really? AND THEY DO NOT DO TEST DRIVEN DEVELOPMENT? WOW.

Hummm...not quite. Most of the so called Morts have no freaking clue what Test Driven Development. People like the Code Better guys, Ayende and a lot others are precisely doing that: Trying to raise the level of awareness in the community about TDD/DDD and improving the way they write software. They are the ones trying to improve our field by making developers more aware of how they work and get them to write better software that not only works and solves the business problem it is intended to solve, but that is also maintainable. At least grant them that much.

I think Nick is also a bit confused about the whole Alpha Geek business. A lot of them do use dynamic languages and yes, a lot of them use them to create domain specific languages. But what makes you think they don't need TDD? Quite the contrary, Nick, they are far more reliant on TDD and automated unit testing.

Nick then says that "We [Microsoft] are the singlehanded leaders in the space of bringing BASIC developers up to modern computing languages.". Humm.... certainly true, you moved them over to VB.NET. It's certainly possible to write Object Oriented code in VB.NET. That, however, doesn't mean they are. Let's face it, there are millions of developers out there using VB.NET and (yes) C# that are still writing essentially procedural code (if they are lucky). Just because you declare classes doesn't mean you're doing OOD.

It's the tooling

But that's all mostly irrelevant to the discussion. I got the chance to speak about this with Sam and others during the past MVP summit, so I do think I have an idea what the fuzz is about (and because I happen to agree with the consensus). See, none of us are arguing that Microsoft has not done significant contributions to the community and even to the Software Developer profession, because they have.

What we do argue about is the tooling coming out of Microsoft and the way it is meant to be used. A lot of the software coming out of DevDiv these days seems to be very focused on building something quick. That's not a bad thing by itself, we all want software could be developed faster and that we could deliver it to the business as soon as possible.

What is a problem is that the tools enable that quick result at the expense of something else; something fundamentally important if you're not writing sample applications or "quick-n'-dirties". They are fundamentally non scalable (and I'm not talking about performance here), and unmaintainable if you just stick to the whole straight from the db databinding and drag and drop style of development Microsoft is so proud about. Better people than me have argued this more vehemently that I ever could so I'm not going to repeat it here.

I think that, at a fundamental level, there's a significant disconnect here. What some part of the community spends its efforts in is to try to raise the awareness, knowledge and capabilities of developers everywhere as a means. For them this is a key point in getting better software, with better and faster results. Instead of dumbing down the tools to suit the unwary developer, they are trying to smart up the developer to use the available tools more effectively.

[1] Somehow, I can't imagine Nick's Mort doing TDD while writing Excel macros, but maybe I'm just being cynic...

Microsoft, Agile, Morts

In one of his articles on building a CAB-like infrastructure, Jeremy Miller brought out a pretty interesting point: The choice between using Events/Delegates and Direct Method Calls. This can be a controversial topic at hand, but it's an interesting aspect, and can significantly affect the readability of the code.

Events and delegates are very powerful features I use all the time. However, here's a tip: Be careful about building an entire framework or complex structure based purely around delegate calls. I've been spending a lot of time the past few days tracing, debugging and running through the code in the Base Adapter sample code in the BizTalk Server 2004 SDK [1], which is exactly a little framework for building BizTalk Adapters.

Now, the base adapter is OK, in that it does make it a bit faster to develop adapters by taking care of a lot of the boilerplate code necessary. However, a significant part of it is built out of classes that use a lot of delegates (not even proper events, unfortunately) to notify of significant events around each other; like for example that a batch was submitted to BizTalk or that a message needs to be suspended.

The unfortunate side effect of this is that tracing the base adapter code in source code form requires a lot of patience and a lot of Find References/Find Definition jumps all around to figure out exactly what's going on. In fact, I've spent quite a few hours on it and the code can be significantly convoluted; and parts of it are just a lot easier to trace using Step Though/Step Into in the debugger; which is rather frustrating.

So next time you want to build a framework or library and think gluing all together with delegates and events is cool; make sure you've got a real use case for them and that they are not going to end up hindering readability and maintainability.

[1] The version included in the BizTalk 2006 SDK is different and a lot simpler. I still don't quite like it and generally prefer to write directly against the Adapter Framework APIs, but that's just me.

BizTalk, Base Adapter, Delegates

Nicholas Allen has been posting some really great entries on Poison Message handling in WCF and specifically the options the MSMQ binding offers for that. He also mentions that MSMQ v4 (Longhorn server?) will support some new features to make this easier: Retry Queues. Very interesting stuff.

One thing I wanted to comment on to hopefully complement a bit Nicholas' explanations is the relationship between poison messages and in-order processing.

The term "poison messages" might seem to relate to a kind of malicious attack; that is, something intentional. However, that's not necessarily true. A poison message is in essence anything that prevents you from continuing processing by failing repeatedly. This could be something unintentional but well intented, like receiving a malformed message from a trusted business partner that you still need to process.

If you don't need to guarantee in-order processing of the received messages, then dealing with poison messages can be dealt very efficiently in an asynchronous fashion like the one Nicholas describes: You move the problem message somewhere else, and continue processing the messages on the incoming queue; meanwhile you deal with the message, possibly fix it and eventually feed it back into the incoming queue for processing.

However, if you do need to guarantee in-order processing of received messages, this is not an option. If the poison message was a valid business message (like a malformed order from one of your big customers), you can't simply move the message and continue processing. Instead, the situation needs to be resolved in a synchronous fashion: you must inspect and resolve the problem message before you continue processing.

Normally, doing this requires some manual intervention from an operator or analyst, but it also requires your service to proactively deal with this situations. At the least, a responsible application will suspend all processing when it detects a poison message and raise alerts so that the operator knows a problem ocurrs. Notice that it will need to suspend processing, but it doesn't mean it needs to stop receiving messages altogether. Normally, you'll still receive messages and queue them for later processing.

The operator or analyst would then inspect the problem message and determine if it's not something important (or if it indeed was a malicious message) and discard it. If he/she determines it is a valid request message, he/she might need to fix it by hand or with appropriate tools (for example: remove an invalid character that was causing problems or change the message encoding) and then feed it back to the service. In the latter case, you need to provide a way to put the message back into the processing queue in the right position (i.e. at the beginning) and in either case you need to provide a way for the operator to ask the service to restart processing of received messages.

One possible way to support feeding messages to the service for immediate processing and not having to mess around with the incoming queues is to provide private alternative endpoints that bypass the normal receive queues. Fortunately, doing this with WCF is easy because you can expose a single service as multiple endpoints using different bindings, for example, expose an alternative HTTP-based endpoint besides the main MSMQ based one. In BizTalk, for example, one would do it by having an alternative receive location (for this purpose usually a FILE receive location works very well).

One thing to keep in mind is that it's important that these alternate endpoints are private only, are not advertised and are kept secure, as they could be used by rogue agents to disrupt your in-order processing mechanisms.

Windows Communication Foundation, WCF, Architecture