Warning: This post is about a beta product. Details might change before final release.
The beta of BizTalk Server 2009 brings a welcome new feature: Some basic support for Unit Testing BizTalk artifacts; specifically for Pipelines, Maps and Schemas.
You can read the basics about how the new feature works in the product documentation; there’s not too much yet, but it’s enough to get a good idea of how you use it.
I’d like to now share a few thoughts on this feature. Note: This is just my personal opinion from playing with the API for a bit.
The Good
The first really good part about this feature is the fact that it even exists at all. Facilities for creating and automating tests of BizTalk artifacts has been a sorely lacking area of the product.
While Visual Studio had the Test option for maps and schemas since BizTalk 2004, and it was sort of useful, it required way too much manual intervention and there was no way to decently drive it through automation, making it useless for either unit or regression testing.
The second thing I like about the unit testing feature is that, in it’s current incarnation, it’s not intrinsically tied to MSTest (i.e. the Visual Studio Team System stuff). This means you can use it with NUnit, xUnit, or your favorite testing tool.
The third interesting thing about the API is that it is a very simple API. In most cases it really consists of just a single method call. Here’s an example of testing a schema:
string fin = @"c:\temp\test1_in.xml";
Schema1 schema = new Schema1();
Assert.IsTrue(schema.ValidateInstance(fin, OutputInstanceType.XML));
The Not-so-Good
Not everything is rosy in BizTalk-land. Personally, I was somewhat underwhelmed by the implementation of the Unit Testing features available in beta 1; particularly since some of it could be addressed with fairly minor adjustments.
API Issues
Let’s start with the API: I mentioned before that the new testing API was simple. However, it’s actually too simple. For example, consider the small test for a schema I presented above: You simply create an input file to test with, call ValidateInstance() and check the result. Great, no?
Well, if the test succeeds, then yes, it’s just fine. If it fails, however, because the input file does not pass validation, then, it turns out, it’s missing one extremely important functionality: There’s no way to get any details about why the schema validation failed [1].
This means that diagnosing a failed test involves, at the very least, changing your project settings in VS to use the failed input file as input to the schema in VS, use the Validate Instance option in the IDE and examine the results in the output window, where fortunately you do get more detailed error information.
This is extremely sub-optimal and requires way too much manual work. Can you imagine if a change in a schema broke tens of tests and having to do this for each and every single one of those? Ouch!
Another place where I feel the current API is not as nice as it could be is that it’s strictly file-based. That is, you need to create actual files on disk to represent inputs to the tests and store the resulting outputs. This is somewhat convenient for many scenarios, and matches the existing functionality in BizTalk pretty well. It does, however, make it very inconvenient to deal with dynamically generated inputs or inputs stored someplace else.
The classic example here, would be, resources. For example, I very much like to keep my unit tests as independent from the environment and as self-hosting as possible. One very nice way of doing this is to store necessary input files as embedded resources on my testing assembly. I do this all the time with PipelineTesting and it works great. However, since you can’t provide streams instead of file paths to the BizTalk 2009 testing features, you’d need to first save the embedded resource to a temporary file, which adds a bit of friction to the process.
Compilation Changes
My biggest gripe with the Unit Testing features, is that enabling them implies a change in the build process, that changes the generated code for the BizTalk artifacts themselves.
When you enable the Unit Testing option in Project –> Properties –> Deployment, the compilation process will change the generated CLR types so that they inherit from the TestableXXX classes defined in the Microsoft.BizTalk.TestTools.dll assembly, instead of the normal BizTalk artifact classes.
These new Testable base classes, in turn, derive from the original BizTalk classes, so the change isn’t all that big. However, it does smell wrong. Could it introduce any problems/bugs if you were to enable the testing features for all builds? Hopefully no, but no way to know until you run into them.
Of course, you could just enable Unit Testing for debug builds, but then you can’t run your unit tests on the release builds of your BizTalk assemblies. It also introduces yet another asymmetry between development and deployment builds which, to be honest, makes me somewhat nervous.
Conclusion
The new Unit Testing features in BizTalk Server 2009 are a sorely needed and welcome feature, if somewhat lacking in the first beta.
Will it improve? On one hand, I’m somewhat optimistic that some improvements might make an appearance in forthcoming builds. However, I’m also not getting my hopes up, as MS is notable for trying to avoid significant API changes during after beta 1 hits the street (BizTalk, unfortunately, doesn’t use the CTP model, where significant API change requests are more likely to be considered).
[1] TestableMap.TestMap() does throw exceptions on failure instead of a Boolean return value, but said exceptions might not contain detailed error information.