The Windows SDK Team blog has posted an open question about some potential features of future SDK releases and is gathering our opinions about it. I'd like to comment a bit on them and provide some feedback.

There are 9 areas the team is looking for feedback on. Let's consider each one:

1. A new, small download of “Core SDK” components is made available to customers, with only the basics.  What components should be included in this product in addition to the Windows headers and libraries?

I think the idea of offering smaller download options is a good one overall, and one that a lot of people (myself included) have asked for a number of times in the past. It's true that the special web downloader/installer we had in the past allowed you to download just parts of the SDK, but it just didn't work sometimes at all, and it was too rigorous in enforcing dependencies that might, or might have not, applied to your specific needs.

However, I disagree on what the contents of the "Core SDK" should be. To me, the most important part of the SDK nowadays is not the Windows headers/libraries, but the SDK tools. Hence, I want the minimum download option to be just the tools. The headers/libraries package should be built on top of the tools package, not the other way around (which is what has happened some times in the past).

My justification for this is that the SDK tools package today is very relevant to all SDK users; both .NET and Win32/64 developers. The headers/libraries package, however, is only relevant most of the time to the second camp.

2. PowerShell Build environment. The Windows SDK would include a script similar to SetEnv that users could run, which will set up an SDK build environment under PowerShell.

I've already said in the past I think this is an excellent idea, which I fully support. For now, I make do with my own custom version tailored to some particular needs of my dev environment.

3. Quick method to install “only .NET” or “only Win32” components.  For example, Win32 developers could quickly choose to receive only Win32 resources in the documents, tools, samples, etc.  Developers focused on managed code could choose to receive only .NET Framework resources in the documents, tools, samples, etc.

I guess this might be a good choice for many people, but to be honest, it's not one I'd particularly use much.

4. Improvements to documentation: better Table of Contents, better filters, search, etc.

Improved documentation is always welcome. The table of contents is the main entry point into it for a lot of people, so organizing it better would definitely be beneficial.

As for the rest, search and filtering really depends a lot on the underlying medium that hosts the documentation itself (the offline help viewer or the online MSDN site), so not sure what the SDK team can actually do to address those issues.

On the face of this, I'd say just focus on improving the content itself.

This does remind me of something: The current local help viewer MSDN uses (and many of the dev-side tools) is not something I'm particularly fan of. I was around when the original HtmlHelp engine was introduced and replaced the old Infoviewer that MSDN/Technet used, and was one of those that complained a lot about it.

I now long for the good old days of HtmlHelp. For example, I've seen the BizTalk SDK docs (which usually ship in the current help technology) in the old HtmlHelp format, and damn, it was way better/faster than what we have now. That sucks.

5. Improvements to documentation: integrate the SDK docs with MSDN docs.

We've had integration between the SDK docs into the MSDN docs in the past. In fact, I'm somewhat surprised it isn't now (I haven't installed either MSDN or the SDK docs in years!).

I used to think this was a good idea; I'm not so sure anymore. Two things come to mind:

• MSDN is already huge, and somewhat bloated. Adding the SDK docs on top isn't going to improve the situation. In fact, considering point 4 above, integrating them is going to make search and filtering them even harder/worse.
• The MSDN/SDK docs integration we had in the past had one significant flaw, in my humble opinion: It tried to integrate the SDK table of contents into separate sections in the MSDN TOC. So if you opened the standalone SDK help, you'd see all the SDK content grouped together, but if you launched the MSDN docs, there wasn't a single place in the TOC where the entire SDK content was hanged of. Annoying as hell.

The second point, in particular, brings up one question: Do we really want them integrated, or merely accessible from a single place?

6. Ship a new “Tools Explorer” to group the tools and provide a more friendly and efficient way for users to search and use the SDK tools.

Not interested. Keep the SDK lean and mean and get it back into shape.

7. Ship a new “Samples Explorer” to group the samples and provide a more friendly and efficient way for users to search and use the SDK samples.

Not interested on an offline tool for this. I think there might be some use for this as an online tool where I could look for a sample and quickly get the code just for that specific sample, instead of having to download the entire SDK samples package (which is huge) to find it.

8. Windows SDK in non-Visual studio IDEs. Provide additional support for other IDEs, such as Windows SDK integration with non-Microsoft development environments, links to Windows SDK documentation from within other IDEs (Eclipse, IBM VisualAge for C++, Borland’s C++ Builder), among other possible integration scenarios.

Haven't been using them at all, so I'm not qualified to really talk about it. I do believe that, for the most part, the SDK should be completely independent of VS and should be usable in a standalone fashion.

9. Create a new Download portal with SDK ‘nuggets’ so that you can download small packages – perhaps a popular tool or file that shipped broken or was missing from a released SDK.

Might be a good idea, but if you're going to do it; do it right and stick with it. The SDK has seen as many changes in installer/downloaders as name changes, and it's a drag.

Don't get me wrong, it's OK to change installers if they bring a significant improvement to the SDK user, but I've never really found that obvious during the several last installer changes. I'm sure each one might have improved something for the SDK team, but from where I'm sitting, it's just been an annoyance to me without bringing any benefits.

It does bring up the question of just what exactly the SDK needs an installer for, besides just registering environment variables and creating the start menu entries...

Windows SDK

If you're writing text files in an encoding that supports a Byte-Order Mark, you should always try to make sure that you include it, unless you have a protocol in place that precludes you from doing so (such as a legacy application that doesn't know how to deal with them).

One of the reasons you should always remember the BOM is that many applications can use it to try to guess what encoding they should use when trying to read the text you're feeding them.

Encoding detection based on the BOM is not foolproof, but it's better than having nothing at all, particularly in cases cases where simply assuming blindly a predefined encoding such as UTF-8 or UTF-16 might not be an option at all.

One particular case where remembering the BOM is very important is with UTF-8. Let me tell you a story to illustrate why:

We've been working the past few weeks on testing and improving a BizTalk-based solution for a client that some other consulting company had created. One particular piece had been working fine until, suddenly, we started getting an error when BizTalk tried to parse an incoming message with an error stating that "The character is not valid on the specified encoding".

Looking at the message, it was supposed to be UTF-8 encoded, and for the most part looked OK. The character causing trouble was, in fact, a 0xA0 char (non-breaking space) inside an element value. While this was not good, it wasn't clear why it was causing trouble.

Since it was an XML message, we opened it up in Internet Explorer: Yep, that too parsed it incorrectly and got stuck when it reached the problematic character.

Looking a bit further , we found that in this particular case, the original developer had written a piece of code that created a Stream object with the message contents and then fed that to BizTalk. The code looked a bit like this:

The message text itself was a piece of XML that included an <?xml?> declaration with the encoding attribute specifying UTF-8. This seemed OK, even if the code above just seemed like a pretty uncomfortable way of creating the stream.

However, this gave us a clue: UTF8Encoding.GetBytes() won't give you a BOM. Looking at the message in a binary editor, we validated indeed the message did not have a BOM at all. So we tried replacing the code above to one that simply used a StreamReader object (which uses UTF-8 with a BOM by default), and that fixed the issue right away!

This highlights why the BOM is so important for UTF-8: The basic characters in the set share the same values as the ASCII code. This is generally an advantage, but it can also mean that stuff that's incorrectly coded (such as our example above) might seem to work fine for a while until an unexpected character comes along and everything crumbles down. This is unlike other encoding, such as UTF-16, where things usually blow up right away.

In this particular case, the culprit was really a combination of factors: The lack of a BOM together with the presence of the encoding specification in the XML declaration [1]. I'm not sure why the XML stacks get stuck on a BOM-less UTF-8 file with an encoding declaration, but there you have it. So don't forget the BOM!

[1] I personally thing the encoding specification in the XML declaration is probably the single most stupid idea included in the XML spec. It's just downright evil.

XML, .NET, BizTalk

Yesterday I was working on some fairly complex code involving some LINQ to SQL, lots of generics (with some reflection generously sprinkled on top) and some extension methods to complete the cake.

Most was working as expected, until I ran into a little snag. I had one line of code looking a bit like this:

When I ran my NUnit tests, it appeared as if DoSomething() had never been executed. I knew that someCollection had some items in it, so I was pretty surprised. I fired up a debugger, put a breakpoint in the first line in of DoSomething and execute the code. No dice, the breakpoint wasn't even hit.

So I went one step further and put a breakpoint just at the call to DoSomething. The breakpoint got hit, pressed F11 (step into) and.... it went right to the next line without stepping into the method. What was up with that?

After scratching my head a lot, I realized it had been me being clever that caused the problem. See, DoSomething was basically a map of sorts, defined somewhat like this:

Can you spot the problem?

I had created the method so that I could reuse the transformed object stream after the mapping, just in case I needed it. However, on this particular use I was making of the method I was disregarding the return and was simply interested in getting the internal function called for each item in the collection.

This turned out to be a really bad idea, because the way the C# compiler generates the code for this is pretty smart: It's completely pipelined/streamed, such that you need to consume the returned IEnumerable<T> object. If you don't, then the collection provider never gets iterated on, and thus, nothing happened.

In hindsight, this should've been pretty obvious, and it's a very natural behavior, if you think about it. I should've been able to pick it up sooner, but I admit the debugger behavior surprised me a bit.

LINQ, C#, .NET, Generics

I've been spending some time this week coding some changes to a custom PowerShell PSHost for an application. One of the changes I wanted to experiment with was changing the code that executed commands so that it used Pipeline.InvokeAsync() instead of Pipeline.Invoke().

There are a couple of things that need to be handled different in this case: How you process the results from the pipeline and how you handle errors. I'll concentrate on the first one, as it is the one that caused me a bit of trouble to get right.

To process the results from an asynchronous pipeline invocation, you need to use a PipelineReader<PSObject> object, which is what the Pipeline.Output property returns. This allows you to read objects generated by the pipeline execution as they are coming out (i.e. as soon as they are available) instead of waiting until the entire pipeline has executed to grab the results, so the idea is pretty nifty.

Unfortunately, the documentation on how to use this object correctly isn't very good. For example, you can't rely on the Count or IsOpen properties as boundary checks to detect how many items to attempt to read. In particular, the Count property isn't reliable if you're using Pipeline.InvokeAsync() because it only represents how many objects are currently available in the reader, not the total count of objects returned by the pipeline (this is natural once you realize it, but still).

Instead, you should really rely on the EndOfPipeline property of PipelineReader<T> to detect when you've reached the end of the object stream generated by the pipeline execution.

The second issue that's not very obvious is that when you use Pipeline.Invoke(), but you don't need to feed inputs to the command, then the pipeline won't really start executing until you close the PipelineWriter object returned by Pipeline.Input. If you don't do this, then PipelineReader.Read() will simply block forever.

Phantom Objects

The one nasty issue I did run is what appears to be a synchronization issue inside PipelineReader<T> itself. In my original attempt to use Pipeline.InvokeAsync(), I started getting some weird results: Ghost objects were coming out of the reader.

Ghost objects?

Pretty much, yes. Let's say I executed an "ls" command on my pipeline that should return 8 items. Sometimes, I'd indeed get the expected 8 items out of the pipeline before EndOfPipeline changed to true. Other times, however, I'd see 9 items come out of it.

The last item was a "ghost" object that was empty: a PSCustomObject with no properties at all. Where was it coming from?

The only good thing about this was that if it appeared at all, it always did it as the last place in the pipeline. This gave me a clue: Could this be a marker object inserted internally by PowerShell into the object stream to mark the end of the pipeline? It sure looked like some kind of null value.

. It is, in fact, AutomationNull.Value, which, although defined in System.Management.Automation.Internal, is a public type/property.

The reason I say this problem is a synchronization issue is that, for the user of PipelineReader<T>, the use of this marker object should've been transparent. Instead, it is a leaking abstraction that sometimes (and just sometimes!) gets exposed and returned from PipelineReader.Read() when it should never happen!

In the end, I ended up rewriting my code like this to work around this problem:

PowerShell, .NET

A few people have asked already about my PowerShell script for configuring a development environment for .NET / Visual Studio / SDK work, so I thought I might as well break it into it's own script.

Here it is:

Feel free to customize as you see fit .

PowerShell, Visual Studio, .NET