Something I ran into yesterday: There are two variants to Ast.ComplexCallHelper() in the DLR: One takes a MethodInfo and the other a Delegate as the invocation target.

The MethodInfo variant does not check for the presence of a CodeContext argument, while the Delegate version handles this correctly. Not quite sure why the discrepancy between the two, but sure looks like using the Delegate variant is a better choice for now.

Dynamic Language Runtime

Here's one interesting issue I ran into while trying a few things out: Conversions.

Given the dynamic nature of the DLR and the languages implemented on top of it, and the statically typed nature of the CLR and most .NET languages, you need to be able to ensure that the right type conversions happen at the right places.

This involves, among other things, ensuring that the right casts and boxing/unboxing operations happen, as well as ensuring that language-specific conversions can be applied (such as, say, converting an arbitrary object to a CLR string through an explicit method call).

The DLR does a bunch of these automatically in a number of places, like variable writes and assignments, call return values and so on. Most of these can be made to follow language-specific behaviors because the conversion expressions are generated through calls to your language ActionBinder implementation (like the RubyBinder class in the case of IronRuby).

For example, many of these scenarios will generate calls to your ActionBinder.ConvertExpression() method. Here's one case I found where this does not happen, making it a bit more tedious:

Today I was debugging some code because of an InvalidCastException getting thrown. Indeed, the problem was that I had a method call that required converting one of the parameters, and it was a set of types I had not tried before. In this case, the necessary conversion was language specific, so I added the rule into my ActionBinder.ConvertExpression() implementation and made sure as well that ActionBinder.CanConvertFrom() handled the case as well.

But it was still not working. After a bit of debugging, I realized that in this particular case, the ActionBinder wasn't getting involved at all for the conversion! What was the deal?

The issue turns out to be that the call was being done through one of my IDynamicObject implementations. In particular, this one handled a Call action by generating a rule target containing a call to the target delegate using the DLR"s Ast.ComplexCallHelper() method.

The problem is that ComplexCallHelper() doesn't really deal with language-specific conversion needs at all. Instead, it relies on Ast.Convert() to convert the actual argument values to the types declared in the target method/delegate signature.

The problem with Ast.Convert() should be obvious by now: All Ast.Convert() and Ast.ConvertHelper() do is insert an explicit Cast expression into the code.

This works great if all that is needed is a box/unbox operation or an upcast/downcast, but not enough in our case, so it is something to watch out for! I still haven't decided yet how to best work around the issue

Dynamic Language Runtime, .NETworking

I thought I'd use a few posts to talk a bit about some interesting ways that IronRuby does things. To do this effectively, we need to have a way to look at the code that IronRuby generates, as just browsing the code isn't going to do much good if you don't know where to start!

Fortunately, the DLR has our backs covered! As I've mentioned in the past, the DLR allows as to see the ASTs used for code generation as well as the Rules used to execute actions, and this is an invaluable tool both to debug our own language implementations, as well as to see what other implementations are doing.

Looking at both ASTs and Rules is easy if you're using the DLR Console either to execute a script in a file or in REPL mode, using the /X:ShowASTs and /X:ShowRules switches, like this:

Creating .NET Objects

One of the strengths of IronRuby is that it allows us to work against not only all the Ruby goodness, but also access a lot of the functionality already available in the .NET Framework [1].

For example, I could create an instance of System.Text.UTF8Encoding like this:

So what happens when we do this? The current implementation of IronRuby will generate an AST that looks like this:

There are several interesting things about this tree. As you can see, the qualified class name is translated to an object through a series of calls to methods in the RubyOps class. The first two calls return RubyModule objects corresponding to the "System" and "System::Text" namespace, while the last one returns a RubyClass object representing the type for the UTF8Encoding class. Then an InvokeMember action is built to call the "new" method in this RubyClass object, which will return the new instance of the UTF8Encoding class.

However, if you go and look at the code for the RubyClass and RubyModule classes, you won't find a "new" method anywhere. What gives?

The trick is, of course, in the IDynamicObject implementation of RubyClass/RubyModule, which receives the InvokeMember action and generates the right set of rules for this case. The method doesn't need to exists, as long as we can resolve the call and build a StandardRule whose target involves the right action/call being executed at runtime. In our simple example, this should mean creating a new instance of UTF8Encoding, which we can see

The process through which IronRuby builds new instances (or rather, derives the right set of rules for a new expression) is actually fairly complex; even for this simple case. However, in the end, you can think of this process in a simplified way:

1. The GetRule() implementation resolves the "new" method call in the RubyClass object by based on a fairly complicated set of lookups around the underlying .NET type and its superclasses and their corresponding RubyClass objects. What matters to us is that, at some point, IronRuby realizes that it really means "create a new instance",
2. RubyClass.SetRuleForCreateInstance() is invoked, which will configure the rule correctly for this case.
3. Within this, IronRuby notes that UTF8Encoding isn't a class defined in Ruby, but actually a CLR type.
4. The set of candidate constructors of the CLR type is evaluated to find the right constructor to use.
5. A New Expression (Tree.Ast.New()) is provided as the target of the rule.

So this is how our original InvokeMember action is converted into a new expression! Actually, this is a pretty simple case as far as IronRuby goes, because UTF8Encoding doesn't something corresponding in the Ruby language. To see a different scenario try creating an ArrayList (or other object that implements IList) and you will see a lot more interesting things going on to make the object and its class act ruby-like.

What about my language?

Does this mean you have to do it the same way when implementing your own language on the DLR? Certainly not. There are many ways you can choose to implement it and the DLR actually provides a few ways for that.

For example, in this particular scenario IronRuby is relying on an InvokeMember action, but the DLR also has a Create action which provides another way to represent your object construction. You still need to provide a callable object (IDynamicObject) to provide the rule, but it might not necessarily be very complicated once you resolve the typename to a CLR type.

Of course, the trick is finding out the right constructor to invoke for a given expression, and fortunately, the DLR provides some help to do this with the MethodBinder class. Here's a pretty minimal (not particularly efficient) implementation  you could start with:

[1] The examples in this post assume you're 'require'-ing mscorlib.dll.

Dynamic Language Runtime, DLR, .NET

Last time I talked a bit about Call Actions; now let's talk a bit about another kind of action: InvokeMember actions. As their name implies, these are useful for calling instance methods on .NET objects.

I struggled a bit with InvokeMember actions because they are act a little differently than Call actions. In particular, I was confused why sometimes InvokeMember actions seemed to require an extra expression in the argument list to work, when compared to Call actions. Fortunately, Dino Viehland clarified the reason for this, though it took me a lot of trial and error for his explanation to finally "click". I'll explain this in a moment.

To create a new InvokeMember action, you need the name of the method to invoke, the result type (like with a Call action, typeof(object) might suffice), the CallSignature and a list of expressions to use an arguments.

The CallSignature structure is interesting, because it can be used to provide information about how to interpret the call arguments. For example, it can be used to point out if an argument is the instance expression, if it's a named argument and so on (check the ArgumentKind enumeration for all the options). Filling in the CallSignature correctly is key to getting the correct behavior.

Here's what initially confused me about how InvokeMember works. Consider two different scenarios of how an InvokeMember action can be used to call a regular instance method:

1. You create a simple CallSignature that does not have an argument with ArgumentKind.Instance. In this case, building an InvokeMember action is very similar to building a Call action: first the "target" expression (which evaluates to the object on which to invoke the method), followed by the expressions representing the actual arguments to the call.
2. You build a CallSignature with an explicit Instance argument (usually the first one). Now, the call arguments need to be a bit more complex, defined as:
   1. The callable object (an expression evaluating to it, actually).
   2. The target expression evaluating to the object on which to invoke the method.
   3. Zero or more expressions representing the actual arguments to the method.

The reason this can be confusing is because this "Callable object" should not appear in the CallSignature you create to build the InvokeMember action: It really is a "hidden" parameter, so if you're calling a method requiring 3 arguments, you'd create a CallSignature listing 4 arguments (instance + actual args) but provide an expression array containing 5 expressions as the arguments to Ast.Action.InvokeMember().

But what is this Callable object? If you are thinking that it must be similar to what I called "Callable Object" when using Call Actions, you'd be right. Basically, the callable object argument can be used to provide a helper object that can help either make the actual call at runtime or, if it implements IDynamicObject, build the rule used by the DLR to invoke the actual method. [1]

Notice however, that unlike our sample last time, your GetRule<T> implementation will need to create a rule for DynamicActionKind.InvokeMember instead. One thing that is confusing is that you can use scenario (2) above but simply provide the target object as the first expression (that is, pass it twice), then the call will work as well directly, matching the behavior of scenario (1) [2].

In either way, the ability to use a Callable Object provides an easy way to hook the member invocation process to tweak how the call is made, which is a very useful capability in a dynamic language.

For example, IronRuby implements IDynamicObject in its built-in module and class objects (RubyModule and RubyClass, respectively) and are used to resolve method calls to ruby objects; you can look at how the default rule for InvokeMember actions is created inside MakeRuleForInvokeMember() method of the RubyBinder class. If I understand the code correctly, IronRuby takes advantage of this capability to implement method_missing. You can also check how calls to real .NET methods/properties are resolved to the original methods in RubyClass.TryGetClrMember().

[1] By now you can realize this pattern applies to all types of actions, not just Call and InvokeMember.
[2] In this case, a default rule is provided by the ActionBinder.MakeRule<T>() method, which you can also override and customize in your language binder implementation.

Dynamic Language Runtime, DLR, .NET

This time I'm going to talk a bit as to how to implement function calls in the Dynamic Language Runtime. Last time, I mentioned that I had initially had implemented function definitions as simply returning a raw CodeBlockExpression. This works, but it doesn't give you many options to add custom language-specific behaviors.

A more useful approach in many cases is to wrap your function into an actual object, at runtime, which can then be used for different purposes (including actually invoking the function). So instead of simply moving the CodeBlockExpression around you create a subtree that creates a new object with the CodeBlockExpression as an argument (among other things) and return that instead (with possibly other actions around it, like putting it into a variable).

This is how both IronRuby and ToyScript do it (and I imagine, IronPython). If you look at ToyScript's Def class (the language-specific AST node for a function definition), you will this in action:

As you can see, it will generate a Call expression to ToyFunction.Create(), which is a static method that returns an instance of the ToyFunction class. At runtime, ToyFunction.Create() will receive the name of the function, the names of the method parameters and a Delegate instance that can be used to actually invoke the code later on. That last part is the interesting bit.

So later on, you want to invoke your function. You already have a variable somewhere that evaluates to a Function object at runtime (in the case of ToyScript, a ToyFunction object as we mentioned). In more general terms, what you will have is an expression that will evaluate to the function object; it really could be anything (like another function call that returns another function). But how do you actually invoke it?

Turns out there are several ways you can implement this, with varying degrees of performance characteristics and needs.

Generating Calls

In the case of invoking a standalone (local or global) function you already defined in your language, the most obvious choice is to use a direct Call expression, via Ast.Call(). We already saw an example in the short snippet above, but for a different use.

Ast.Call() takes a System.Reflection.MethodInfo object (the method to execute) and a params array of expressions to use as arguments to the call (there are other overloads available). If the MethodInfo object points to a non-static method, then the first expression in the arguments list will be the instance [target] object on which to actually make the call.

This is pretty useful if you want to call methods you're aware of at compile/parse time, such as a built-in function like ToyFunction.Create() above, but it's not useful at all to do what we want: Invoking a CodeBlock we created somewhere else.

For this we need to turn to generating a Call action, using Ast.Action.Call():

The first parameter is the type of the return value of the call; which can be typeof(object) unless you have more advanced information about what the call will return. The second argument is the list of parameter values you want to pass to the function.

...So how do you tell Call() what to actually invoke?

Yep, that tripped me up as well. Turns out that a CallAction expression expects that the first expression in the arguments list evaluates to a callable object. At least that's how I think of it. But what's a callable object? As I understand it, it means that it references an object that either:

1. Implements IDynamicObject, or
2. Has a public instance method with the following signature:
   object Call(CodeContext context, params object[] arguments);

The current ToyScript incarnation follows path 2. Here's the implementation of the Call() method:

The code can look a bit convoluted, but it's actually quite simple: It simple checks the actual arguments supplied to the call to see if the CodeContext has already been included explicitly in it; otherwise it adds it as the first argument, and then uses ReflectionUtils.InvokeDelegate() to actually invoke the ToyScript function represented by this instance of ToyFunction (_target is a field containing our Delegate instance). It's easy, but I think it doesn't have the best performance.

IDynamicObject

A slightly more complex implementation would instead implement IDynamicObject. This what IronRuby's Proc class does, and it's what I currently have working on my own language implementation. The core of the IDynamicObject idea is that, unlike with the special Call() method above, our function object is no longer responsible for actually doing the call on the target function. Instead, we're merely responsible for providing the DLR with a rule it can use to invoke it. This is exactly what you do inside your GetRule<T>() implementation.

In our case, what we really want is to respond with a new rule for the Call dynamic action, which is the only one we're interested in it. My current (simplistic) implementation works like this:

As you can see, we create a new StandardRule but a very simple one:

• The test part of the rule simply checks that the target object on which to invoke the call is the same as our instance. This is what tells the DLR if the call should or should not be made. Here's I'm simply doing an identity test, which seems to work fine in my unit tests. IronRuby, for example, instead uses a unique ID assigned to each Proc instance for the call test.
• The second part will create a new array with the actual arguments to the call (i.e. without the first 'this' argument) and use that to build the call using Ast.ComplexCallHelper().

One we return the rule, the DLR can cache that information to make following calls more efficient, for example.

Something to watch out for here and that really confused me at first: If you read the above code carefully, you'll notice that the DLR Actually gives us an object[] to GetRule<T>(). This will contain the actual values of arguments to the call (not expressions that evaluate to them), so you can examine them to decide how to best to create your rule. Notice however that I don't actually use them. Instead I use the already existing expressions in StandardRule<T>.Parameters, which will be already populated with the expressions for your call arguments.

The reason for this is that, as it turns out, is that if you used the original values to build your own Expressions to create the test (and possibly parameters) for the call, you will run into an issue that can cause the DLR to go into an infinite recursion of nested DynamicSite.Invoke() calls. Not fun to debug, and looking at the rules or AST dumps won't tell you why it's failing. I think the relevant issue is mentioned in a comment in the DLR source code that reads "The test should be such that it does not become invalid immediately. Otherwise, ActionBinder.UpdateSiteAndExecute can potentially loop infinitely."

Anyway, once you realize how Ast.Action.Call() works and what it expects from you, it all starts making a lot more sense.

ActionBinders

It is also important to note that already in these simple scenarios your language's ActionBinder implementation starts to kick in. In particular, you need to ensure that it implements the required type conversions so that the DLR can make any conversions necessary between the actual argument values to a call and the types of the parameters as declared in the CodeBlock (through ActionBinder.ConvertExpression()).

Next time I'll talk a bit about InvokeMember actions.

Dynamic Language Runtime, DLR, .NET