A couple of week ago I blogged about using Pygments to do live syntax highlighting in the browser using Silverlight.

A major problem with the sample was that it did the pygmentizing on the UI thread which caused most browsers to become unresponsive. Today I wanted to fix that by using the BackgroundWorker to do the pygmentizing in a background thread.

Firstly I refactored the pygmentizing into a method that didn't interact with the UI.

def pygmentize_text(self, text, language):
    # attempt to pygmentize input with current language 
    try:

        from pygments import highlight
        from pygments.lexers import get_lexer_by_name
        from pygments.formatters import HtmlFormatter

        lexer = get_lexer_by_name( language, stripall=True)
        formatter = HtmlFormatter(linenos=False, cssclass="source")
        markup = highlight(text, lexer, formatter)

        return markup

    except:

        return "Error Generating Markup"

I then added a method that could be passed into a DoWorkEventHandler. It gets it arguments as a tuple from the event arguments and then sets the event argument result with the marked up HTML. The lack of explicit typing and use of tuples is good example of how some python idioms can be used when working with the .NET framework.

def worker(self, sender, e):

    # do work off UI thread. 
    e.Result = self.pygmentize_text(e.Argument[0],e.Argument[1])

The required BackgroundWorker and DoWorkEventHandler can be simply imported from the System.ComponentModel namespace.

from System.ComponentModel import BackgroundWorker, DoWorkEventHandler

The BackgroundWorker can then be setup and started. Again it's syntactically nice how the tuple can be created and passed as a RunWorkerAsync parameter.

def start_pygmentize(self):

    # update application state
    self.input_changed = False        
    self.pygmentizing = True
    self.show_message("pygmentizing..")

    # get paremters
    input = self.input.GetProperty("value")
    language = self.language.value

    # setup background worker
    worker = BackgroundWorker()
    worker.DoWork += DoWorkEventHandler(self.worker)
    worker.RunWorkerCompleted += self.complete

    # start the worker
    worker.RunWorkerAsync( (input,language) )

The completed event handler is the responsible for taking the markup generated by the BackgroundWorker and updating the DOM. It also fires off another worker if the source has changed since the last worker started.

def complete(self, sender, e):

    if e.Error:

        # handle errors/exceptions in worker
        self.source.SetProperty("innerHTML",e.Error.Message)

    else:

        # show the result
        self.source.SetProperty("innerHTML",e.Result)

    if self.input_changed:

        # input has changed, starty pygmentize again
        self.start_pygmentize()

    else:

        # no work queued
        self.pygmentizing = False
        self.hide_message()

The update has made the sample much more responsive, however it appears downloading the Silverlight application is still causing some browers to become a little unresponsive which is annoying. I will be interested to find out if this effect can be mitigated.

The actually pygmentizing processing could possibly be made a little faster by reusing the BackgroundWorker and only doing the Pygment imports once but the responsiveness of the browser has improved the sample enormously.

Check out the updated demo here.

After my post yesterday Krzysztof Kozmic (@kkozmic) messaged me saying I should use the Castle DictionaryAdapterFactory instead.

For the record Krzysztof has a great series of posts on DynamicProxy and was also very helpful answering a specific question of mine yesterday.

Anyway this is great news! I can do the same thing only referencing one assembly. The DictionaryAdapterFactory returns an object that passes all my tests from yesterday and slides staight into the model binder.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext,
                                          ModelBindingContext bindingContext,
                                          Type modelType)
    {
        Dictionary<string, object> dictionary = new Dictionary<string, object>();
        return new DictionaryAdapterFactory().GetAdapter(modelType, dictionary);
    }
}

It was fun playing with DynamicProxy yesterday, but I did think it might not have been a good idea for a model binder. I'm glad I was pointed in the right direction.

Cheers Krzysztof!

Update: I posted an update to this post which uses the Castle DictionaryAdapterFactory after getting some feedback from this post.

This started while I was playing with some ASP.NET MVC architectural ideas in simple application.

I wanted to less code than other projects I'd worked on but still have it decoupled and testable.

At some point I started ranting on twitter (1, 2, 3) about wanting a Model Binder that could bind to an interface without needing a concrete type.

It was quickly pointed out that what I probably wanted was a Model Binder that resolves an implementation at runtime with IOC.

While this was probably true as I was using already using StuctureMap and it did seemed like a much better idea I still some my reservations. But mainly I wanted to try using the Castle DynamicProxy in an application.

Clearly the reason the default model binder can't bind to an interface is because it can't create an instance of it. Thanks to some nice extensibility hooks in the default model binder, all I had to do was override the CreateModel method and return my DynamicProxy instance. The default model binder then could do the rest of binding work as normal.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext,
                                          ModelBindingContext bindingContext,
                                          Type modelType)
    {
        var generator = new ProxyGenerator();
        return generator.CreateInterfaceProxyWithoutTarget(modelType,
                new FakePropertiesInterceptor());
    }
}

What I needed then was a DynamicProxy object that had properties the behaved like you'd expect properties to on a normal class. I came up with some test scenarios I'd expect the proxy to pass using a test interface.

public interface IDynamicProxyTest
{
    int Integer { get; set; }
    string String { get; set; }
    DateTime Date { get; set; }
}

I figured the Model Binder would set values on the Dynamic Proxy object and the controller would get the properties back to validate and pass them to the data layer or something like that. I came up with these tests.

[Test]
public void If_date_has_been_set_then_return_it()
{
    var date = DateTime.Now;
    var proxy = CreateProxy();
    proxy.Date = date;
    Assert.AreEqual(date, proxy.Date);
}

[Test]
public void If_string_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.String = "Hello Proxy";
    Assert.AreEqual("Hello Proxy", proxy.String);
}

[Test]
public void If_int_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.Integer = 42;
    Assert.AreEqual(42, proxy.Integer);
}

After implementing these I was surprised to find the default model binder did some gets on the properties before sets. So I added some more tests.

[Test]
public void For_date_instance_if_value_not_set()
{
    var proxy = CreateProxy();
    var b = proxy.Date;
}

[Test]
public void For_int_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.Integer;
}

[Test]
public void For_string_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.String;
}

My property interceptor isn't as clean as I'd like it but did the job passing tests.

public class ProperyInterceptor : IInterceptor
{
    Type _type;

    Dictionary<string, object> properties;

    public ProperyInterceptor(Type type)
    {
        properties = new Dictionary<string, object>();
        _type = type;
        var fields = _type.GetProperties();
    }

    public void Intercept(IInvocation invocation)
    {
        if (IsSetter(invocation))
        {
            properties[Name(invocation)] = invocation.Arguments[0];
        }

        if (IsGetter(invocation))
        {
            if (properties.ContainsKey(Name(invocation)))
            {
                invocation.ReturnValue = properties[Name(invocation)];
            }
            else
            {
                GetDefaultValue(invocation);
            }
        }
    }

    private static void GetDefaultValue(IInvocation invocation)
    {
        if (invocation.Method.ReturnType == typeof(string))
        {
            invocation.ReturnValue = "";
        }
        else
        {
            invocation.ReturnValue = Activator.CreateInstance(invocation.Method.ReturnType);
        }
    }

    private string Name(IInvocation method)
    {
        return method.Method.Name.Substring(4);
    }

    private bool IsSetter(IInvocation method)
    {

        return method.Method.IsSpecialName &&
               method.Method.Name.StartsWith("set_", StringComparison.Ordinal);

    }
    private bool IsGetter(IInvocation method)
    {

        return method.Method.IsSpecialName &&
               method.Method.Name.StartsWith("get_", StringComparison.Ordinal);

    }
}

I used the better model binder Jimmy Bogard discusses as it makes writing binders for derived types and specific "types of type" more straight forward.

public class InterfaceBinder : FakeInterfaceModelBinder, IFilteredModelBinder
{
    public bool IsMatch(Type modelType)
    {
        return modelType.IsInterface;
    }
}

I did try writing a test which ran my model binder, but found it too much trouble. When I tried it out in a sample web project it worked exactly as I expected.

While this was fun I don't actually think it's a very good idea. I was thinking about the interface as DTO's and it breaks down if you want any additional methods on the interface, as they just won't work. Anyway it's been fun as always and maybe I'll find a good place to use these dynamic proxies in the future.

Source code with the sample website is all here.

I really like the way you can script your data model from a python REPL console on the Django and the Google App Engine web frameworks.

For me it is a hands down better way of working with data in your domain model than writing SQL.

I've since wanted to do it in .NET projects I work on, but it wasn't till I was playing with Castle ActiveRecord yesterday that I decided I'd try it out.

The ActiveRecord pattern is an intuitive way of programming with data persistence, so it's also nice to script with.

It turned out to be really easy in the project I was playing with. I didn't need to write a single additional line of C# as I already had the configuration decoupled for integration testing with an in-memory SQLite database.

namespace SimpleBlog.Data
{
    public class Configuration : IBootstrapperTask
    {
        public void Execute()
        {
            Configure(GetDefaultSettings());
        }

        public static void Configure(IDictionary<string, string> properties)
        {
            InPlaceConfigurationSource source = new InPlaceConfigurationSource();
            source.Add(typeof(ActiveRecordBase), properties);
            ActiveRecordStarter.Initialize(source);
            ActiveRecordStarter.RegisterAssemblies(Assembly.GetExecutingAssembly());
        }

        ...
    }
}

I then just wrote this little script to help with the configuration, it uses the static method above and passes in the properties for working with a development database.

import clr
clr.AddReferenceToFile("SimpleBlog.Data.dll")

from System.Collections.Generic import Dictionary
from SimpleBlog.Data import Configuration

# NHibinate Setting

properties = Dictionary[str,str]()

properties.Add("connection.driver_class",
               "NHibernate.Driver.SqlClientDriver");

properties.Add("dialect",
           "NHibernate.Dialect.MsSql2005Dialect");

properties.Add("connection.provider",
           "NHibernate.Connection.DriverConnectionProvider");

properties.Add("connection.connection_string",
           "Data Source=[CONNECTION_STRING]"); # Add

properties.Add("proxyfactory.factory_class",
           "NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle");

Configuration.Configure(properties)

Using the helper script it's pretty easy to get in and start working with the data in the data model.

IronPython 2.6 (2.6.10920.0) on .NET 2.0.50727.4927
Type "help", "copyright", "credits" or "license" for more information.
>>> import ActiveRecord
>>> from SimpleBlog.Data.Models import *
>>>
>>> post = Post()
>>> post.Title = "Working with SQL sucks!"
>>> post.Content = "Try using a scripting langauge instead. It rocks!"
>>> post.Author = "tarn"
>>> post.Save()
>>>
>>> post.Id
2
>>>
>>> posts = Post().FindAll()
>>>    
>>> for p in posts:
...     print p.Title, "by", p.Author
...
Hey, It's alive by tarn
Working with SQL sucks! by tarn
>>>

This is a simple example of a database agnostic data script using your domain model and a powerful scripting language. I think scripting data models like this could add a lot of value in many .NET development scenarios.

17-02-2010 I've updated the demo to use the BackgroundWorker and posted about the update

I decided it might be fun to try get Python Markdown and Pygments running in the browser to enhance a markdown preview experience by eliminating the server-side round trips and provide a more responsive preview.

I managed to get it working entirely in Python but I found the application size excessively large (almost 3mb) and, more annoyingly, it seems to block the entire browser when it initially loads the pygments module. I think there is some sort of silverlight background thread I should be using.

I think it would work better with MarkdownSharp as pure C# silverlight applications are a fair bit leaner in size and probably run a little quicker than dynamic language applications. But this was for fun and I prefer coding in Python when not working.

17-02-2010 I've since found the MarkdownSharp doesn't do syntax hightlighting

I ended up having to write so little Python code to get this working that I can include it all here, syntax highlighted with Pygments of course.

from System.Windows import Application
from System.Windows.Controls import UserControl
from System.Windows.Browser import HtmlPage
from System import EventHandler

class App:

    def __init__(self):

        # load relevent HTML DOM elements
        self.input = HtmlPage.Document.GetElementById("input")
        self.source = HtmlPage.Document.GetElementById("output")
        self.language = HtmlPage.Document.GetElementById("lang")

        # fire javascript functions to indicate the application has been load
        HtmlPage.Window.CreateInstance("silverlight_loaded");

        # pygmentize initial 
        self.pygmentize()

        # register events
        self.input.AttachEvent('onkeyup', EventHandler( self.update_handler )) 
        self.language.AttachEvent('onchange', EventHandler( self.update_handler ))

        # fire javascript function to indicated the pygments has been loaded
        HtmlPage.Window.CreateInstance("pygments_loaded");

   # handle language or input changes by pygmentizing 
    def update_handler(self, sender, e):

        self.pygmentize()

    def pygmentize(self):
        input = self.input.GetProperty("value")

        # attempt to pygmentize input with current language 
        try:

            from pygments import highlight
            from pygments.lexers import get_lexer_by_name
            from pygments.formatters import HtmlFormatter

            lexer = get_lexer_by_name(self.language.value, stripall=True)
            formatter = HtmlFormatter(linenos=False, cssclass="source")
            markup = highlight(input, lexer, formatter)

            # update the preview
            self.source.SetProperty("innerHTML",markup)

        except:

            # indicate there was an error in pygmentize
            self.source.SetProperty("innerHTML", "Error Generating Markup" )

# Do it!    
App()

Despite the fact that there isn't much code, the development experience writing silverlight application in python is a bit of a pain in the arse. Granted it's much better with the python console in the browser and better error reporting in most recent SDK, but it still sucks; debugging and logging support is very limited and on some errors the application dies without reporting anything.

Another difficulty is having to manually copy all the python standard library modules required by the module from the library folders into the application (which explains something about the bloated application size). And even though the code in the demo works, some very similar code from the pygments quick start doesn't.

Check out the live demo.

Context

I was asked do a presentation about developing Silverlight applications to the Melbourne Python User-group. Hell yeah, I love participating in user groups but I obviously needed some content. Unfortunately I didn't know this local user-group existed, so I've haven't been and have little idea of number of people or the type of stuff they do. This has made preparing relevant content a little challenging, but hopefully I can find the right mix.

Given it's a Python user-group I felt I might look like a bit of dick rolling with Windows and doing a demo with an environment and .NET tools that some part of the group couldn't/don't use. Besides I've always felt the Python language support and being able to build on any platform is a very compelling reason for non-.NET developers get into building Silverlight applications.

I don't yet have a Mac (I'm really starting to want one) so my other option was to do the demo on Linux. This presented some difficulties and some opportunities. Firstly I'm still a bit of Linux noob and, although I'm wrapped to have learned a lot in the last couple of weeks there has been some pain and some compromise.

Also Microsoft don't have a Silverlight browser plug-in for Linux, that support is provided by Moonlight. Moonlight is currently at version 2 and although it supports some of the features of Silverlight 3 it doesn't have the key version 3 features which have really improved the Python development environment (namely IronPython 2.6 support) and application deployment (namely allowing static assemblies that add dynamic languages support to be downloaded independently of the application and therefore cached in the browser).

Plugin and development support on both Mac and Windows is much better, but I'm happy here sacrificing some of the newer features for inclusivity (and apparently inventing words). We're targeting Silverlight 2/Moonlight 2 and effectively using no development tools besides a couple of Python scripts which I'll get onto.

Dynamic Language Silverlight Applications

Silverlight applications are packaged into binary files with an "xap" extension. It turns out these packages are just standard zip files containing all the "stuff" needed by the application. The minimum "stuff" needed in dynamic language application is;

• An XML based application manifest file (app.manifest) that describes the target runtime, language type, entry point and some other information needed to run the application
  

• A collection of assemblies that add dynamic language and Python support to the Silverlight runtime.

• A plain text file containing the your Python script to make the application dance.

The package can contain more Python scripts, binary assemblies (which can be imported as modules) and XAML files (which are declarative xml type format typically used for describing user interfaces). Resources such as images and video can also be included in the package but it usually make more sense for the application to download them independently when they are required.

Serving the application on a webpage is also quite straight forward. An object tag in the html has attributes to tell the browser the URL to download the application and that the Silverlight/Moonlight plugin should be used to run the application.

Helper Scripts and Templates

With a couple of templates and Python scripts we can get started developing Silverlight applications in any environment with that has Python and a browser with a Silverlight/Moonlight plug-in.

xap.py builds xaps by adding all the files from a source code directory and an assemblies directory into xap file and writing it to disk.

http_server.py is the simplest python server which just serves all the files in the current directory on a specified port.

index.html contains the object tag referencing sample.xap and a tiny bit of Javascript to catch, format and display critical error messages that may be caused by the application.

app.manifest is a template with everything needed to describe a typical Python Silverlight application. The template should be fine as-is unless additional assemblies are required or you want to change the script entry point. The template is surprisingly small and undaunting.

app.py is the script entry point of the application.

assemblies a folder with all the additional assemblies need by the Silverlight runtime to run Python scripts.

Packaging and Running an Application

So you've skipped the previous three sections and jumped straight here. I don't blame you. Here's how to get going with the script and templates described above.

Download this zip containing the assemblies, scripts and templates and unzip it someone, anywhere.

From the terminal/console/command-line go the the place you unzipped it to and run the following command to build an xap.

python xap.py samples.xap assemblies source

A sample.xap should be generated in the same directory.

To fire-up the basic web server just run

python http_server.py

Then just fire up a browsers and go to http://localhost:8000/ and you should see the sample Silverlight application running.

That's it. Now you can edit app.py to add functionality and repeat.

Going further

Clearly this a very bare bones approach to development which has very limited development and debugging tools (and a server that can only serve files). There is more tooling out there, but surprisingly litte for Python developers.

MonoDevelop 2.2, the Moonlight SKD and Python bindings may provide some sort of IDE support but it's only just been released, it is not easy to get up and running and I not sure if it even supports packaging or debugging of dynamic language Silverlight applications.

You might think Visual Studio 2008/2010 would provide significant additional support, but as far as I'm aware only the Windows Debugger which is free in the Windows SDK is much help.

Chiron is a .NET tool provided in the Microsoft Dynamic Language SDK which can package and server XAP's much like I've described above. I found I could compile it on Mono, but couldn't get it serving files. I don't think your missing much here.

XamlPad is my favourite free XAML editor, but it doesn't work on Linux. Blend and Expression Studio are pay to play Microsoft tools for designing and developing XAML. Maybe someone should write a good one in Silverlight.

IronPython can be downloaded and run on Mono.

Some Thoughts

It's easy to get started writing and deploying Silverlight applications in Python on any platform but I think it's fair to say the tooling and support is just not there yet. On any platform.

The Silverlight 2/Moonlight implementations of Python are still missing large parts of the Python standard library, which will come as a bit of a shock to Python developers and makes it difficult to import modules that depend on it. Almost the entire Silverlight 2 Base Class Library is accessible through Python but although the Python/.NET integratation is very impressive, working with the BCL does seem to limit the expressivness of the code.

Silverlight 3 appears to have some additional built-in support for interacting with the application at runtime through a Python REPL console, it supports IronPython 2.6 (which implements a lot more of the Python standard library) and has additional deployment improvements discussed above.

Finally

I hope to do another post before the meeting with some examples I'll be showing, but as always I won't promise anything (in case I get distracted writing a Socket Server in Erlang I've been thinking about). In the mean time I do have some examples on PythonSilverScripting that can be run from the browser.

I'd love to know the tools other people are using to develop Python/Silverlight and thoughts people have of it. If your in Melbourne and your at all interested you should come down to the meeting.

Update 26/01/10: Updated scripts and templates to work in Windows

Here is a zip with the bits and pieces.