I decided it might be fun to try get Python Markdown and Pygments running in the browser to enhance a markdown preview experience by eliminating the server-side round trips and provide a more responsive preview.

I managed to get it working entirely in Python but I found the application size excessively large (almost 3mb) and, more annoyingly, it seems to block the entire browser when it initially loads the pygments module. I think there is some sort of silverlight background thread I should be using.

I think it would work better with MarkdownSharp as pure C# silverlight applications are a fair bit leaner in size and probably run a little quicker than dynamic language applications. But this was for fun and I prefer coding in Python when not working.

Despite the fact that there isn't much code, the development experience writing silverlight application in python is a bit of a pain in the arse. Granted it's much better with the python console in the browser and better error reporting in most recent SDK, but it still sucks; debugging and logging support is very limited and on some errors the application dies without reporting anything.

Another difficulty is having to manually copy all the python standard library modules required by the module from the library folders into the application (which explains something about the bloated application size). And even though the code in the demo works, some very similar code from the pygments quick start doesn't.

Check out the live demo.

Context

I was asked do a presentation about developing Silverlight applications to the Melbourne Python User-group. Hell yeah, I love participating in user groups but I obviously needed some content. Unfortunately I didn't know this local user-group existed, so I've haven't been and have little idea of number of people or the type of stuff they do. This has made preparing relevant content a little challenging, but hopefully I can find the right mix.

Given it's a Python user-group I felt I might look like a bit of dick rolling with Windows and doing a demo with an environment and .NET tools that some part of the group couldn't/don't use. Besides I've always felt the Python language support and being able to build on any platform is a very compelling reason for non-.NET developers get into building Silverlight applications.

I don't yet have a Mac (I'm really starting to want one) so my other option was to do the demo on Linux. This presented some difficulties and some opportunities. Firstly I'm still a bit of Linux noob and, although I'm wrapped to have learned a lot in the last couple of weeks there has been some pain and some compromise.

Also Microsoft don't have a Silverlight browser plug-in for Linux, that support is provided by Moonlight. Moonlight is currently at version 2 and although it supports some of the features of Silverlight 3 it doesn't have the key version 3 features which have really improved the Python development environment (namely IronPython 2.6 support) and application deployment (namely allowing static assemblies that add dynamic languages support to be downloaded independently of the application and therefore cached in the browser).

Plugin and development support on both Mac and Windows is much better, but I'm happy here sacrificing some of the newer features for inclusivity (and apparently inventing words). We're targeting Silverlight 2/Moonlight 2 and effectively using no development tools besides a couple of Python scripts which I'll get onto.

Dynamic Language Silverlight Applications

Silverlight applications are packaged into binary files with an "xap" extension. It turns out these packages are just standard zip files containing all the "stuff" needed by the application. The minimum "stuff" needed in dynamic language application is;

• An XML based application manifest file (app.manifest) that describes the target runtime, language type, entry point and some other information needed to run the application
  

• A collection of assemblies that add dynamic language and Python support to the Silverlight runtime.

• A plain text file containing the your Python script to make the application dance.

The package can contain more Python scripts, binary assemblies (which can be imported as modules) and XAML files (which are declarative xml type format typically used for describing user interfaces). Resources such as images and video can also be included in the package but it usually make more sense for the application to download them independently when they are required.

Serving the application on a webpage is also quite straight forward. An object tag in the html has attributes to tell the browser the URL to download the application and that the Silverlight/Moonlight plugin should be used to run the application.

Helper Scripts and Templates

With a couple of templates and Python scripts we can get started developing Silverlight applications in any environment with that has Python and a browser with a Silverlight/Moonlight plug-in.

xap.py builds xaps by adding all the files from a source code directory and an assemblies directory into xap file and writing it to disk.

http_server.py is the simplest python server which just serves all the files in the current directory on a specified port.

index.html contains the object tag referencing sample.xap and a tiny bit of Javascript to catch, format and display critical error messages that may be caused by the application.

app.manifest is a template with everything needed to describe a typical Python Silverlight application. The template should be fine as-is unless additional assemblies are required or you want to change the script entry point. The template is surprisingly small and undaunting.

app.py is the script entry point of the application.

assemblies a folder with all the additional assemblies need by the Silverlight runtime to run Python scripts.

Packaging and Running an Application

So you've skipped the previous three sections and jumped straight here. I don't blame you. Here's how to get going with the script and templates described above.

Download this zip containing the assemblies, scripts and templates and unzip it someone, anywhere.

From the terminal/console/command-line go the the place you unzipped it to and run the following command to build an xap.

python xap.py samples.xap assemblies source

A sample.xap should be generated in the same directory.

To fire-up the basic web server just run

python http_server.py

Then just fire up a browsers and go to http://localhost:8000/ and you should see the sample Silverlight application running.

That's it. Now you can edit app.py to add functionality and repeat.

Going further

Clearly this a very bare bones approach to development which has very limited development and debugging tools (and a server that can only serve files). There is more tooling out there, but surprisingly litte for Python developers.

MonoDevelop 2.2, the Moonlight SKD and Python bindings may provide some sort of IDE support but it's only just been released, it is not easy to get up and running and I not sure if it even supports packaging or debugging of dynamic language Silverlight applications.

You might think Visual Studio 2008/2010 would provide significant additional support, but as far as I'm aware only the Windows Debugger which is free in the Windows SDK is much help.

Chiron is a .NET tool provided in the Microsoft Dynamic Language SDK which can package and server XAP's much like I've described above. I found I could compile it on Mono, but couldn't get it serving files. I don't think your missing much here.

XamlPad is my favourite free XAML editor, but it doesn't work on Linux. Blend and Expression Studio are pay to play Microsoft tools for designing and developing XAML. Maybe someone should write a good one in Silverlight.

IronPython can be downloaded and run on Mono.

Some Thoughts

It's easy to get started writing and deploying Silverlight applications in Python on any platform but I think it's fair to say the tooling and support is just not there yet. On any platform.

The Silverlight 2/Moonlight implementations of Python are still missing large parts of the Python standard library, which will come as a bit of a shock to Python developers and makes it difficult to import modules that depend on it. Almost the entire Silverlight 2 Base Class Library is accessible through Python but although the Python/.NET integratation is very impressive, working with the BCL does seem to limit the expressivness of the code.

Silverlight 3 appears to have some additional built-in support for interacting with the application at runtime through a Python REPL console, it supports IronPython 2.6 (which implements a lot more of the Python standard library) and has additional deployment improvements discussed above.

Finally

I hope to do another post before the meeting with some examples I'll be showing, but as always I won't promise anything (in case I get distracted writing a Socket Server in Erlang I've been thinking about). In the mean time I do have some examples on PythonSilverScripting that can be run from the browser.

I'd love to know the tools other people are using to develop Python/Silverlight and thoughts people have of it. If your in Melbourne and your at all interested you should come down to the meeting.

Update 26/01/10: Updated scripts and templates to work in Windows

Here is a zip with the bits and pieces.

I’ve just build and published a simple site on Google App Engine: PythonSilverScripting.

PythonSilverScripting is based on crazy idea I have that it should be possible to make Silverlight applications in Python on the web. No tools, no SDKs.. just a browser (and obviously the Silverlight browser plug-in).

I've played around a bit with Silverlight and Dynamic Languages here and there and thought it was probably possible. When I got a chance I looked into it and found I was able to build working XAPs by zipping up plain text IronPython code, the IronPython DLR assemblies, an xml application manifest file and all the additional resources such XAML, images and more source.

All I needed was a web server that could zip files. Google App Engine with Python is good fun and I was able to get this working prototype together in an evening. I'd like to build the features to support building bigger projects (more source files, XAML documents, images and assemblies). On the other hand I'd also like to build features to share, discuss and rate scripts. We'll see how it goes.

I hope to take a closer look at Gestalt which makes it possible to write Python and Ruby directly in the HTML document. On quick inspection it appears to be using an XAP with the DLR stuff as a kind of bootstrapper which then inspects the DOM for more script it can execute. It seems to do some dynamic downloading resources which looks interesting, particularly given the size of the assemblies I need to package in the XAP and send to the client every time that “Generate Silverlight” button is clicked.

There is also of also the Silverlight Interactive Session on the IronPython website which is pretty cool.

I’ve got a tiny Acer Aspire “netbook” I use while commuting to work. It’s a very low spec’d machine but it has a very functional keyboard and I love programming on it!

I think I enjoy programming on it because it doesn’t have Visual Studios installed. I have the Windows SDK so I can build .NET projects with MsBuild, but I mainly use it for developing Python code. With some free tools and notepad++ plug-ins its a great light weight environment for coding simple Django and Google App Engine web applications and dynamic language .NET applications.

I’m not really into computer gaming, but I do like 2d physics games. I remember losing days work when I first found Magic Pen. I decided to try using the cool Farseer Physics Engine, which is described on the Codeplex page as; 

The Farseer Physics Engine is an easy to use 2D physics engine designed for Microsoft’s XNA and Silverlight platforms. The Farseer Physics Engine focuses on simplicity, useful features, and enabling the creation of fun, dynamic games.

Getting started

The Farseer engine is written in C# and I found once I’d installed the Silverlight SDK I could use MSBuild to build the Farseer engine that targets the Silverlight runtime. The Farseer engine doesn’t itself reference anything Silverlight specific, but it does need to be built against the limited .NET framework available in Silverlight.

I’ve previously blogged about writing dynamic language Silverlight applications; the Silverlight Dynamic Languages SDK contains the Silverlight IronPython assemblies and a command line application called Chiron. Chiron is a web server that runs from the command line and dynamically packages up XAPs as they are requested. 

A little about XAPs

XAPs are packaged Silverlight applications that are downloaded and run on the clients Silverlight runtime. It turns out the XAP is just zip files containing the assemblies, resources and in the case of dynamic language applications, some plain text source code!

Additionally there is an AppManifest.xaml included in the XAP which tells Silverlight about what is included in the XAP and where to find an entry point. Below is an example of an AppManifest.xaml automatically generated by Chiron.

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment" 
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
            RuntimeVersion="2.0.31005.0" 
            EntryPointAssembly="Microsoft.Scripting.Silverlight" 
            EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication">
  <Deployment.Parts>
    <AssemblyPart Source="Microsoft.Scripting.Silverlight.dll" />
    <AssemblyPart Source="Microsoft.Scripting.ExtensionAttribute.dll" />
    <AssemblyPart Source="Microsoft.Scripting.Core.dll" />
    <AssemblyPart Source="Microsoft.Scripting.dll" />
    <AssemblyPart Source="IronPython.dll" />
    <AssemblyPart Source="IronPython.Modules.dll" />    
    <AssemblyPart x:Name="FarseerPhysics" Source="bin/FarseerPhysics.dll" />
  </Deployment.Parts>
</Deployment>

• There is a whole lot of additional referenced assemblies required to compile Pyhton code in the Silverlight client, and yes, they are included in the XAP sent to the client.
• The FarseerPhysics assembly is also included the XAP and is referenced in the manifest.
• The EntryPointType is really the only difference between XAP manifest files generated for statically typed languages. For dynamic languages there are additional assemblies that compile the dynamic code, there doesn’t seem any reason you couldn’t feasibly write EntryPoint assemblies that interpreted lol-cat or brainfuck code.
• And, I’m still targeting Silverlight 2! (Just haven’t got round to upgrading..) 

Writing an application

I started by having a play with the Farseer engine by writing some IronPython scripts and running them from the console. As it’s just a physics engine, it can be coupled with any user interface, even a console.

When I tried to get the same script working in Silverlight I did encounter some difficulties referencing the Farseer assembly. I had to use the assemblies strong name, which I found using Reflector, but seems unnecessary given it was sent in the same XAP as the code referencing it!

Instead of just hacking this demo together in one giant script I decided to structure my code a little, including separating the view and the simulation model over different namespaces. Generally I’m learning heaps about what is pythonic, what I carry over from C/C++/C# and what is good software design.

I’m not entirely happy with my view classes, which are just wrappers around Silverlight controls. Even though the Silverlight control can be accessed through a property, I feel this isn’t the elegant solution I should be able to build in a dynamic language. It seemed more right for the physics wrappers, but still overly verbose.   

Conclusion

I limited the example for this post to a very simple, but interactive 2d physics demo. Clicking any of the objects in the simulation creates a spring between the object and the mouse cursor until the mouse click is released or leaves the simulation view area. 

The Farseer physics engine is a lot of fun to use, it’s easy to get into and you can do some really cool stuff with it.

The source code is sent as plain text and compiled (or interpreted) on client. As the XAP is just a normal zip file, it can unzipped by almost any zip tool. Inside is all the plain text source code, compiled assemblies and manifest file.   

It’s awesome to be able to develop Silverlight applications with physics libraries using a high level language and light weight development environment. Happy hacking..

I checked out NGourd, a project to build a BDD tool based on Cucumber which Michael Minutillo has started. I’d never previously used or even heard of Cucumber, but as I read on I liked a lot about it. It’s an agile development tool and it has a colourful console!

The concept is illustrated beautifully at http://cukes.info/, but here is my take anyway. Behavior is described in the form of a series of plain text steps. The steps are then implemented as parameterized test methods. Using some reflection and regular expressions the plain text steps are mapped to corresponding implementations and run as red/green light tests.

I started wondering if WPF applications could be tested with NGourd. There’s a great IronPython sample in which a WPF application is started in a new thread and built dynamically from the IronPython interactive console.

Could something like this could be done in NGourd steps?

	
Scenario: Can Perform basic arithmetic

Given I start the application
When I enter 2 + 2
And I click GO
Then I should see 4

By using a thread to run the application and a dispatcher to invoke methods on it I had a crude framework to write the steps to make a System.Windows.Application dance. The steps class library has references to the application (to run and test it) and to the NGourd.Core (which has the Attributes I need to decorate the step classes and methods). I’ve exposed the controls of application publically to make the test easier to write.

[Steps]
public class Behaviour
{
    public Thread _thread;
    public Dispatcher _dispatcher;
    public App _app;

    [Step("start the application")]
    public void StartApplication()
    {
        AutoResetEvent are = new AutoResetEvent(false);
        _thread = new Thread(() =>
        {
            _app = new App();
            _app.Startup += (sender, eventArgs) =>
                    {
                        _dispatcher =
                            Dispatcher.FromThread(_thread);
                        are.Set();
                    };
            _app.Run();
        });
        _thread.SetApartmentState(ApartmentState.STA);
        _thread.Start();
        are.WaitOne();
    }

    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.UserScript.Text = script;
            }));
    }

    ...

    public void After()
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.Close();
            }
            ));
        _thread.Join();
    }
}

Writing the application

The demo application is a trivial calculator that executes Python statements.

The method below is the model for this application. It’s worth mentioning it’s very easy to include C# objects to the scope of the ScriptSource, but only for this post, I’ve restrained myself to the more simple behavior I specified earlier (but I did add it to the demo which can be downloaded here)

public string EvaluateExpression(string expression)
{
    try
    {
        ScriptEngine engine = Python.CreateEngine();
        ScriptRuntime runtime = engine.Runtime;

        ScriptSource source =
          engine.CreateScriptSourceFromString(
                       expression,
                       SourceCodeKind.Expression);

        return (source.Execute() ?? "").ToString();
    }
    catch(Exception ex)
    {
        return ex.Message;
    }
}

Of course this could be unit tested itself.

[TestMethod]
public void ModelCanDoBasicArithmetic()
{
    Model model = new Model();
    string output = model.EvaluateExpression("2+2");
    Assert.AreEqual("4", output);
}

So what are we testing? I guess its this

public void Execute_Click(object sender, EventArgs e)
{
    output.Text = _model.EvaluateExpression(input.Text);
}

The only thing I had to do the application specifically to get it working in test environment was delete an attribute in App.xaml root element

StartupUri="Window1.xaml"

This has consequences. No window loads on start up! By added a constructor in my App class I was able to remedy this. This may have wider reaching consequences I’m not aware of, and I’m also not sure if using the OnLoad event is better practice. Anyway it’s working fine for now.

public partial class App : Application
{
    public App()
    {
        MainWindow w = new MainWindow();
        w.Show();
    }
}

It was fun implementing the steps in my scenario then writing the application and watching the behavior test turn green like a cucumber! 

Isolating scenarios in separate application domains

Even though everything looked looks like it was working, I wanted to add another scenario (which won’t surprise readers of my recent posts)

	
Scenario: Can Execute Python statements

Given I start the application
When I enter “hello”.upper()
And I click GO
Then I should see HELLO

It turns out this extra test didn’t need any additional step implementations or changes to the WPF application itself, but did show a very real deficiency in what I was doing. I was reminded of the rule enforced by InvalidOperationException:

Cannot create more than one System.Windows.Application instance in the same AppDomain.

I knew of this rule, I still tried to get round it anyway. Can I close one somehow? Delete one? Delete one and de-reference its friends and family?

Creating a new AppDomain seems a clever way round the rules reasoning, but the wary fear this may be some type of pandora's box, at very least additional complexities. Just thinking of MarshalByRefObject proxies makes me feel I should be spending more time enjoying life with my friends and family.

I got some way using DoCallBack until I realized I couldn’t make the steps parameterized as DoCallBack doesn’t have any parameters! (I also stubbornly know it throws runtime errors if one tries to use the looser lambda scoping to cheekily pass data across application domains, luckily though it’s the same stubboness that stopped me giving up this post all together).

As I was thinking I should use WCF for communication betweens application domains, another way I could do to it clicked. I just needed to create my original steps class in a new application domain. By deriving the test class from MarshalByRefObject I could call its step methods from the control domain using it dynamic proxy. I then created a wrapper class which creates a new AppDomain, creates the previous steps in the new domain and uses the dynamic proxy to call the steps. This wrapper class now has the Step attributes for NGourd to pick up.  

[Steps]
public class BehaviourWrapper
{
    AppDomain domain;
    Behaviour behaviourProxy;

    public void Before()
    {
        domain = AppDomain.CreateDomain("TestDomain");
        behaviourProxy = domain.CreateInstanceFromAndUnwrap
            ("DLRCalculator.BehaviourTests.dll",
             "DLRCalculator.BehaviourTests.Behaviour") 
                as Behaviour;
    }

    public void After()
    {
        behaviourProxy.After();
        AppDomain.Unload(domain);
    }

    [Step("start the application")]
    public void StartApplication()
    {
        behaviourProxy.StartApplication();
    }

    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        behaviourProxy.EnterScript(script);
        Thread.Sleep(5000);
    }

    [Step(@"click GO")]
    public void ClickGo()
    {
        behaviourProxy.ValidateOutput();
    }

    [Step(@"should see (.*)")]
    public void ValidateOutput(string expected)
    {
        behaviourProxy.ValidateOutput(expected);
    }
}

Running some tests with NGourd

I’m impressed with NGourd and have generally had a lot of fun and success playing with it. It’s a little disappointing that no results are displayed until all the tests are complete. Perhaps it’s because my tests take ages and it would be nice to see what they’re doing (starting an AppDomain and running an Application in it isn’t a lightweight task). Mainly though, I wanted to include an action shot of it firing up windows during the test.

Some thoughts and reflection

I didn’t write this to actually use it in a real project, this was a personal project deliberately devised to cover some some things I was interesting in learning about and trying out. To do automation on a real project I would look further into White an open source project released by ThoughtWorks, which I hear is really good.

The concept of Cucumber is pretty cool and I was nice to have a look at the NGourd source code. In writing this post I enjoyed putting in practice what I new could be done relatively easily (creating and managing multiple AppDomains, Applications and Threads in a process) and I was happy to find I had no problems at all with NGourd itself getting the UI tests working.

This is all I planned to do with NGourd, the UI steps and the Calculator, but I’d be interested to hear what you think of all this rambling. I know I got a lot out of it, did you? What of the other behavior driven development tools MSpec, NSpec or NBehave? What are your experiences?

I’ve upload the project from this post here.

While twittering (or tweeting) about an IronPython presentation I was doing, @voidspace messaged me asking if I might mention his book. I’d contacted Michael Foord a while back about some code I was playing around with. I also knew of him from his blog, his prolific contribution to IronPythonURLs and as the co-author of the new book IronPython in Action. He’s so all over everything IronPython he even reads this blog.

Mainly because I was too cheap to buy myself a copy it, I hadn’t read his  book. I was wrapped when he offered to get me a copy if I’d review it on this blog. I wanted to read it as I was learning IronPython and I knew it would be a good book, I’d read chapters 1 and 7 which are online. I happily recommended it in my presentation, here’s what I thought about the book.

IronPython in Action is a fantastic resource for anyone learning IronPython and for anyone wondering what this dynamic language IronPython is all about and whether they should learn it. It feels nicely balanced for both .NET users looking to learn about IronPython, and for Python users looking to learn about IronPython on .NET.

I prefer software books that have plenty of discussion and can be read from start to finish otherwise I end up skipping through chapters I never go back too. I found this book well written and fun to read with interesting insight all the way through. There is enough about the python language and .NET to get you through the book, and rather than being a reference book it instead provides information on how to learn more.

The book is divided into four main sections. The first section Getting Started with Iron Python contains a good introduction and history of IronPython on the .NET Framework.

The Core Development Techniques section is a walkthrough of building a simple Windows Forms application. I think this section is fantastic as the text places high value on Agile Development philosophy, using a Model-View-Controller pattern and Test Driven Development. I think these are exciting ways to develop software on the .NET framework. There is also a really good section discussing the difficulties, pitfalls and work-arounds using IronPython and dynamic programming languages generally on the .NET framework.

The third section IronPython and Advanced .NET contains chapters about using the major pillars of the .NET framework with IronPython. It has chapters on WPF, Powershell, ASP.NET, Databases, Webservices and Silverlight. It really is amazing how much of the framework can be used in IronPython.

The final section Reaching Out with IronPython has fantastic chapters on extending IronPython with C#/VB and embedding an IronPython Engine in C#/VB applications to provide extensibility.

I highly recommend this book, I enjoyed reading it and learned a lot from it. If you’re looking for a little more insight, the forward by Jim Hugunin frames the language, the technology and author well.