I checked out NGourd, a project to build a BDD tool based on Cucumber which Michael Minutillo has started. I’d never previously used or even heard of Cucumber, but as I read on I liked a lot about it. It’s an agile development tool and it has a colourful console!

The concept is illustrated beautifully at http://cukes.info/, but here is my take anyway. Behavior is described in the form of a series of plain text steps. The steps are then implemented as parameterized test methods. Using some reflection and regular expressions the plain text steps are mapped to corresponding implementations and run as red/green light tests.

I started wondering if WPF applications could be tested with NGourd. There’s a great IronPython sample in which a WPF application is started in a new thread and built dynamically from the IronPython interactive console.

Could something like this could be done in NGourd steps?

	
Scenario: Can Perform basic arithmetic

Given I start the application
When I enter 2 + 2
And I click GO
Then I should see 4

By using a thread to run the application and a dispatcher to invoke methods on it I had a crude framework to write the steps to make a System.Windows.Application dance. The steps class library has references to the application (to run and test it) and to the NGourd.Core (which has the Attributes I need to decorate the step classes and methods). I’ve exposed the controls of application publically to make the test easier to write.

[Steps]
public class Behaviour
{
    public Thread _thread;
    public Dispatcher _dispatcher;
    public App _app;

    [Step("start the application")]
    public void StartApplication()
    {
        AutoResetEvent are = new AutoResetEvent(false);
        _thread = new Thread(() =>
        {
            _app = new App();
            _app.Startup += (sender, eventArgs) =>
                    {
                        _dispatcher =
                            Dispatcher.FromThread(_thread);
                        are.Set();
                    };
            _app.Run();
        });
        _thread.SetApartmentState(ApartmentState.STA);
        _thread.Start();
        are.WaitOne();
    }

    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.UserScript.Text = script;
            }));
    }

    ...

    public void After()
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.Close();
            }
            ));
        _thread.Join();
    }
}

Writing the application

The demo application is a trivial calculator that executes Python statements.

The method below is the model for this application. It’s worth mentioning it’s very easy to include C# objects to the scope of the ScriptSource, but only for this post, I’ve restrained myself to the more simple behavior I specified earlier (but I did add it to the demo which can be downloaded here)

public string EvaluateExpression(string expression)
{
    try
    {
        ScriptEngine engine = Python.CreateEngine();
        ScriptRuntime runtime = engine.Runtime;

        ScriptSource source =
          engine.CreateScriptSourceFromString(
                       expression,
                       SourceCodeKind.Expression);

        return (source.Execute() ?? "").ToString();
    }
    catch(Exception ex)
    {
        return ex.Message;
    }
}

Of course this could be unit tested itself.

[TestMethod]
public void ModelCanDoBasicArithmetic()
{
    Model model = new Model();
    string output = model.EvaluateExpression("2+2");
    Assert.AreEqual("4", output);
}

So what are we testing? I guess its this

public void Execute_Click(object sender, EventArgs e)
{
    output.Text = _model.EvaluateExpression(input.Text);
}

The only thing I had to do the application specifically to get it working in test environment was delete an attribute in App.xaml root element

StartupUri="Window1.xaml"

This has consequences. No window loads on start up! By added a constructor in my App class I was able to remedy this. This may have wider reaching consequences I’m not aware of, and I’m also not sure if using the OnLoad event is better practice. Anyway it’s working fine for now.

public partial class App : Application
{
    public App()
    {
        MainWindow w = new MainWindow();
        w.Show();
    }
}

It was fun implementing the steps in my scenario then writing the application and watching the behavior test turn green like a cucumber! 

Isolating scenarios in separate application domains

Even though everything looked looks like it was working, I wanted to add another scenario (which won’t surprise readers of my recent posts)

	
Scenario: Can Execute Python statements

Given I start the application
When I enter “hello”.upper()
And I click GO
Then I should see HELLO

It turns out this extra test didn’t need any additional step implementations or changes to the WPF application itself, but did show a very real deficiency in what I was doing. I was reminded of the rule enforced by InvalidOperationException:

Cannot create more than one System.Windows.Application instance in the same AppDomain.

I knew of this rule, I still tried to get round it anyway. Can I close one somehow? Delete one? Delete one and de-reference its friends and family?

Creating a new AppDomain seems a clever way round the rules reasoning, but the wary fear this may be some type of pandora's box, at very least additional complexities. Just thinking of MarshalByRefObject proxies makes me feel I should be spending more time enjoying life with my friends and family.

I got some way using DoCallBack until I realized I couldn’t make the steps parameterized as DoCallBack doesn’t have any parameters! (I also stubbornly know it throws runtime errors if one tries to use the looser lambda scoping to cheekily pass data across application domains, luckily though it’s the same stubboness that stopped me giving up this post all together).

As I was thinking I should use WCF for communication betweens application domains, another way I could do to it clicked. I just needed to create my original steps class in a new application domain. By deriving the test class from MarshalByRefObject I could call its step methods from the control domain using it dynamic proxy. I then created a wrapper class which creates a new AppDomain, creates the previous steps in the new domain and uses the dynamic proxy to call the steps. This wrapper class now has the Step attributes for NGourd to pick up.  

[Steps]
public class BehaviourWrapper
{
    AppDomain domain;
    Behaviour behaviourProxy;

    public void Before()
    {
        domain = AppDomain.CreateDomain("TestDomain");
        behaviourProxy = domain.CreateInstanceFromAndUnwrap
            ("DLRCalculator.BehaviourTests.dll",
             "DLRCalculator.BehaviourTests.Behaviour") 
                as Behaviour;
    }

    public void After()
    {
        behaviourProxy.After();
        AppDomain.Unload(domain);
    }

    [Step("start the application")]
    public void StartApplication()
    {
        behaviourProxy.StartApplication();
    }

    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        behaviourProxy.EnterScript(script);
        Thread.Sleep(5000);
    }

    [Step(@"click GO")]
    public void ClickGo()
    {
        behaviourProxy.ValidateOutput();
    }

    [Step(@"should see (.*)")]
    public void ValidateOutput(string expected)
    {
        behaviourProxy.ValidateOutput(expected);
    }
}

Running some tests with NGourd

I’m impressed with NGourd and have generally had a lot of fun and success playing with it. It’s a little disappointing that no results are displayed until all the tests are complete. Perhaps it’s because my tests take ages and it would be nice to see what they’re doing (starting an AppDomain and running an Application in it isn’t a lightweight task). Mainly though, I wanted to include an action shot of it firing up windows during the test.

Some thoughts and reflection

I didn’t write this to actually use it in a real project, this was a personal project deliberately devised to cover some some things I was interesting in learning about and trying out. To do automation on a real project I would look further into White an open source project released by ThoughtWorks, which I hear is really good.

The concept of Cucumber is pretty cool and I was nice to have a look at the NGourd source code. In writing this post I enjoyed putting in practice what I new could be done relatively easily (creating and managing multiple AppDomains, Applications and Threads in a process) and I was happy to find I had no problems at all with NGourd itself getting the UI tests working.

This is all I planned to do with NGourd, the UI steps and the Calculator, but I’d be interested to hear what you think of all this rambling. I know I got a lot out of it, did you? What of the other behavior driven development tools MSpec, NSpec or NBehave? What are your experiences?

I’ve upload the project from this post here.

While twittering (or tweeting) about an IronPython presentation I was doing, @voidspace messaged me asking if I might mention his book. I’d contacted Michael Foord a while back about some code I was playing around with. I also knew of him from his blog, his prolific contribution to IronPythonURLs and as the co-author of the new book IronPython in Action. He’s so all over everything IronPython he even reads this blog.

Mainly because I was too cheap to buy myself a copy it, I hadn’t read his  book. I was wrapped when he offered to get me a copy if I’d review it on this blog. I wanted to read it as I was learning IronPython and I knew it would be a good book, I’d read chapters 1 and 7 which are online. I happily recommended it in my presentation, here’s what I thought about the book.

IronPython in Action is a fantastic resource for anyone learning IronPython and for anyone wondering what this dynamic language IronPython is all about and whether they should learn it. It feels nicely balanced for both .NET users looking to learn about IronPython, and for Python users looking to learn about IronPython on .NET.

I prefer software books that have plenty of discussion and can be read from start to finish otherwise I end up skipping through chapters I never go back too. I found this book well written and fun to read with interesting insight all the way through. There is enough about the python language and .NET to get you through the book, and rather than being a reference book it instead provides information on how to learn more.

The book is divided into four main sections. The first section Getting Started with Iron Python contains a good introduction and history of IronPython on the .NET Framework.

The Core Development Techniques section is a walkthrough of building a simple Windows Forms application. I think this section is fantastic as the text places high value on Agile Development philosophy, using a Model-View-Controller pattern and Test Driven Development. I think these are exciting ways to develop software on the .NET framework. There is also a really good section discussing the difficulties, pitfalls and work-arounds using IronPython and dynamic programming languages generally on the .NET framework.

The third section IronPython and Advanced .NET contains chapters about using the major pillars of the .NET framework with IronPython. It has chapters on WPF, Powershell, ASP.NET, Databases, Webservices and Silverlight. It really is amazing how much of the framework can be used in IronPython.

The final section Reaching Out with IronPython has fantastic chapters on extending IronPython with C#/VB and embedding an IronPython Engine in C#/VB applications to provide extensibility.

I highly recommend this book, I enjoyed reading it and learned a lot from it. If you’re looking for a little more insight, the forward by Jim Hugunin frames the language, the technology and author well.

I’m doing an introduction to IronPython presentation at the Victoria .Net user group tomorrow, I’ve been attending the user group for about a year and I’m pretty excited about presenting. I think python is a fantastic language and it’s great to be able to build software on the .Net framework with it.       

The Slides

	Introduction I wanted to start with the second slide and have a photo of a pub on the contact slide but decided to get through this before trying out PowerPoint jokes.
	Why Python I just plan to discuss Why I started using Python. The cartoon is, of course, XKCD.
	Python Basic information about the language and its background. Would have liked to have added meta programming or at least “..” to the paradigms. I think it’s a fantastic language so I’ll mention it’s expressive, fun, productive or whatever adjectives I’ve got in my head at the time.
	People using Python I really want to demonstrate that Python is real language, being used by real people, to write real code.  I probably should also have mentioned Google App Engine here which I’ve been tinkering with a bit recently and think is pretty cool.
	IronPython The Python language on .Net. Most of the demo is doing cool stuff in .Net with the Python language. I think it goes further than this and allows developers to use their own paradigms and patterns to do .NET development.
	C# and IronPython comparison I saw this example in the awesome Iron Python in Action book. I included it as it’s a great introduction IronPython code, as it clearly and simply shows differences on similarities with C#.
	Dynamic Language Runtime The Dynamic Language Runtime project is open source. Its quite well described here as:  A dynamic type system Dynamic method dispatch Dynamic code generation IronLisp has been superseded by the IronScheme project.
	Dynamic Typing This slide is just a brief overview of what duck typing is. It is great for writing test objects, but I have a slide for the later.
	Interactive Interpreter I think it’s awesomely cool, and I use it through the entire demo so I thought it might be worth warning people.

The Demo

The demo is a series of scripts run into the IronPython Interactive Console. I’ve uploaded the very small collection of scripts for this demo here.

IronPython and the CLR

Create a string, show how its methods can be reflected. Import the CLR and show how the .Net String methods are now available.

# Introduction to IronPython and the CLR

# Standard first Python console command
2 + 2

# Create a string
s = 'Hello IronPython'

# Inspect its methods, note that its only Python methods.  
dir(s)

# Python has a method to make the text uppercase
s.upper()

# Import the CLR
import clr

# Now look at the same objects methods,
# We now see the .NET string methods!
dir(s)

# And can use them!
s.ToUpper()

Screen Shot

Using a .Net Classes

Create a WebClient object from the .Net framework to download an RSS feed. Use an xml2py module written by Harry Pierson to deserialise the RSS feed into an object graph. Then send the dates and titles to the Speech Synthesizer. I’ve seen this done before in an IronPython presentation, and just thought it was to much fun to leave out. 

import clr

# Create a WebClient and download a string of XML
from System.Net import WebClient
w = WebClient()
xml = w.DownloadString('http://feeds.theage.com.au/rssheadlines/technology.xml')

# Lets have a look at first 60 characters of the xml..
print xml[0:60]

# If bad stuff happens in the presentation..
#xml = open('theage.rss').read()

# Deserialize to objects using xml2py
from devhawk import xml2py
rss = xml2py.parseString(xml)

# Print the results
for post in rss.channel.item:
  print post.pubDate, post.title 

# Import Speech  
clr.AddReference('System.Speech')
from System.Speech import *
ss = Synthesis.SpeechSynthesizer()

# Say titles and dates using the SpeechSynthesizer 
for post in rss.channel.item:
  s = "%s. %s." % (post.pubDate[:16], post.title)
  ss.SpeakAsync(s)

Screen Shot

SQL

I really don’t want to do this in the presentation. I did spend a fair amount of time playing round with asynchronous SQL queries for use in a WPF application, but it seemed too complicated for the demo so I’ve left it out. I use

# import a simple wrapper around System.Data.SqlClient  
# the wrapper is on my blog, I just wanted to show the AdventureWorks 
# database, it seems a bit of .NET presentation tradition. 
from demo import sql

# setup a query on the AdventureWords database
query = 'SELECT * FROM Production.Product'
sql.connection = 'Data Source=localhost\SQLEXPRESS;Initial Catalog=AdventureWorks;Integrated Security=SSPI;'

# execute the query
products = sql.SqlQuery(query)

# print some results
for p in products[:5]:
  print p.Name

WPF, Data Binding and Exec

I think there is some pretty cool stuff in this demo. Although it takes a lot more to write real applications it does open new ways to develop and test .Net applications.

# More pythonic way of getting the feed.. lets hope I have internet
from devhawk import xml2py
rss = xml2py.parse('http://feeds.theage.com.au/rssheadlines/technology.xml')

# Importing avalon starts a new thread to run the WPF application
# It sets up some dispatures so the console can talk to the GUI 
from samples import avalon

# Create a window, show it and load some XAML into it
w = avalon.Window()
w.Show()
xaml = avalon.LoadXaml('demo.xaml')
w.Content = xaml

# Loading names is really cool, it traverses the control tree 
# described in the XAML and creates a dictionary of names and 
# controls. We can add those dictionary items to the local scope,
# so we get the same effect as a generated designer files in
# Visual Studio. 
avalon.LoadNames(xaml,locals())

# We can bind a Python list of Python objects to the ListView
listView.ItemsSource = rss.channel.item

# Create a button click callback that will execute the script
# in the query TextBox.
def OnClick(*args):
  try:
    exec (query.Text)
  except:
    print "Script threw an exception"
  pass  

# Wire up the click event to our function  
execute.Click += OnClick
  
# We can now execute this on the client  
listView.ItemsSource = [p for p in rss.channel.item if p.title.find('to') >= 0]

Screen Shot

More Slides

	Testing in Python It’s really cool you can do true test driven development in that you can write unit tests for classes and methods that haven’t yet been written. Some cool GUI testing is also possible using some the stuff in this presentation.
	Tools / SDKs I’m not expecting to mention Python Studio or IpyDbg as I’ll probably be running out of time. I really should have included a link to python.org, I’ll make sure I mention it. You can get IronPython here and the Silverlight Dynamic Languages SDK
	Resources Dive Into Python – Free online, python for programmers. IronPython In Action – Python and everything .NET with TDD and MVC practices. IronPython Cookbook – Online collection of IronPython .Net scripts. IronPython Urls, MSDN, Devhawk – Harry Pierson, Program Manager at Microsoft. Voidspace – Michael Foord, Author IronPython in Action.
	Thanks By this stage I’ll almost certainly be running late, and probably be really worried about some minor disaster during the presentation. I hope it goes well and everyone takes something out of it, I look forward to hearing what you thought.

My interest in a light-weight command-line debugger developed a while back after I watched one used in a Ruby on Rails video. I suspect there are many .Net developers with more experience than me who have used command line debuggers and never want to go back. The more I code, the more I long to leave my mouse behind and communicate with my development tools using only my keyboard, Excalibur.

I have been following a series of blogs by the awesome Harry Pierson (aka DevHawk) on writing an IronPython debugger. When I decided I actually wanted to play with the debugger, I found myself going back over these blogs taking notes as I went. This post is an attempt to summarize all this information and provide the links for going deeper, for myself if no one else.

Writing an IronPython Debugger: Introduction 

Explains why he built a debugger when there are other alternatives. The Visual Studios debugging was required to much mouse clicking, although he has posted about doing that too in Debugging IronPython Code in Visual Studio. MDbg which is a .Net Command Line Debugger doesn’t support Just My Code debugging. He also notes the reason for not using/porting pdb (The Python Debugger) is due to IronPython not yet implementing settrace.

Writing an IronPython Debugger: MDbg 101 

Describes the basics of MDbg. I found it interesting there is also an IronPython extension for MDbg which isn’t for debugging IronPython code, but to interact with MDbg using IronPython. Sounds cool, I’ll have check it out some time.

Writing an IronPython Debugger: Hello, Debugger! 

Goes over the worlds simplest debugger, and it really is pretty straight forward. It just passes the path of ipy.exe and the python code to debug as a parameter into CorDebug (provided by MDbg). CorDebug starts the process (it can be used to debug any .Net process) and provides some events to hook into; OnCreateAppDomain and OnProcessExit. The world simplest debugger basically just prints some text when the AppDonain is starts and stops.

Writing an IronPython Debugger: Setting a Breakpoint

Provides some background information:

• ipy.exe produces debug information when the –D parameter is used.
• IL generation is dynamic and in memory so the debugger API provides the Symbols (equivalent to .pdb files) as a Stream and has a callback when the Symbols change.
• The .Net framework has an API to read and write these symbols a files and MDbg provides a wrapper to read them from a Stream.

Then covers the python code to load the symbols, translate document/line into a function/offset, set a breakpoint. Does not implement user defined breakpoints in the post. 

Writing and IronPython Debugger: Adding Interactivity

Covers the basics of how the interactivity will happen, which I’ve just realized isn’t that complicated. The process being debugged just runs until a breakpoint event occurs (or the symbols change or the process completes). Once the process has stopped the the breakpoint the interactivity can occur. It would be nice to be able to set breakpoints while the process is running, but it does keep it more simple for now.

Writing an IronPython Debugger: Dynamic Stack Trace

Background info on MDbg:

• Two stack chains in a typical managed app; unmanaged and managed. The debugger for managed code so only the managed chain used
• Each contains a collection of stack frames which is the familiar call stack. There are three types of stack frame; IL, native and internal.
• Explains Dynamic Methods are usually used created by IronPython but can’t be debugged and are implemented as non-dynamic methods when the –D option is set on ipy.exe. 

MDbg has a wrapper around the unmanaged metadata API to get method information for displaying the call stack.

The command in ipydbg is used to view the trace from the interactive console is “T”.

Writing an IronPython Debugger: Refactoring

Mostly discussion about code changes and refactoring. Add code to automatically generate the MTA apartment state if the –X:MTA argument is not used. Explains the effect of this on the debugger design.

Writing an IronPython Debugger: Stepping Thru Code

Introduces console commands for step (S), step in (I), step out (O)

Writing an IronPython Debugger: Debugging Just My Code

Skipped over this one, and feel I may start skipping over a few where I’m not specially interested in the implementation. I can always refer back to them later if I really need to understand something.

Writing an IronPython Debugger: Showing Source Code

Describes an issue when stepping into a Python function the CLR breaks at some infrastructure code, presumably there to manage Pythons decorators and such. Hence there is no line of user code to display. To resolve this an additional automatic step is added when stepping into a function so it can be mapped to a line of user code.

To avoid multiple hits to the file system source files used to retrieve user code are cached.

Writing an IronPython Debugger: Colorful Console

Describes how he choose to implement colors in the console, while bemoaning the stateful nature of the console foreground colours (ie when you change the foreground colour it will stay that way until you change it back). I’m sure I’ll come back to this post when I want to add colours to the console but for now I skipped through it, it doesn’t have much to do with writing or using an IronPython debugger. I guess that’s why he discusses moving it out of the ipydbg and into its own module in his next post.

Writing an IronPython Debugger: A Little Hack…err…Cleanup

Discusses some issues regarding mapping to the debugger COM object instance. This didn’t sound like a whole log of fun and I hope I don’t have to come back and fully understand this one.

Writing an IronPython Debugger: Getting Local Variables

Uses the GetLocalVariable(int index) and GetLocalVariablesCount() methods from the MDbg CorFame class. This post finally made me look up what lexical scoping meant, its a term I’ve heard heaps usually in discussions about compilers that I never really understood. Its fair to say that I still not confident in my understanding of it.

Discusses matching up debug symbols with variable names from the user code. Doesn’t actually evaluate the variable in this post. Notes get_locals from the IronPython process emits some locals used internally, these are prefixed with a dollar sign.

Writing an IronPython Debugger: Displaying Values

This post covers some pretty tricky interfaces for dealing with all the different types, which requires a pretty good understand of how the CLR handles variables under the cover. I didn’t read into to much detail but will consider revisiting it later as the content is quite interesting and it’s good to understand core CLR stuff.

Writing an IronPython Debugger: Command Routing

Discusses how console commands are routed to functions. This really has nothing to do with debugging IronPython but it’s interesting the way it’s implemented. He starts by implementing a switch using a dictionary of input commands and functions (Python has no C# switch keyword equivalent). He then takes it further by making use of Python decorators to bind commands to functions.

Writing an IronPython Debugger: Getting Arguments

Discusses getting function arguments as locals.

Writing an IronPython Debugger: REPL Console

Discusses implementing a REPL console in the debugger with the IPyDebugProcess object available in the console scope. This is awesome for exploring the API using python, just by reflecting the methods of the process object I realised it would be trivial to add a command to list the source files currently being debugged as it’s just a property on the object. Explorative coding using reflection is a really powerful concept in Python.   

In the current implementation is a new local scope is created and used by the by the REPL console, it does not yet support executing code in the process being debugged.

Writing an IronPython Debugger: Breakpoint Management

Implements console commands for setting Breakpoints, as the original post was about how it worked and only implemented breaking on the first line of the user code. Introduces multi-key commands for breakpoints. Adds a breakpoint function (B) with sub functions add (A), list (L), enable (E), disable (D).

Here’s a screenshot of me debugging a really simple app which might help explain what I’ve been talking about for this whole post.

I’m pretty amazed by the productivity and brilliance of Harry Pierson, I’ve learned heaps reading his blogs and I’m really impressed by the progress he’s making with IpyDbg. The debugger still has a long way to being a really useful tool, but its coming along very quickly. I think I will have to try using MDbg with the Python extensions, but I really hope to make some use of ipydbg and maybe even find something I can contribute to it.

If your interested in trying out the debugger I recommend checking his blog and the latest version of the project on GitHub.  

My first Code Camp Oz was the best Microsoft event I've attended. It’s basically a large user group meeting held over an entire weekend at Charles Stuart University in Wagga Wagga. Its a long way to travel but some cool people from the Australian .Net development community attend and it’s a great opportunity to get to know them.

I found the meeting people aspect more interesting than the presentations which was the reason I came anyway, I can watch the Mix video anytime. I met an awesome group of guys from Albury outside a Silverlight 3 presentation. Due to lateness we spent the entire presentation outside chatting which was lots of fun. I didn’t realize until I signed into Twitter one of the guys was Matt Hamilton who I knew of from his massive Stack Overflow reputation.

The Alt.Net presentation was one I'd penciled in as interesting before the event and was pleased find I shared common development values and principles with the Alt.Net community. I had a good chat with the presenter Liam McLennan and ended up hanging out with him a fair bit during the weekend.

Saturday night at the pub was good fun. I had an interesting chat about what was involved in becoming a Microsoft MVP and the value of vender certification. It turns out you don't need to have any Microsoft Certification to become an MVP which I think is good. For me personally I think there are other things I’d prefer to do to become a better developer.

An interesting aspect of the event was Twitter. I've never really got that into Twitter, but the constant background commentary and discussion really added to the experience. The informal way you can listen and contribute to conversation is great. It meant I didn't take any notes, but I can always search the #ccoz09 tags for quotes, links and photos from the event.

I think it would be really cool break from the power point presentations and maybe looking at some open source code like the ASP.NET MVC framework at this type of event. I could see this working with a series of presenters reviewing the architecture and highlighting certain areas of code for discussion. I'm not sure if this would actually work, but I think it could be really fun and interesting.

I was a little concerned my return train departed Wagga at 3am Monday morning and arrived in Melbourne at 8am, but I met a couple of guys to who gave me a lift back to Melbourne on Sunday and happily abandoned my 3am train.

Overall it was a really well organized and attended event. I extend my gratitude to Mitch Denny and all the people involved in making it happen.

Hopefully I'll see you all at Code Camp Oz 2010.

Since playing with JQuery I've started thinking more about unobtrusive JavaScript and graceful degradation. Despite less than 5% of users browsing without JavaScript enabled, I found it very interesting disabling JavaScript on my Browser and checking out which sites still work and which ones fail. I found the Microsoft Live Search Maps has absolutely no clue your browsing the site without JavaScript, it just renders a loading spinning GIF.

Without JavaScript Enabled	With JavaScript Enabled

Google maps performs much better, providing a rich client side library with JavaScript enabled and a functional mapping service without it.

Without JavaScript Enabled	With JavaScript Enabled

There are also plenty of sites that lose some or all of their functionality, but let you know why. YouTube can't play videos, but you can browse them and it provides an message indicating why the movies don't work.

Hello, you either have JavaScript turned off or an old version of Adobe's Flash Player. Get the latest Flash player.

StackOverflow has graceful degradation implemented well. You can still ask and answer questions but there is a banner at the top of the page

Stack Overflow works best with JavaScript enabled

You then can't expand comments and you don't get the cool Ajax features that indicate when you've earned a badge or someone else has answered the question your currently answering, but its still pretty functional.

Geocoding without JavaScript

It turns out the good guys at Google have provided a REST API to do geocoding without using their fantastic client side JavaScript libraries. Its all documented in the Geocoding API Developer's Guide.

Google do state this in their terms for the REST geocoding service;

This service is designed for geocoding static (known) addresses using a REST interface, for placement of application content on a map. For dynamic geocoding of user-defined addresses (for example, within a user interface element), consult the documentation for the JavaScript Client Geocoder or the Maps API for Flash Client Geocoder.

I don't think this is a huge problem as a well designed website will only use this geocoding service for the 5% of users who don't have Javascript enabled. If JavaScript is enabled, the DOM can be updated using the rich JavaScript functionality.

I wrote a little script to wrap around the service, I wrote the script in Python as I'm hoping to use it on Google App Engine, but more about that later. To use this script you'll need to sign up to get a Google Maps API key and use it to set the GeoCoder.geoCodeKey property.

import urllib2
from xml.dom import minidom
class GeoCoder():
def __init__(self):
self.geoCodeUrl = 'http://maps.google.com/maps/geo?q=%(address)s&output=xml&oe=utf8&sensor=false&key=%(key)s'        
self.geoCodeKey = ''
def GetUrlKey(self,address):
return self.geoCodeUrl % {'key':self.geoCodeKey,'address':urllib2.quote(address)}
def GetKml(self, address):
response = urllib2.urlopen(self.GetUrlKey(address))
return response.read();
class KmlParser():
def GetText(self,node,nodeName):
return node.getElementsByTagName(nodeName)[0].childNodes[0].data
def GetCoordinates(self,node):
items = self.GetText(node, "coordinates").split(',')
return { 'latitude':items[0], 'longitude':items[1] }
def ParseAddress(self, node):
address = { 'address':self.GetText(node,'address'),
'country':self.GetText(node,'CountryName'),
'street':self.GetText(node,'ThoroughfareName'),
'postcode':self.GetText(node,'PostalCodeNumber'),
'coordinates': self.GetCoordinates(node) }
return address 
def Parse(self, xml):
results = []
dom = minidom.parseString(xml)
for placeNode in dom.getElementsByTagName('Placemark'):
results.append(self.ParseAddress(placeNode))
dom.unlink()
return results;
def GetAddresses(address):
return KmlParser().Parse(GeoCoder().GetKml(address))
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
for address in GetAddresses(' '.join(sys.argv[1:])):
print address
else:
print "Enter Address to Geocode, using the Google Geocoding API"

from xml.dom import minidom
class GeoCoder():
def __init__(self):
self.geoCodeUrl = 'http://maps.google.com/maps/geo?q=%(address)s&output=xml&oe=utf8&sensor=false&key=%(key)s'        
self.geoCodeKey = ''
def GetUrlKey(self,address):
return self.geoCodeUrl % {'key':self.geoCodeKey,'address':urllib2.quote(address)}
def GetKml(self, address):
response = urllib2.urlopen(self.GetUrlKey(address))
return response.read();
class KmlParser():
def GetText(self,node,nodeName):
return node.getElementsByTagName(nodeName)[0].childNodes[0].data
def GetCoordinates(self,node):
items = self.GetText(node, "coordinates").split(',')
return { 'latitude':items[0], 'longitude':items[1] }
def ParseAddress(self, node):
address = { 'address':self.GetText(node,'address'),
'country':self.GetText(node,'CountryName'),
'street':self.GetText(node,'ThoroughfareName'),
'postcode':self.GetText(node,'PostalCodeNumber'),
'coordinates': self.GetCoordinates(node) }
return address 
def Parse(self, xml):
results = []
dom = minidom.parseString(xml)
for placeNode in dom.getElementsByTagName('Placemark'):
results.append(self.ParseAddress(placeNode))
dom.unlink()
return results;
def GetAddresses(address):
return KmlParser().Parse(GeoCoder().GetKml(address))
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
for address in GetAddresses(' '.join(sys.argv[1:])):
print address
else:
print "Enter Address to Geocode, using the Google Geocoding API"

Which makes geocoding pretty easy from Python

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import geocoder
>>> addresses = geocoder.GetAddresses('main st, melbourne, victoria')
>>> for a in addresses: print geocoder.AddressToString(a)
...
Main St, Mordialloc VIC 3195, Australia (145.0858270,-38.0058450)
Main St, Lilydale VIC 3140, Australia (145.3507800,-37.7575460)
Main St, Mornington VIC 3931, Australia (145.0407850,-38.2231010)
Main St, Greensborough VIC 3088, Australia (145.1058060,-37.7025570)
Main St, Diamond Creek VIC 3089, Australia (145.1478460,-37.6744410)
Main St, Thomastown VIC 3074, Australia (144.9994220,-37.6787790)
Main St, Blackburn VIC 3130, Australia (145.1491980,-37.8258150)
Main St, Upwey VIC 3158, Australia (145.3366500,-37.9050420)
Main St, Croydon VIC 3136, Australia (145.2811170,-37.7961170)

Google also note this in the terms;

Geocoding is a time and resource intensive task. Whenever possible, pre-geocode known addresses (using the Geocoding Service described here or another geocoding service), and store your results in a temporary cache of your own design.

Even though the geocoding service is actually quite quick, it does make a lot of sense not to make a call to an external service if its not required. I suggest following this advice.

Maps without JavaScript

Normally some very clever JavaScript dynamically adds new map tile images to the DOM when sections of the map need to be rendered out, so the normal Google Maps service won't work without JavaScript. Luckily Google provide a REST Static Maps service which serves up map images. Its all documented in the Static Maps API Developer's Guide.

I wrote this Python script to get a map URL, again a Google Maps API key is required.     

key = 'MAPS_API_KEY'
url = 'http://maps.google.com/staticmap?'
colours = ['brown', 'green', 'purple', 'yellow', 'blue', 'gray', 'orange', 'red', 'white','black']
def GetImageUrl(params, markers):
if markers: 
marker = [','.join(p) for p in markers]
params['markers'] = '|'.join(marker)
if not params.has_key('key') : params['key'] = key 
return url + '&'.join(['='.join(p) for p in params.items()])     
if __name__ == '__main__':
# Generate map URL with 10 locations in Melbourne Australia
letters = 'ABCDEFGHI'
address = [['144.9994220','-37.6787790'],
['145.2811170','-37.7961170'],
['145.0407850','-38.2231010'],
['145.3507800','-37.7575460'],
['145.3366500','-37.9050420'],
['145.1491980','-37.8258150'],
['145.1058060','-37.7025570'],
['145.1478460','-37.6744410'],
['145.0858270','-38.0058450']]
markers = [ [e[1],e[0],colours[i]] for i, e in zip(range(len(address)), address)]
params = { 'size':'256x256', 'maptype':'roadmap', 'sensor':'false', 'markers': markers }
print GetImageUrl(params, markers)

Using this script and little more Python hacking I could quickly generate some HTML with a map and links to more detailed maps.

A. Main St, Thomastown VIC 3074, Australia   B. Main St, Croydon VIC 3136, Australia   C. Main St, Mornington VIC 3931, Australia   D. Main St, Lilydale VIC 3140, Australia   E. Main St, Upwey VIC 3158, Australia   F. Main St, Blackburn VIC 3130, Australia   G. Main St, Greensborough VIC 3088, Australia   H. Main St, Diamond Creek VIC 3089, Australia   I. Main St, Mordialloc VIC 3195, Australia

Putting it all together

There is still some consideration to put this all together in website that supports server side mapping and geocoding without JavaScript and enhances the mapping with unobtrusive JavaScript.

Page responses need to return HTML that will work without JavaScript. If the browser executes the pages JavaScript, the script can attach an events and disable default behaviors to implement the richer client side functionality.  

On a page a user entered address can be geocoded, the requested HTML should contain a form that can post back to the server to do the geocoding. If client side JavaScript is not used the server can handle the post and use the geocoding service to respond appropriately, of course with JavaScript the addresses can be added to the DOM dynamically, probably with some slide or fade effect.

On pages that display maps, the static image URL can be sent in the requested HTML and used if the JavaScript Google Maps can't be loaded. As on Google Maps some additional controls can be added to allow some panning and zooming without JavaScript.

I'm going to see if I can get the all working on the Google App Engine, I'll let you know how I go.