I was chatting with Matt Brailsford earlier about the idea of using Selenium to do automated testing of the Umbraco 5 back office (the UI).

Back in June I gave a demo at the Core Retreat about doing it, but against the v4.7 back office (since v5 wasn't in a good state to test back then), the code of which you can find here - http://hg.apwll.me/umbraco-automated-testing/overview

Here's the basic idea of what I was trying to illustrate then (and what Matt and I were talking about today):

• There's nothing more annoying than the back office UI breaking
• The Core devs (and HQ) can't cover all facets of clicking around the UI every time a build goes out
• The back office is a complex piece of HTML, CSS and JavaScript and a minor change in one place can have cascading effects that don't get picked up
• We could be using Selenium (a web browser automation library) to run through scenarios in the back office and report errors, ultimately stopping builds
• This can be used as a last-line of QA before releasing to the community for QA of a release

While this sounds all well and good there are a few roadblocks:

• Automated tests can be tricky to do
• Someone has to work out (and write down) all the scenarios
• Selenium can be a tricky beast to tame
• The back office itself is quite a complex bit of UI

Something that I proposed back then (and am proposing again now) is that we use Behavioural Driven Development (BDD - http://en.wikipedia.org/wiki/Behavior_Driven_Development) rather than standard unit tests because:

• Tests (often) are written in 'plain english'
• This would mean that the test scenarios could be written by people who may not be comfortable writing the actual test code, but having the scenario is half the battle!
• Here's an example of a BDD test I have in the sample scenario: http://hg.apwll.me/umbraco-automated-testing/src/968d1817fc5d/Umbraco.AutomatedTesting.Tests/UmbracoContentTree.feature and the implementation: http://hg.apwll.me/umbraco-automated-testing/src/968d1817fc5d/Umbraco.AutomatedTesting.Tests/Steps/UmbracoContentTree.cs

The idea behind it is that we use the Given, When, Then approach:

1. Given an initial state
2. When something happens
3. Then I expect a certain outcome

This means that you don't have to know how to implement the test, that could be done by someone who knows more about it (but the hopefully they can teach you while doing it so you can try as well!), instead you think about what scenario you're wanting to cover.
Here's a few example scenarios:

Given I am not logged in
When I use valid credentials
Then I am given access to the back office

Given I am not logged in
When I use invalid credentials
Then I am not given access to the back office 

Given I have deleted a node
Then I can find it in the recycling bin

Given I have created a DocType
When I create a new page
Then I can choose my new DocType

Given I click the logout button
Then I am redirected away from the back office
And I cannot get back in without logging in

These 5 scenarios very in complexity of what they are doing but hopefully they illustrate the point that I'm trying to get across, as users of Umbraco you know what the system should do, even if you don't know how it happens.

So what do people think does it seem like a good idea? What would stop you from being involved with writing scenarios and/ or writing the tests?

One nice bonus with using this method is you can extend it to do basic load tests using services like : https://browsermob.com/performance-testing

We are moving more towards this kind of testing internally, so def keen to help where I can...

Nice one! Thanks @slace for some great recommendations. 

I think the idea to use BDD is a great one, as as you say, it means pretty much anybody can write up a scenario. In my mind, I'm thinking we could have a bit of a brain storming session bashing out as many scenarios as possible creating a backlog of tests that need to be written, then anybody who wants to, can come along and claim one and write the test for it. Maybe something like https://trello.com would be good for this? We could share a workspace, and have everybody submit scenarios (slides in trello money) into an ideas column, then anybody can just grab one and move it into an in progress column, and finally into a done column when finished. Or, we could just use a google doc spreadsheet.

What do people think?

I think once we have the beginnings of a decent backlog, we can then come up with some standards for how to write the tests (based upon @slace's current efforts) and a couple of us could also get a central selenium test server setup while the initial scenarios are being written up.

Matt

Matt, I like the idea of using Trello, that would suit this kind of thing really well! I can think of quite a few test already. Someone will have to keep an eye out for duplicates/similar tests though! Maybe have a google spreadsheet initially, and then have someone compile them onto the Trello board?

Where's the LIKE button?

Sounds like a great idea to me.  We already use Trello to, its a great tool and would fit this well.  As a (not so) technical user, I'm happy to help out with scenarios where necessary from a user perspective.

#H5YR :) 

So I managed to have a play last week with Selenium + SpecFlow as per @Slace's suggestion, and I think it works really well. I reckon this is definitely the way to go.

So what do people think would be the best plan of action then for working out a virtual community hackathon to bash out some features / scenario definitions? I think it could be cool to try a google hangout, though I think these can only handle up to 9 people. Anyone got any other suggestions? Other than that, I guess it's just trying to pick a time that allows as many people as possible to participate (@Slace being the most awkward one :P)

Matt

Hi Matt

We have a gotomeeting account that allows upto 15 people which you can use if you want?

Let me know...

Cheers

@Slace I just thought we'd get a lot more across in a lot less time if we did it via voice rather than typing. It might be that we start off with a voice call, then split off to write a bunch of features / scenarios, and come back together as a group towards the end. Having a group IRC channel open during the whole thing though wouldn't be a bad idea aswell. I'm open to suggestions though (I am just making it up as I go along at the moment =).

Matt

slace wrote:

Why not just irc?

 

IRC is for geeks and don't allow screensharing, video and voice does it ;-)

/n

@matt this is also pretty good http://www.yuuguu.com/pricing/web-conferencing

up to 30 people can join....

Let's stop worrying about tools and steer the conversation in the direction of *doing*. GotoMeeting will do just fine and we're several that have accounts that accommodate enough people, now on to next step.

GoToMeeting is free for the project too, because there's already enough accounts in place. And you don't get things done if there's more than 15 in the initial conversations anyway.

Should we move back to the topic of actually get this going rather than being stubborn on tools? In the end it's *never* the tool that gets the work done - it's whether they're used.

Agreed,

So I'll look to put out a http://doodle.com/ page on twitter to see what dates are best for people. In my head, a day long Friday event would be perfect, more specifically looking at Friday the 2nd of Dec which would give us 2 weeks to drum up interest, and let people free up the day. But a weekend would work for me equally as well.

Matt

ok, so I've put an event page up here:

http://our.umbraco.org/events/virtual-umbraco-hackathon

and a doodle page here for people to vote for the best days to suite them (Sunday 4th the current best):

http://www.doodle.com/tc4b4tup24yez8uq

Have your say.

Matt

I think this is a great idea.

The only thing I'd throw in is that because the back office is partly developed, it would not be bdd so much as automated ui (regression?) testing.

I'm not saying this to be a semantic pedant (and I am pretty new to bdd myself), more that bdd can be incredibly time consuming and so it may be possible to save time/effort by focussing on the right things. I don't claim to know what they are ;-)

So, I've set the date as Friday December the 16th.

Would be great if everyone could sign up to the event on the umbraco event page if you can attend:

http://our.umbraco.org/events/virtual-umbraco-hackathon

And if you wouldn't mind retweeting:

http://www.twitter.com/mattbrailsford/status/139286050597113856

Cheers guys. 

Matt

Seeing as I wont be joining the call I'll refer you to my initial story set: http://jupiter.umbraco.org/User-Stories.ashx

@slace really sorry you can't make it as I feel this is as much your baby as it is mine, but thanks for the sharing everything you've done so far. It'll make for a great starting point. #h5yr

Made a good start after the virtual hackathon on ui testing the installer,added to Matts repository at http://bit.ly/tTXO0z in case anyone is interested.

#H5YR

From: slace

Why would this be in a separate repository? Why is it not in the codeplex one?

It doesn’t make sense to be splitting code across multiple repositories and providers

Aaron Powell
MVP - Internet Explorer (Development) | FunnelWeb Team Member

http://apowell.me | http://twitter.com/slace | Skype: aaron.l.powell | Github | BitBucket

From: js4xon [email removed]
Sent: Tuesday, 20 December 2011 9:48 AM
To: Aaron Powell
Subject: Re: Automated web UI tests [umbraco5contrib:279728]

From: js4xon

Made a good start after the virtual hackathon on ui testing the installer,added to Matts repository at http://bit.ly/tTXO0z in case anyone is interested.

I thought I should update this with what I was discussing with Matt the other day on skype.

Putting this into another repository, be that a bitbucket one or in here isn't a good IMO because of:

• Disconnect from the main project, your tests will only ever reflect the code that was last 'synced' to the external repository. If the "owner" of the sync is unavailable for a period of time this can result in a big disconnect and your tests risk becoming invalid
• You loose a feedback cycle in the core project. One of the reasons you want automated tests is so that you make a change to the backoffice UI and find out quickly if you broke something. Separate repositories breaks this workflow meaning you wont find out about your break until you're moved on from that task (potentially)
• It's difficult to integrate into a build process if it's not in the main repository. This is what I think really needs to be done with the integration tests, they are run as part of the build process (what ever produces nightlies.umbraco.org). I wouldn't recommend it run on every build, that's not viable (since these kinds of tests take a long time). Separate repositories makes this a very challenging task to setup

The principle argument against having a single repository that I've had people present is that it's "easier" to open up the contrib project (or a private repository) than the core repository. I think that this argument is a bit misguided given the use of Mercurial as Umbraco's SCM. With Mercurial and CodePlex's fork system it's very easy to create a fork of the repository and work against that resulting in a pull request. CodePlex also supports multiple collaborators on a fork which can also improve the situation. While it's true that you need to ensure syncing into a fork to keep it up to do I believe that that is a simpler sync process than into a separate repository.