Brandon Stirnaman

FluentAutomation Creator
Simple, fluent automated testing for web applications.

F14N Founder
Fast, easy automation test creation, execution and management.

RELEASED: RemoteCommand API

So we’ve talked about it before.. But now I’m ready to call it ‘released’ and I want to give this some more exposure because I think it will serve as the main testing method for most users very soon.

Scalability of UI automation tests is a big concern for development teams. The primary point of the RemoteCommand API is to allow tests to be run across multiple machines at once. There are a few solutions to this specific problem already including Selenium Server/Selenium Grid. Our alternative solution, to keep our abstraction in place, is to utilize the pieces that already exist in the Fluent framework to create our own test execution engine: FluentAutomation.Server.

As a proof-of-concept (PoC), we took the current bits and built a test coordinator application; coupled with a custom EC2 AMI and Amazon’s CloudWatch metrics, the PoC system scales up or down automatically based on tests in progress. It has worked better than we’d ever thought it would in this environment. Perfectly capable of testing on EC2 Micro images, we can utilize cost-effective services to test our applications.

With this, I want to talk about some of the limitations to this approach… First, the basics of how remote execution works. For managed bits such as the .NET API, you write your Fluent code as normal integration tests bundled in with the rest of your project. Very tidy and clean, an example test such as ..

I.Expect.Text("Blog").In("#menu-item-8 a");  
I.Click("#menu-item-8 a");  
I.Expect.Text("News").In("li.cat-item.cat-item-6 a");  

.. that opens a page, makes a few assertions, takes an action or two and completes; feels very natural. In either normal or remote execution modes this test will perform precisely the same steps in the same ways. Sometimes when writing a test you want to take an action only in the event of a failure. We see users commonly catching AssertExceptions and making decisions based on that. An example ..

try {  
    I.Expect.Text("Blogs").In("#menu-item-8 a");
    I.Click("#menu-item-8 a");
} catch (AssertException) {
    I.Expect.Text("Docs").In("#menu-item-236 a");
    I.Click("#menu-item-236 a");

I.Expect("News").In("li.cat-item.cat-item-6 a");  

.. that opens the page, asserts that the Blogs link exists and clicks it. But if its missing for some reason, perhaps under maintenance, continue on and click on Docs instead. Both of these paths eventually test that the “News” list item exists on the resulting pages. This is essentially branching within a single test based on the pages current state. When run locally, it will behave as the user expects. When parsed for remote execution, the test methods in the catch will never be hit.

With the asynchronous nature of assertions and actions while executing via FluentAutomation.Server, inline code blocks such as the try { } catch() {} can’t be serialized and sent to the server to make that decision. This is the core reason that we have avoided allowing the user to retrieve data from the page into local variables. In general I’d advise against this type of logic because it makes each test more complex than they need to be. If there is an issue you can’t test around with this type of code, its a shortcoming in our framework and I’d love to help you solve these problems. Hit me up via email, comments on the blog or create issues on our GitHub site and we’ll figure out the best path to get you moving forward.

Along with the cons, comes some pretty striking pros in my opinion:

Single test, multiple browser execution is baked into the RemoteCommand API at the lowest level. Know exactly how your test executes against any browser without having to mess with different drivers, tools or any of that.

Any automation framework with a compatible AutomationProvider implementation (Selenium and WatiN presently) can be used in this manner. No changes to your code required.

Any language that can form valid JSON is supported. First class providers are being built for .NET, JavaScript, CoffeeScript (my new favorite toy), PHP and others. Looking for people willing to help expand the use of the Fluent API to other languages.

Test recording! The contrib project by Paul Zumbrun, FluentAutomation Recorder allows tests to be recorded and played back via the chrome extension utilizing the RemoteCommand API.

I think we have a fairly compelling automation story playing out here, but I’d love to hear more feedback from people using or interested in using the Fluent API. The next step for Paul and I is to actually build the documentation we’ve been promising for months. This will take some time to become ‘good’ but I know our users need it.

Thanks for the support!

Recorder updates: Expect Builder and more to come

Guest post by @pzumbrun Time for an update! It’s been a while but Brandon and I are back actively developing after the holidays, and we have some fun stuff coming up for Fluent Automation. I just finished a working draft of the Expect Builder for the Fluent Automation Recorder… Read More

Still going..

I’ve received a few emails wondering about progress of the Fluent API. Just throwing this out there so you all know, I am still working on it regularly. It has been a very busy few weeks to prepare for some vacation time I’m taking now with family in… Read More

Continued work.. and some questions for the users

I’ve been continuing work on making the RemoteCommand API awesome, including better debugging a test in progress against a remote box, proper error handling and integrating into your Continuous Integration processes. This has driven some questions as to where we should focus first as far as verifying compatibility with… Read More

Introducing the FluentAutomation Recorder

Guest post by @pzumbrun In a previous post, Brandon introduced the RemoteCommand API, a REST endpoint that accepts and runs Fluent Automation tests serialised into JSON. Having a portable format for tests has let us start to branch the project out into a few interesting tools outside the .NET world… Read More

Runtime Provider Selection

Recently we added a great little change that allows you to stop compiling your tests directly against the Providers FluentTest class implementation. Now you can do this: [TestClass] public class ProviderLoadingTests : FluentAutomation.API.FluentTest { [TestMethod] public void Test() { I.Open("http://www.google.com/"); } } Notice there is no reference to… Read More