beta

After struggliing with running Selenium with Cruisecontrol I hit on a new solution today and am now able to run Selenium and post results more consistently than before. This way of doing it has the advantages that I did not need to run a rails app to collect results. I have managed to do it with Webrick. Since it was my first time working with Webrick it took me lot longer than I thought.

To start with, here is the problem I was facing. I was trying to do the Cruisecontrol with Selenium all over again on a fresh setup. Once I installed Cruisecontrol and Java(u have to have Java - I had ruby already). I was ready to run the same batch file I had. But I was not able to post the results. Installing a rails application with all the required gems seemed painful and an overkill for the purpose. So I started poking around with WEBrick. Some documentation on WEBrick website got me started and I could send a request to the server. But what I was not able to do was read the request parameters. All the documentation I looked up talked about the same kinds of examples as on the WEBrick website. They all mentioned and showed with an example how to handle simple requests and responses. Printing out the request did not help as it did not have parameters anywhere. On further digging I found some code in the PickAxe book (edition 2 page 248)

where the example spelled out exactly where the query object was 'req.query'. That was where one will find the parameters hash that Selenium populates when it posts results.

Here is a reference of the code that achieved the desired result.

require 'webrick'

if ARGV.empty?
  p "Usage: ruby post_results.rb browser_name"
  p "Example: ruby post_results.rb iexplore" 
  exit(0)
end

@browser = ARGV[0]
url = APPLICATION_UNDER_TEST #"http://www.yourapp.com"
testSuite=PATH_OF_TESTSUITE_RELATIVE_TO_SELENIUM
resultsURL= "http://localhost:2000/postResults" #the server that gathers the results..
testRunner="#{url}/selenium/core/TestRunner.html?test=#{testSuite}&auto=true&resultsUrl=#{resultsURL}"
command = "#{@browser} \"#{testRunner}\""

system "start #{command}" #DOS batch command start starts the command in a new thread

include WEBrick

s = HTTPServer.new( :Port => 2000 )

# HTTPServer#mount_proc(path){req, res ...}
# You can mount also a block by `mount_proc'.
# This block is called when GET or POST.
def save_to_file(params)
Dir.mkdir("results") unless FileTest.exist?("results")
Dir.chdir("results")
Dir.mkdir(@browser) unless FileTest.exist?(@browser)
Dir.chdir(@browser)
results_file_name = "selenium-results.html"

save to file as before

end

s.mount_proc("/postResults"){|req, res|
  params = req.query
  results_file_name = save_to_file(params)
  s.shutdown
  system "TASKKILL /F /IM #{@browser}.exe" #KILL the browser since selenium is done.
}

trap("INT"){ s.shutdown }
s.start

For those who are closely following - this post has been edited for clarity of content and language ;) - ohh and to fit your screen

Selenium is a nice and slim tool for testing. If I may start with the history it was built by Jason Huggins and Paul Gross* from ThoughtWorks and then opensourced on http://www.openqa.com/selenium (List of all the contributors can be found there too).There are 2 versions of Selenium - Selenium Remote Control and Selenium core. Selenium core is the essence of Selenium where as Selenium RC packages the core to enable you to write tests in the language of your choice. There is a recorder (and much more) with Firefox which allows you to point and click and create your tests in any of the languages that Selenium RC supports.

All these niceties made Selenium the choice of testing tool at my current project.

Since it was designed by developers we also thought that Selenium was easy to automate. But as it turned out not so much. Here is the issues we faced and what we did to fix it.

We tried to run Selenium in IE just to realise that we have a same origin policy problem - we had our tests open http://localhost/ and then navigate to http://location.localhost/. Yes you guessed it our domain name changed during the test run. This is what Selenium cannot handle. As per the Same Origin Policy your javascript(Selenium tests/testrunner) cannot run/access content that is on a different domain, protocol or port. If you still would like to use Selenium you have to use the experimental modes (chrome for Firefox or hta for IE) which are really beta and flaky.

So having thrown that out of consideration we had to run tests from the same machine as the application under test(AUT).

We decided to go with Selenium core.(since Selenium RC for some reason stripped our pages of css - ugh?)

To do this we installed Selenium core in our application code ie in the public directory(since ours is a ruby app - for a regular java app in apache tomcat for example webapps would be the equivalent location). Then we could run TestRunner with our TestSuite and fire the tests.

The next step was to automate all this testing so that it runs periodically and runs as part of a build so that your application is functionally tested continuously.

Some details you would like to note before we dig deeper.

Selenium core just runs as a url :

Our URL was as follows

http://localhost:3000/selenium/core/TestRunner.html?test=../tests/TestSuite.html&auto=true&resultsUrl=http://localhost:3000/postResults?browser=ie6

This URL does a few things -

1. it specifies a test suite TestSuite.html
2. that the tests are going to run automatically (basically as soon as you point your browser to the above URL that is)
3. they are going to post results to a results URL.

All this is well documented. What is not documented clearly is that one needs to write a POST handler (our POST handler is at http://localhost:3000/postResults). This could be the hard one if a non-developer is trying to set this thing up.

I have written a quick and dirty one in ruby which I will post here for reference

class PostResultsController < ApplicationController
def index
browser_name = params["browser"]
time = Time.now
timestamp = "#{time.month}/#{time.day}/#{time.year} #{time.hour}:#{time.min}:#{time.sec}"
results_file_name = "public/results/#{browser_name}(#{timestamp})-results.html"

f = File.new(results_file_name, "w+")
# f.write(params)

f.write("<html><LINK href='selenium-test.css' type=text/css rel=stylesheet><body>")
result = params["result"]
f.write("<table>")

f.write("<tr>")
f.write("<td>Result : #{result}</td>")
numTestTotal = params["numTestTotal"]
f.write("<td>Tests Total : #{numTestTotal}</td>")
f.write("</tr>")

f.write("<tr>")
totalTime = params["totalTime"]
f.write("<td>Total Time : #{totalTime}secs.</td>")
f.write("</tr>")

f.write("<tr>")
num_test_passes = params["numTestPasses"]
f.write("<td>Tests Passed : #{num_test_passes}</td>")
num_command_passes = params["numCommandPasses"]
f.write("<td>Commands Passed : #{num_command_passes}</td>")
f.write("</tr>")

f.write("<tr>")
numTestFailures = params["numTestFailures"]
f.write("<td>Tests Failed : #{numTestFailures}</td>")
numCommandFailures = params["numCommandFailures"]
f.write("<td>Commands Failed : #{numCommandFailures}</td>")
numCommandErrors = params["numCommandErrors"]
f.write("<td>Commands Errored : #{numCommandErrors}</td>")
f.write("</tr>")
f.write("</table>")
totalTests = params["numTestTotal"].to_i - 1
for i in 1..totalTests
index_str = 'testTable.' + i.to_s
testTable = params[index_str]
f.write(testTable)
end
f.write("</body></html>")
f.close
end

end

Another thing that I did not allude to was the browser=ie6 parameter on the results URL. This was another requirement for our scripts -that they be run in 6 different browsers and post results for us to look at. That means I have to know for each test what I am running against and this extra parameter on the resultsURL would be nice to have.

To make sure Selenium understood it I had to change selenium-test-runner.js as follows

add the following to HtmlTestRunnerControlPanel prototype

getBrowser: function() {
return this._getQueryParameter("browser");
},

and then use it in TestResult prototype like so

//added getBrowser so that we can run tests on a browser and send the browser name with results.
var browser = this.controlPanel.getBrowser();

If this is not enough Selenium also adds a constraint on how you automate using cruisecontrol. The only way to do that is to fire the browser for test with the URL like above. So cruisecontrol has to take the help of a batch file to automate this - which is less than ideal.

Cruisecontrol by design waits for the build tasks to complete(in this case the batch file). But as the tests run inside the browser Cruise has no way to know after they have finished as browser does not change state.

We had to do something like this in our batch file

set PATH = %PATH%;path_to_ie

start iexplore.exe selenium_url_like_above

rem approx wait of 60secs hack

ping -n 61 127.0.0.1 > nul

taskkill /F /IM iexplore.exe

This batch file above will set the path so that IE can be run. then 'start' the browser in a separate process then wait for 60 secs and then kill IE. Once IE is killed the batch process is over and cruise can wakeup.

You can automate running Selenium tests for any browser as shown above. Just make sure your ping interval is set to appropriate time so that the browser is closed only after the tests are done not before.

(A good reference for batch files and command prompt automation is - http://www.allenware.com/icsw/icswidx.htm. I found the info for 'start' and the hack for sleep using ping here)

2 more hacks are on my mind but I will post in a later discussion.

Phew! This has been my first blog post in a long time. Thanks for the patience.