For those who are closely following - this post has been edited for clarity of content and language ;) - ohh and to fit your screen
Selenium is a nice and slim tool for testing. If I may start with the history it was built by Jason Huggins and Paul Gross* from ThoughtWorks and then opensourced on http://www.openqa.com/selenium (List of all the contributors can be found there too).There are 2 versions of Selenium - Selenium Remote Control and Selenium core. Selenium core is the essence of Selenium where as Selenium RC packages the core to enable you to write tests in the language of your choice. There is a recorder (and much more) with Firefox which allows you to point and click and create your tests in any of the languages that Selenium RC supports.
All these niceties made Selenium the choice of testing tool at my current project.
Since it was designed by developers we also thought that Selenium was easy to automate. But as it turned out not so much. Here is the issues we faced and what we did to fix it.
We tried to run Selenium in IE just to realise that we have a same origin policy problem - we had our tests open http://localhost/ and then navigate to http://location.localhost/. Yes you guessed it our domain name changed during the test run. This is what Selenium cannot handle. As per the Same Origin Policy your javascript(Selenium tests/testrunner) cannot run/access content that is on a different domain, protocol or port. If you still would like to use Selenium you have to use the experimental modes (chrome for Firefox or hta for IE) which are really beta and flaky.
So having thrown that out of consideration we had to run tests from the same machine as the application under test(AUT).
We decided to go with Selenium core.(since Selenium RC for some reason stripped our pages of css - ugh?)
To do this we installed Selenium core in our application code ie in the public directory(since ours is a ruby app - for a regular java app in apache tomcat for example webapps would be the equivalent location). Then we could run TestRunner with our TestSuite and fire the tests.
The next step was to automate all this testing so that it runs periodically and runs as part of a build so that your application is functionally tested continuously.
Some details you would like to note before we dig deeper.Selenium core just runs as a url :
Our URL was as follows
http://localhost:3000/selenium/core/TestRunner.html?test=../tests/TestSuite.html&auto=true&resultsUrl=http://localhost:3000/postResults?browser=ie6
This URL does a few things -
- it specifies a test suite TestSuite.html
- that the tests are going to run automatically (basically as soon as you point your browser to the above URL that is)
- they are going to post results to a results URL.
All this is well documented. What is not documented clearly is that one needs to write a POST handler (our POST handler is at http://localhost:3000/postResults). This could be the hard one if a non-developer is trying to set this thing up.
I have written a quick and dirty one in ruby which I will post here for reference
class PostResultsController < ApplicationController
def index
browser_name = params["browser"]
time = Time.now
timestamp = "#{time.month}/#{time.day}/#{time.year} #{time.hour}:#{time.min}:#{time.sec}"
results_file_name = "public/results/#{browser_name}(#{timestamp})-results.html"
f = File.new(results_file_name, "w+")
# f.write(params)
f.write("<html><LINK href='selenium-test.css' type=text/css rel=stylesheet><body>")
result = params["result"]
f.write("<table>")
f.write("<tr>")
f.write("<td>Result : #{result}</td>")
numTestTotal = params["numTestTotal"]
f.write("<td>Tests Total : #{numTestTotal}</td>")
f.write("</tr>")
f.write("<tr>")
totalTime = params["totalTime"]
f.write("<td>Total Time : #{totalTime}secs.</td>")
f.write("</tr>")
f.write("<tr>")
num_test_passes = params["numTestPasses"]
f.write("<td>Tests Passed : #{num_test_passes}</td>")
num_command_passes = params["numCommandPasses"]
f.write("<td>Commands Passed : #{num_command_passes}</td>")
f.write("</tr>")
f.write("<tr>")
numTestFailures = params["numTestFailures"]
f.write("<td>Tests Failed : #{numTestFailures}</td>")
numCommandFailures = params["numCommandFailures"]
f.write("<td>Commands Failed : #{numCommandFailures}</td>")
numCommandErrors = params["numCommandErrors"]
f.write("<td>Commands Errored : #{numCommandErrors}</td>")
f.write("</tr>")
f.write("</table>")
totalTests = params["numTestTotal"].to_i - 1
for i in 1..totalTests
index_str = 'testTable.' + i.to_s
testTable = params[index_str]
f.write(testTable)
end
f.write("</body></html>")
f.close
end
end
Another thing that I did not allude to was the browser=ie6 parameter on the results URL. This was another requirement for our scripts -that they be run in 6 different browsers and post results for us to look at. That means I have to know for each test what I am running against and this extra parameter on the resultsURL would be nice to have.
To make sure Selenium understood it I had to change selenium-test-runner.js as follows
add the following to HtmlTestRunnerControlPanel prototype
getBrowser: function() {
return this._getQueryParameter("browser");
},
and then use it in TestResult prototype like so
//added getBrowser so that we can run tests on a browser and send the browser name with results.
var browser = this.controlPanel.getBrowser();
If this is not enough Selenium also adds a constraint on how you automate using cruisecontrol. The only way to do that is to fire the browser for test with the URL like above. So cruisecontrol has to take the help of a batch file to automate this - which is less than ideal.
Cruisecontrol by design waits for the build tasks to complete(in this case the batch file). But as the tests run inside the browser Cruise has no way to know after they have finished as browser does not change state.
We had to do something like this in our batch file
set PATH = %PATH%;path_to_ie
start iexplore.exe selenium_url_like_above
rem approx wait of 60secs hack
ping -n 61 127.0.0.1 > nul
taskkill /F /IM iexplore.exe
This batch file above will set the path so that IE can be run. then 'start' the browser in a separate process then wait for 60 secs and then kill IE. Once IE is killed the batch process is over and cruise can wakeup.
You can automate running Selenium tests for any browser as shown above. Just make sure your ping interval is set to appropriate time so that the browser is closed only after the tests are done not before.
(A good reference for batch files and command prompt automation is - http://www.allenware.com/icsw/icswidx.htm. I found the info for 'start' and the hack for sleep using ping here)
2 more hacks are on my mind but I will post in a later discussion.
Phew! This has been my first blog post in a long time. Thanks for the patience.
1 comments:
Selenium remote control (or "selenium rc") provides a way to start and stop browser sessions by means of your selenium rc test.
Selenium rc also solves the "same origin" problem when run in proxy injection mode (but note that this is a newish feature which is solid only in the nightly build; a new release is expected in the next month or so).
See http://www.openqa.org/selenium-rc/ for more info.
Post a Comment