GTAC 07: Chow & Etchebehere: Bldg Framework Around Selenium

ALAN: Hi, everybody.

So how are you feeling about the day so far?

Well, we have one talk left this evening
and it's a good one.
So Apple and Santiago both work at Google on some of our
IS applications.
And they're going to talk about building a flexible and
extensible framework around Selenium.
And I am really looking forward to this one.
APPLE CHOW: Thanks, Alan.

So welcome to the last talk of today.
Today we can talk about how to build a flexible and
extensible framework around Selenium.
We want to use our experience as an example of how people
can build such a framework.
So that also for those of you who are getting started with
Selenium and building test frameworks, hopefully this
will give you some new ideas on how to
kickstart your project.
Our framework's name is Ringo.
As you might have guessed, most of our team members are
fans of the Beatles.
And that's also where our motto came from.
Our motto is, "It's getting better all the
time." So let's kickstart.
So this the outline of today's presentation.
First I'm going to talk about the challenges that we face in
testing web applications.
And then I'm going to talk about how our design decisions
address those challenges and thus enable
us to meet our goals.
And then we're going to talk about the framework
architecture as well as the framework's features.
And we'll conclude today's presentation by going over how
Ringo has helped us in dealing with issues that have come up
along the way.
And at the end there will be a Q and A session.
So first, in the world of testing web applications, we
have faced many challenges.
For example, we are working with agile projects, which
means they have frequent requirement changes and
frequent releases, which means that on one hand we have to
deal with the frequent release cycles, running the regression
tests, and on the other hand, we have to keep up with adding
tests for the new features that are
coming out every week.
Another challenge that we face is that we're dealing with
complex web applications.
And so there's a lot of Ajax and dynamic HTML in our
applications.
And from the user's point of view, those technologies make
the applications more usable and interactive.
But from the testing point of view, it's actually making it
harder to test because of the asynchronous nature of those
technologies.
And also, we're dealing with limited resources, working on
multiple browsers and platforms. I guess all of you
have similar stories.
So in order to deal with those challenges we came up with a
few design principles.
First, we decided to pool together our resources from
some of the projects we have to build a common framework so
that we can reuse the same framework across different
projects, thus saving us time and resources.
And we can come up with a better solution.
And, as for the design principles, the first and
foremost is simplicity.
So we want to isolate anything that's not test case specific
from the test case, so that the test case can be more
resilient to application changes.
And in order to do that, we are taking
the following actions.
First, we are isolating all the application-specific logic
into an application client class.
And again, from the past projects, one observation that
we made is that a lot of times when you fill forms or when
you interact with tables, a lot of times you end up
writing really specific and reusable code.
So we want to be able to extract that out and have the
framework provide utilities for you to fill in forms, for
you to reading from or writing to tables.
And also, one other challenge in testing is that not knowing
an action has completed, especially when it comes to
Ajax applications--
and hard-coded sleeps are error-prone.
And it not only pollutes your code-based error-prone.
And because it can fail due to external factors such as the
machine speed, congested network, or the machine's
overloaded and things like that.
So we would also be able to provide utilities for dealing
with those timing conditions by the framework, so that the
test case writer doesn't have to deal with those timing
conditions.
And another thing that we found from past experience is
that a lot of times when you interact with specific UI
objects, you're required to use specific methods.
For example, interacting with a link, you say click.
And when you type in the text where you say, "type." So we
want to be able to provide a uniform way of accessing those
UI elements.
And that also is the foundation for many other
features of our framework.
As for flexibility, we want to be able to provide the test
case user the ability to group the test cases into whichever
way that makes sense to them and be able to control what
tests you run during run time.
And we also provide place holders for an end user to
provide dynamic configuration data or just data
for the test case.
And we also know that we want to have the features as
generic as possible.
But on the other hand, we know that today's web application,
you're no longer dealing with a simple HTML element.
Because, think Google Maps, think those iGoogle, think
those drag-and-drop advanced controls.
So we want to also provide the end users ability to extend
and customize the interaction with the particular UI object
that they have for the application into whichever way
that they want to customize it in, because we don't know what
are the possibilities.
And as for maintainability, we want to be able to externalize
anything that changes often into external files.
And also, one other thing that we learned from past
frameworks is that the data tend to scatter across the
different test cases.
And then it gets larger and larger, and you don't even
know what kind of data you're using for the different test
cases anymore.
And it gets harder to maintain, because there's a
lot of redundancy of data.
And so when something changes, it's not really easy to make
changes because it's all scattered out.
So we want to be able to take that into consideration and
eliminate the data duplication issue.
One other design consideration we have is that we want to
support data-driven tests by providing the ability to run
the same tests with different sets of data, so that you take
full advantage of automation.
And so you can run the different permutations.
And also, we provide a data generator for the end users to
be able to substitute in dynamically generated but
structured data, like an email address or like a phone
number, those kind of structured data, to that test
case as well.

We want to be able to leverage third party tools to provide
important functionality, so that we can save focus on
building abstraction layers and customize it to--
and then also integrate different pieces together to
fill our needs.
And there's a couple requirements or desirements
that we have when it comes to the tool selection criteria.
First we want to be able to handle all those events, Ajax
controls or dynamic HTML, JavaScript on our web pages.
Another thing that we consider is that we are working with a
mostly J2EE web applications, so it would be nice if we can
use the same technology, or language technology or tools,
as the developers, because that will allow for tighter
collaboration among the two groups.
So that's why we picked Selenium.
For those of you who don't know, it's an open sourced web
testing tool that uses JavaScript, basically to
control web application that you're testing.
So highlighted here are some of the features that we like a
lot, and also are the foundations, are the pieces
that's making use of by the framework family.
For example, there's the concept locators.
So this is a way for Selenium to identify an
element on a UI page.
And it supports different locator strategies, as I've
shown here.
So you can identify an element by name, ID,
DOM, xpath, et cetera.
So it's very flexible to identify any
element on a page.
Another feature that we really like a lot is the
waitForCondition.
So as shown in the example here-- so we run into a case
where, when the page first loaded, a button is disabled.
And then, after a few seconds, it becomes enabled.
And so waitForCondition is great in this case, because it
takes like a JavaScript expression and a timeout, and
it will run this, evaluate this JavaScript expression in
a loop until it's valid to use and until it times out.
So that is definitely more reliable
than hard-coded sleeps.
And another feature that we like is the getEval.
It's basically something that we can use to evaluate
JavaScript, whether you want to extend the Selenium
functionality by providing-- or your own JavaScript
function using user extension, or just by invoking JavaScript
functions from the web application you're testing,
this can be really useful.
As for the testing framework, we're using TestNG.
Basically, it's a JUnit like framework except that it
supports more flexible annotations.
But the single feature that really stands out about TestNG
is the data providers feature.
This is something that we make use of heavily to support for
data driven testing.
So what it does is that, basically, you can define it
to return either an iterate object or a two-dimensional
object array.
In this example, I'm showing it to return a two-dimensional
object array.
So each role in the two-dimensional object array
represents one call to the test method
that you define there.
And the second parameter, the number of columns, is mapped
to the different parameters for your test method.
So underneath a cover, TestNG will handle looping with the
different roles inside your data provider and then
invoking the method with the different
data that you provide.
So this is very powerful, because you can define the
data provider to be any way you like.
The logic is up to you, how to define it.
You can read from XML file, you can read from a database,
or you can use your data generator to provide those
dynamic values.
Another tool that we are using is Firebug.
We are using it as a development tool.
So it's something--
it's a file FAS extension where people normally use it
for inspecting UI elements on the page.
So the way that we use it is we kind of piggyback on top of
that, because we found one time consuming activity in web
automation is actually to be able to find out the different
locators of the different UI objects on the page.
And so we piggybacked on top of that to generate the UI
configuration file automatically for us.
So that saved us a lot of time.
And so, you can see the screen shot there.
And UImap is the extension that we added to that.
So this is the framework architecture.
The top layer, you can see, is the configuration layer.
So it consists of three objects, so the application
config, the UI map and the data map.
And the second layer is the heart of Ringo, because it
contains a lot of our UI abstractions.
It contains our Selenium rapper as well as utilities
and our data provider class.
And the two green areas are the two entry
points to our system.
So how an end user will use it is they will extend the base
client and base test. So for the application client, it's
something that contains the high level method that you use
for interacting with your application.
And that will extend from our base client, so we can make
use of our Selenium helper in all those configuration
objects that we have.
And for the base test, you can--
again, you implement our base--
you examine our base test to provide application specific
test. So the point I want to emphasize here is that the
application test class will make use of the application
client, so that doing so greatly simplifies the writing
a test case, because a lot of your important logic will be
located in your application client class.
So that makes the test case simpler to write.

So moving on, this is the application config file that
we have. So this is something that you can use to--
because you write down all the different parameter--
name, value, pairs, global configuration parameters for
your application.
So in this case, you can use any aperture in your name
value pairs, because if it's just using a
hash to pause this.
And the bolded area, the reference to the UI map and
the data map, so it pretty much
holds everything together.
And so here you can see the other global parameters, it's
just a global timeout that you define here, like five
minutes, and the URL and things like that.
So you can use this to define the different test
environments that you're testing against.
So UI object.
What's in UI object?
Basically, it's our abstraction for the UI element
on a webpage.
And it contains the read and write interface.
So read is for reading, like extracting
information from it.
And then write is for interacting with it.
So for example, in this example, I'm showing a text
field UI object.
So the read will be call and get text and then
write will be type.
And then if, on the other hand, if you talk about
length, the read will be, again, getting the text from
the link and write will be clicking on it and vice versa.
So having this uniform way of accessing UI objects--
it's very powerful.
Because if the UI object changes, then--
if the UI object type changes, maybe your test code doesn't
have to change because they have to extract this level of
abstraction.
And our framework provides all the defaults, implementations,
for the common UI objects.
And then you can also extend your own.
And then I think Santiago will go into that later.
And so, having this also provides the foundations for
some other features we have in the framework.
So UI map is something that we use to centralize the location
of UI elements on the page.
It contains one or more pages.
And each page contains pages inside, so it can have
embedded pages.
And a page doesn't really have to be a
literal page, a webpage.
Think of it as a container.
It's like a grouping of the UI elements that
you're dealing with.
So if you have a form in a page, maybe you want to
consider that a page object.
Or if you have different sections, you may consider
them different pages, because they made load at different
times and things like that.
And so inside, you can see that it contains
a list of UI elements.
And each UI object has a name which is a symbolic alias that
we are making use of inside the framework.
And then there's a type indicating what type of UI
object it is.
SOL is a locator, which is what Selenium uses for
interacting with it.

As for the data, we think of it as two different groups.
So a lot of times, the way that people use test data,
usually you have a default set of data profiles that's shared
amongst your test case.
And then your test case may want to reference to those,
and then overwrite one or two properties, and then to test
for a particular corner case for your test case.
So, that's how we model it in our data map.
So here and in the top section you have the different--
so you can categorize your global data into different
categories.
And then at the bottom you can see an example of a test case
kind of referencing to the common data using the name.
And then it overwrites one of the properties.
So this is actually a small example.
But think of a really big form, like a user sign-up
form, that contains a lot of data.
Using it this way greatly reduces data redundancy,
because then you can define your common data in one place
and have your data, test case specific data, point to that
and overwrite only the parameters that you need to
change and also have this in one place--
give you visibility as to what kind of data you're using.
Because you don't want to be using random data that's
doesn't really add value for your test case.

And one common action for web testing is
filling forms, right?
So a lot of times what you do you as--
so we make the observation of--
it doesn't matter which forms you're talking about, you can
kind of abstract out that there's a
commonality amongst them.
So basically what you're doing is that you are looking at all
the UI elements inside a form, and you're looping through
them and filling out the values one by one, right?
And so, that's why we abstract out this thing into our helper
class and we have a field format method.
So basically what it does is that it takes a page object
and a data object and then for each of the UI objects that
you define in the page, you find out the data objects, the
corresponding data value to be filled in, and then it just
fills each of the UI elements.
It calls right of each UI object that we have.
So again, having it like this makes the test case a lot more
maintainable and simple to write, because you made this
use fill form and passing in your paste object and your
data object.
And then, if something changes, probably all you need
to do is update your UI map and your data map and your
test case doesn't really have to worry about it, because
it's all handled underneath the cover.

So this is a picture of our application client class.
So like I mentioned earlier, you define all your high level
application APIs in this class.
And underneath it, you use our Selenium wrapper to interact
with the web application that you're testing.
And since a lot of the logic is located here, it simplifies
the logic of your test case and also makes
it easier to maintain.
And another point I want to make here is that--
the application client doesn't really mean that you can only
have one class, it's actually a subtree of classes.
If your application gets more complex, you can have a client
that has different page objects or component objects
and then have the client delegate actions to the
different components if needed.
So this is our application test object.
So what it does is it contains test cases that you define in
the test case repository.
And then it makes use of the high level APIs provided by
the client class.
And same thing, similar to the client class-- again here, it
doesn't mean that you can have one test class.
Again, you can have a global application test class, you
define common setup, take down or data providers or common
utility methods for your test case to reuse.
And you can have--
each feature can have their specific test case that can
extend from this one.
So it can have its own subtree here.
So this is a common scenario for test case flow for the
Google Search test. So here you can see that it just makes
a call to the client's search method passing in the data.
And then underneath it, it calls a field form.
And then the data comes back and then it just
asserts on the data.
So you can see that writing a test case is actually pretty
simple, because a lot of the logic is either handled by the
framework or it's handled by the client class.
And this shows the client and the test class side by side.
And so this is, again, an example for a
Google Search client.
Because we're testing with internal applications, we
can't really show you examples using those.
So we're just using some common Google applications as
examples here.
And again, you can see that the client class will contain
high level APIs such as basic search, and event search.
And then for your Google Search test cause, it will
call the client's APIs.
And another point I wanted to make here is that sometimes
you have a test case that you are testing different
applications, or integration of different applications.
And having this design is also flexible in that way because
your test case can use multiple clients.
And so that's one way that we find it really useful, because
we have multiple projects all using this approach.
And the test case can talk to the different client
application classes.
So now I'm going to hand this over to Santiago who is going
to talk about the more advanced
features of our framework.
SANTIAGO ETCHEBEHERE: OK.
Hi, everyone.
So as Apple was saying, I'm going to go through into a
little more detail in some of the features of the framework.
I'll talk about some Ajax handling utilities to deal
with some common problems in Ajax applications.
Custom UI objects and decorators to allow extending
the framework for particular needs, a class that we created
with a lot of helper functions to deal with tables--
which are usually along all applications, and place
holders, to allow dynamic configurations, the data
generator tool that creates random data and run time using
structured data like emails, phone numbers, zip codes, and
database support-- it's just basically a JDBC wrapper to
allow queries into the database and navigating the
results in case the test case requires some verifications
against a database.
So as working with Ajax, one of the hardest problems is
with asynchronous stuff.
So as [UNINTELLIGIBLE] was saying today, the
waitForCondition.

Also, I believe they allow a lot more flexibility.
And what we did in this case was extracted it from the test
case writer.
So a test case writer doesn't need to worry about that.
What we did is we have this tag that can be placed into
page elements.
That's the loaded condition tag.
And there, you basically type your JavaScript condition to
test for true using Selenium's waitForCondition method.
So imagine, as a hypothetical example, the iGoogle homepage,
which has all these gadgets that load and separate on
their own time, individually of each other.
And if we wanted to test just the search part, we don't need
to wait for the gadgets to be there in order to start
testing the search.
We just need to search, a text box and a search button, and
that's all.
That's it.
We don't have to wait for all the page to finish loading.
And in this case, what we do is we create a page tag in our
UI map, which represents the search form.
And this page will have a loaded condition that we'll
check for, in this case for a query, text box, and the
search button to be there.
And then, we can in our code just call our waitFor page to
load, which actually is our own implementation that uses
this loading condition of the page you specify.
And it will wait until that condition is
true in order to continue.
A useful thing of this is that since we have pages defining
sections of the whole page, we can have nested pages.
And each page might have its own loaded condition.
For instance, again, in the iGoogle search page, every
gadget there might have its own loaded condition, which
tells when it's finished loading.
And so, if we wanted to wait for the whole page, we can
call this method to wait for the container page, and it
will retrieve all the loaded conditions of the inner pages
and combine them and wait for all to be true
in order to go on.
This was pretty handy.
But we still have this waitFor something there.
And we didn't want to keep writing tests saying, OK,
click here, then wait there, then click there.
So we tried to simplify this by adding this reload trigger
attribute to our UI objects.
Basically, every--
or not every but many interactions with the
application causes a page to be loaded or another section
of the page to change.
And we need to wait for that in order to
continue with our tests.
As a simple example here, the next button in the search
results page in Google, if you click on it, it will reload
the page with the results for the next page.
And we want to specify that if we click in that link, we want
to wait until that page is loaded to continue testing.
So combining this with the loaded condition, we just
specify a page name as a reload trigger attribute.
So the framework will automatically look for that
page that's identified in the reload trigger.
And it will look for its loaded condition and wait for
that condition to be true in order to go on with the test.
This is useful not only for a whole page reloading but also
when a change in a value in a combo box or something may
trigger the reloading of an iframe or of a div, or a
section of the page that's identified by its page
element, which is its own loaded condition.
So you just declared this loaded condition, and the
reload trigger pointing to that page, and so the
framework will wait until that section is loaded before
continuing.
And by doing this, we abstracted all this logic for
the test case writer point of view.
So a test will be just, click on the next item and do
whatever you want to do next.
And all the waiting is hidden from the test case writer.
And also, it encourages the reuse of code because we don't
want to be specifying everywhere that we interact
with the same button or with the same link, then
we'd have to wait.
So we just specify it once and we click the link, click the
button and go forward.

So we provide--
the framework provides a lot of classes to handle regular
elements in the UI like text boxes, buttons, images, links.
But many times, the obligation defines the wrong kind of UI
controls that are actually Ajax stuff or just a different
behavior of regular control.
And so, one of the features we added to the framework was the
ability to create our own UI objects wrappers
or UI objects classes.
So, imagine in this example-- we have the Google Search
page, which you start typing a search and it will drop a list
where you can pick an element from the list of others'
searches similar to the one you're searching.
So if for some reason we wanted to test this element
and we required that this element had a specific
interaction that required us to wait after
each character type.
So we couldn't use a regular text box to
model these UI objects.
So we define our own suggest text box in this case.
And this text box, this UI object, will type the text
into the text box one letter at a time and waiting for the
specific time.
This is, again, just an example.
It doesn't make real sense in real life.
But it's a way of extending the framework for
filling your own needs.
And another interesting issue here is that you can add your
own methods to the UI objects.
You implement the UI object interface or extend the
abstract UI object class.
But you can also implement your own methods, like for
this UI object we could have a method like, select first item
from list or select the nth item from list. And that way
we keep a uniform way of interacting with our elements.
But we can also add new functionality to interact with
complex elements.
UI object decorators--
this is a feature I really like.
It's for cases where we need to add some behavior to a
specific element, or a specific
interaction with an element.
And it's another way of extending
the regular UI objects.
So imagine we have an example here that there's a text box
where you type a credit card number.
And once you move to the next control it will select the
type of the credit card from the list of types of credit
cards there.
The thing is, if you use a regular text box to just write
our credit card number in there, it won't trigger the
blur event which actually triggers the algorithm that
detects the type of the credit card.
So what we can do in this case is add this decorator, which
is just an implementation of the decorator pattern, which
consists of wrapping an object with another object, which
implements the same interface to the outside in order to
make it transparent and add behavior to the original
object that's being wrapped.
So for this case, for instance, we created this blur
decorator that--
it simply calls the right method of the UI object that
it's wrapping and then it triggers the blur event.
The great thing about this is that it's reusable.
Any other element that we need to trigger the blur event, for
instance, in this case, we can add the decorator to the
definition in the UI map and that will be handled
completely transparent to the test case
writer's point of view.
And another great thing is that we can change decorators
and wrap one on top of the other to keep adding behavior
to the standard UI object that we're wrapping.
And for instance, Ringo already provides some
decorators like waitForElement decorator that will wait until
the element is present for typing or reading or whatever
interaction is required.
And element present decorator for verifying that the element
is there before reading from it or writing into it and
returning null in case it's not there.
So it's another way of extending behavior and helping
avoid the duplication of code, because again we have this in
a declarative way in the UI map.
And it's reused along all the tests that
use the same objects.

So tables are something that are all over applications
which handle data usually and are usually complex to handle,
because they're not always HTML tables as we see them.
Tables might have layouts, rows added there just for
making things look nice and they're kind of complicated to
handle sometimes.
So we created a class to abstract some of these
functionalities.
And we called it table objects.
We have something we're calling our objects object.
And so, what we did here is we modeled this as if it was a
page, using a UI object for each cell of a given row.
So we use a placeholder in our locator definitions that will
be ultimately handled by the framework and that will
iterate through all the rows.
And we define every cell as the UI object.
So any behavior that can be abstracted in a UI object can
be abstracted into any cell of a table.
And then we provide some methods to-- for instance,
accessing cells by row number and column number, by reading
a whole row as a map of column name value, or accessing a row
with a column name and row number, which might be really
useful for tables that can't change the
order of the columns.
We can read or write from cells in case we have tables
that have drop-down lists or check boxes
that we need to click.
And there are other features that the table object
provides, like verifying that a value is present in all the
cells of a given column, retrieving a whole column.
And also there are other configuration items we can
specify, like if the data doesn't start at the first row
of the table, if the data doesn't end at the last row of
the table, if there are rows in between the actual data
rows that need to be skipped, we can specify that of other
parameters in order to make it easy to handle.
So from the test case writer point of view, it's just a
table object with data in there that you can access as
if it were a simple table.
And since we are using UI objects to model all the
cells, it doesn't really need to be an HTML table.
If it is displayed as a table and has a list of rows, or
something that can be modeled as a table, we can handle it
by abstracting everything as UI objects and locators.
And anything that can be identified by locators can be
considered a cell of the table.
And if we can iterate through them, then we can
read it as a table.

So the last feature I'm going to talk about is placeholders.
This is just a way for having more
flexible integration files.
We can have placeholders both in general application
configuration files as well as in our data maps or UI maps.
So we can define-- the framework already defines some
standard placeholders like, if the placeholders start with
the nth prefix, it will read an environment variable and
replace the placeholder with that variable.
If it starts with a sys.prefix, it will be a
system variable passed through the JVM.
You can create any placeholders you want, and
then in run time, tell the framework to replace a given
placeholder with a given value.
You can also--
time stamp is a standard placeholder.
And we can make use of the data generator tool through
the placeholders, which is very useful when you have to
create dynamic data for creating an element of the
whatever application you're testing.
So for instance, the data generator can create strings
of a given length with random characters, email addresses,
phone numbers, zip codes, any other pattern that you can
specify in regular expression.
And as for providing extensibility, you can define
your own placeholder handler classes.
If you want to create your own placeholder to replace it with
a value extracted from wherever you want, for
whatever you want, you just create your own placeholder
handler class.
And you can use that for your configuration files.
So summing up a little bit, how is the automation process
using the Ringo framework?
Basically, what we do when we write our test cases is we
first find the application configuration, like the url of
the application, the process we're going to test, and the
platforms, and any other general parameters, we'd find
that in our application config file.
Then we create what we call the UI map, which is this
collection of pages with UI objects inside.
We actually have a sample worth mentioning.
This Firebug extension that allows us to go through the
page, clicking on the elements, and then create at
least a skeleton of the UI map, and then we can extend it
by hand if we need to.
Having the UI map we can create the client class which
is, as Apple was stating today, does that actually
interact with the application, and providing to the test case
writer an API of something much higher level, like
perform a search in the Google Search example.

Then we define data.
Then we usually try to separate it in a common set of
data that can be reused across many tests.
And then, for specific tests we override that data if we
need to, or we declare our own specific data
for our test case.
And finally, we just implement our test cases, which are
simple TestNG classes annotated with TestNG
annotations and the data providers that will do simple
actions defined by the client, and retrieve results and
verify that the results are what are expected.
So the test case pretty much looks like the specification.
If there was a test case specification or a description
of the test case, when you see the test case, it
just looks like that.
It's like, perform a search, get results, verify that the
results are correct.
It's much simpler to write.
And we have common actions abstracting inside our client
classes, so we can reuse them across test cases.
So I'm going to go through just a small set of examples
of how Ringo was useful for us.
One of the things that more commonly happens
is changes in UI.
There are many types of changes.
Here I'm illustrating just a small example
of a case we had.
We had this text box to enter a country name.
And in one release it was changed to use a drop-down
list, to select a country name from the list. And all we had
to do to have our test working was just change the UI object
decoration--
change it from being a text object to
being a select object.
And the client class didn't have to change anything.
The test case class didn't have to change at all.
And there were more than one test cases that were using
this control, and they were all affected with the change,
but they were all solved by just changing the XML file.
Another case was the case when we had to implement our own UI
object to handle--
in this case, this multiple selection list. We had two
constraints in this case.
We didn't know the values from beforehand, so the values were
generated by the applications, and we needed to select a
given number of items randomly.
So we created our own random multiple selection UI object,
which actually extends to regular select UI object.
And what we do is, in the write method, we pass it the
number of items to select and it will retrieve all the
possible options and randomly select that number of options.
So again, this was a case where we needed to extend the
core functional idea of our framework.
And it was handy because we used this control in many test
cases and in more than one section of the application.
So we just needed to implement this once and then we could
reuse it everywhere we needed to.
And another case I'm going to show, to finish, today is a
case where we started--
the application started to handle i18n data.
So we needed to test it with different character sets.
And all we had to do was, in those test cases that needed
to be tested for i18n, we just added a new test set with the
data corresponding to that country.
For instance, for that character set, we added data
for Chinese, Japanese, Latin accented characters.
And the test cases were run now once per each data set.
So all we had to do was add the new data as a new data set
in our data maps, and the test didn't have to change at all.
The same test was run with different character sets.
So before finishing, I would like to mention the whole team
that's been working with us.
This is just two of us today, but the team is a whole bunch
of people working behind the framework.
So I'd like to give some special thanks to Albert Chen,
Alejandro Bologna, Ariel Rodriguez, Nicolas Frontini,
Raul Bajales, and Andrew Salamatov, which not only are
part of a framework, but also helped us a lot with the
presentation.
So if there are any questions, just shoot.

AUDIENCE: Apple, you talked a little bit about Firebug with
a UI map extension.
Can you just explain a little bit how that
relates to your XML files?
APPLE CHOW: Because, like I said, we externalize all those
UI elements in the UI map right?
And then, we find it actually is really time consuming to
manually write all those HTML tags for the different UI
elements for the locators and things like that.
And if you don't have that extension, you will just look
at xpath or some other extensions to find out the
locator of the UI elements, and then you'll be manually
writing them inside a UI map.
So that's time consuming.
So that's why we kind of piggyback on top of Firebug so
that we can just click on the button and then it just
generates the UI objects and the locators for us
automatically.
And then we just need to maybe customize on the names and
things like that, if we need to.
AUDIENCE: So you're using the UI map to generate the Excel,
the XML config file.
APPLE CHOW: Yeah, to generate a UI map, XML.

AUDIENCE: I had a couple of questions.
Is the UI map generator in Firebug a custom extension to
Firebug that you have written?
APPLE CHOW: Yeah, we added an extra tab on that?
AUDIENCE: Is that something that you can share?
SANTIAGO ETCHEBEHERE: I think we are--
APPLE CHOW: I can answer.
We're thinking about it, actually.
SANTIAGO ETCHEBEHERE: We're thinking about it.
APPLE CHOW: It's just that we have to make a couple changes
to our framework and things.
But stay tuned on the testing blog for information.
AUDIENCE: That'd be great.
A couple other questions--

what are the advantages of keeping a page-map in XML, as
opposed to encoding those attributes directly in the
client class?
SANTIAGO ETCHEBEHERE: I would say the most important
advantage is reusability and having all the configurations
in one place.
And so whenever we have to interact with the element, we
pick the configuration from there and we don't have to
write the same stuff all over.
APPLE CHOW: And also keep in mind that because we're using
XML file, so it doesn't have to--
not all the pages have to be in one XML file.
You can use includes or entities to spread them
across, maybe using a different XML for each of the
tabs on your application and things like that.
So having this is just more organized.
It's easier to change things when something changes.
It's easy to find things and change them.
And then it's all in a central place.
AUDIENCE: Do you ever have to drop down and parse the raw
HTML that's on the screen?
SANTIAGO ETCHEBEHERE: No.
For reading that--
AUDIENCE: For anything complex that you can't get Selenium to
do naturally?
SANTIAGO ETCHEBEHERE: We usually--
if we have something that Selenium cannot solve by
itself, we usually use JavaScript.
And we call the JavaScript with getEval and get
the value that way.
AUDIENCE: Do you have any problems with xpaths being
slow on IE and what do you do about it?
APPLE CHOW: Yeah, that's a common problem.
So we tried to talk to developers and convince them
into giving us IDs or names.
Yeah, and things like that.
And then even when sometimes they cannot give us IDs or
names, at least they give us something, at least ID at the
table level, so if we need to access a cell, at least the
xpath is kind of short or something like that.
AUDIENCE: Very cool framework.
Very nice.
APPLE CHOW: Thank you.
SANTIAGO ETCHEBEHERE: Thank you.
AUDIENCE: Hello.
I do have actually three questions.
Again.
The first one is, in the workflow that you showed off,
how to execute a test, somewhere at the end there
was, and then we do some asserts on the data.
I think that is usually one of the most difficult parts and
also one of the parts that tends to break when it
shouldn't break.
Is there any further support in your framework for this
assertions on the data?
APPLE CHOW: I think it's mostly in a data map.
Sometimes you can define what you verify against the data.
You can parse different properties, you define
multiple data values and the data can contain multiple
properties, so again, easily define them and have it all in
a central place again, and you know--
SANTIAGO ETCHEBEHERE: Yeah, usually the input data for the
test cases, they input data for the application and the
verification data.
APPLE CHOW: I mean, usually it's the same but then, again,
in the way that we structure the data map, you can also
overwrite something once it gets to [UNINTELLIGIBLE] or it
shows something different.
But you can, again, using the placeholder
or whatever to overwrite.
Just point to that and say overwrite this property, I'm
expecting a particular value.
And you can also do that.
AUDIENCE: The other thing that I think I didn't notice is
that some of your XML descriptions actually look
pretty much like they would contain some Java code
somewhere deep inside.
Especially the one you showed with the Ajax example and the
wait and load where you had, if this--
actually, it looked like Java objects and method course.
Isn't there kind of a danger that people start to abuse
that to actually distribute code into the XML, a little
bit of code into the client, a little bit of data into the
client and a little bit of data into the XML.
SANTIAGO ETCHEBEHERE: Actually, well this is
JavaScript But we had it this way because we wanted to--
everything that's related to, in this case, the pages and
the UI, it's in the same place.
I mean, both locators and the JavaScript code that needs to
be used to handle those elements.
It's all in the same place.
As for the data, we externalize it completely.
So we use data providers to pass the data in to test
methods and reading the XML files.
APPLE CHOW: I mean, usually we're only doing this for
loaded conditions, just to provide a
consistency when you--
even before you start interacting with objects, you
want to make sure that the page is ready.
So just to provide that consistency.
And we find doing it this way is more reliable than if you
were to have hard-coded all the conditions in your test
case and something changes again, you know it's all over
the place again.
So having it centralized in one place and provide you that
consistency, actually reduces a lot of the
flakiness of the test case.
That's what we find.
AUDIENCE: OK, one last thing before Andrew tears away the
mic from me.
Actually, it seems to me that you went to great lengths and
great difficulty to hide programming language which is
under it, and most of the things that you modeled in XML
and some of the XML facts are actually quite complicated,
declarative.
programs. Could be done easier and more straightforward
through an API in Python or Java or whatsoever, without
losing the advantages that you just mentioned you're getting
through XML, what was the reason to go down this path?
SANTIAGO ETCHEBEHERE: Basically we, again, we wanted
to keep everything that's description, or that's
descriptive, out of the code.
So the code for the test case writer looks cleaner.
It just--
like the example we showed, search for something and
verify the data or the results.
APPLE CHOW: Yeah I mean the idea is also that we have more
technical people working on the application client and the
framework, and then have those configurations in place so
that when the test case writer is using it, then he or she
doesn't have to worry about those specifics.
And also, that's why we also try to facilitate the creation
of those files, by the UI map extension and things like that
to make things easier.
So yeah, that's another angle we are looking at to improve
this framework, to just make this easier task
automatically generated.
And maybe a future plug-in will be just, once you create
a UI map, just create a corresponding data that maps
to this thing and then can just fill out the values and
things like that.
AUDIENCE: Can you give us a quick idea of how big the
scripts used to be and how big they are now?
Because this thing seems to simplify the script writing,
or certainly the end script writing.
APPLE CHOW: Yeah, like we mentioned in the--
Google Search is a simple example, but then there'll be
calling up application clients method and then just doing
some verification after that.
If you don't have that, you'll be copying and pasting all
those waitForConditions and make sure that, even before
you can feel it out, the page is loaded.
And then, once you hit the submit button, you have to
make sure, again, the results are rendering correctly before
you can do a search.
I mean, all those things like that deal with timing
situations.
AUDIENCE: 30%?
20% 80%
APPLE CHOW: More than that.
SANTIAGO ETCHEBEHERE: Yeah I think more than that.
APPLE CHOW: A lot more than that actually.
SANTIAGO ETCHEBEHERE: Because the framework we were using
previously, we had to go through all the details.
APPLE CHOW: Especially, you'd have large forms like a user
sign-up form.
That's huge.
And then you're using it on many test cases.
SANTIAGO ETCHEBEHERE: Yeah, to one method.
APPLE CHOW: And then here it's just one UI page and then you
probably have a few default data profiles.
And then, that greatly simplifies the test case
because you'll be just calling field form, and you don't even
care what it does underneath it.
So for those kind of cases, it can reduce by more than 50%.
I mean, it depends on how complex your test case is, how
many steps--
AUDIENCE: Thank you.
APPLE CHOW: --And how big the forms.
AUDIENCE: Anybody else questions?

So is this going to be available for
non-Googlers any time soon?
APPLE CHOW: Yeah, like I mentioned earlier, this is
something that we've been planning on.
But in order to do that we have to make a few changes
first. But that phase is in the plans.
So stay tuned on the testing blog.
AUDIENCE: No date though?
APPLE CHOW: We'll get to it.
We're talking about it.
We're trying to find out what kind of actions we need to
take and things like that and then plan it accordingly.
AUDIENCE: Any more questions?

One more.

I'm wondering if in the application client, when you
have an action, if you encode any assertions within that?
APPLE CHOW: It depends on what kind of action, right?
Sometimes you may, inside the client, you may want to make
sure the page is rendering correctly, or you may have
some waitForConditions that indirectly
assert things for you.
But then for test case data specific verifications, you
would do it inside a test case.
AUDIENCE: And kind of a related question, with the
xpaths that you assigned to different UI objects, do you
ever kind of encode an implicit assertion in the
xpath where the object in question is related to another
visible object or some text or something on the page?
APPLE CHOW: Yeah, then you can use a Java-- again, you can
extend a UI object, have [UNINTELLIGIBLE] in some place
if you have JavaScript--
SANTIAGO ETCHEBEHERE: We have a special type of UI object
which is a JavaScript UI object, which--
the locator is actually JavaScript code.
It's going to be [UNINTELLIGIBLE].
APPLE CHOW: Yeah, that's for something that changes
dynamically.
AUDIENCE: Thanks.

APPLE CHOW: OK.
SANTIAGO ETCHEBEHERE: OK.

Thank you.
AUDIENCE: All right, thanks guys.