19 June 2015

UI Testing in Xcode


In my previous post I did iOS UI-automating tools overview. I start work at the post before the WWDC'15 has started so I've missed one tool introduced at the keynote - Xcode UI Testing. In this post I gonna try to bring justice.

Lets start with key features:

Good news:

+ Brand new native tool for iOS UI automation testing.
+ Support continuous integration with Xcode server
+ Works on simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Test automated recording: you just start record, perform test case on a device (or simulator) once and get a code (on Swift or Objective-C) you could run reproducing that test again and again.
+ Provide detailed test report including steps to reproduce, spent time for each step and screenshots for key moments so you get not only functional tests but aesthetic tests too.
+ Good environmental support: UI tests are integrated in Xcode with all its features including autocompletion, refactoring and others.


Bad news:

- Require iOS 9+ to run tests. Not only SDK, but latest iOS must be installed on testing device/simulator.
- Require device to be configured for development and connected to trusted host running Xcode.
- Require access to accessibility features set from privacy settings.
- Running on Mac OS only: if your QA-team have no Macs (like our do) that may be a problem.
- You could test only your main app: extensions neither watch app testing is not supported.
- Can't find any documentation for the moment.
- Can't find way to test anything outside the app. Can't switch to safari, make URL request and then go back again.

Ok. Now, if you get interested, lets see how does it work.
Before we start: I will assume that you are familiar with concepts of Xcode unit-testing by now.


Essentials

UI testing in Xcode works pretty same as unit-tests does. To begin testing you create a new UI-testing target, set testing app to it (on target's creation wizard) and Xcode configures the rest for you.

Structure of the test classes is same as unit-tests, in fact UI testing class is XCTestCase subclass.

So you have all I-know-this-guy stuff: setUp & tearDown methods are running the same way you are used to, tests methods are start with test- word, XCTAsserts validates stuff as always and you can use all you testing code base if you wish.

Test runs in a separate process along with your application.

Elements are accessed (mostly) by identifiers provided by the accessibility system (as always in all UI testing tools).

Any UI-testing process consist of 4 parts:
1 Access elements.
2 Validate if they exist.
3 Perform actions with those elements.
4 Validate results.

For doing all this UI-testing SDK provide 3 key classes:
1 XCUIApplication.
2 XCUIElement.
3 XCUIElementQuery.
+ set of our well known XCTAsserts.

XCUIApplication

XCUIApplication is a proxy for the app being tested. It is independent from the application because of running in a separate process.

It has not so many methods, just two:
func launch()

which launches your application and
func terminate()
which terminates.

Typically you call the launch() method in your test's setUp method. App launches every time in a new process. If app was running by the time launch() got called, app stops and a new process are started. It makes sure that tests won't affect each other and every single test will have a clean new app running.

In addition to those two method you can set the launching parameters and environment for the app.

XCUIApplication is the root of UI elements tree.



XCUIElement

XCUIElement is a proxy for UI elements.
Elements can be different types: buttons, cells, windows, etc.
Elements has its own identifiers - strings generated by the accessibility system: accessibility identifiers, labels, titles etc.

Elements in application creates a hierarchy - a tree with XCUIApplication as its root (yes, XCUIApplication is an XCUIElement too). Tree are built for current screen and looks like one in the picture (here and further I use pictures from UI Testing in Xcode talk):

When you access an element system automatically validates if it exists. It the element you access does not exist in UI at the time the test will fail. What does not validates by the system is UI state, you should do it yourself with XCTAsserts.

Elements you access to must be identified uniquely. If there are more then one element satisfy a query or if there isn't element at all, test will fail.

Some time it's useful to check if there’s an element and do not fail if there’s not. For example if you tap on a button, some part of UI is updating and now there is no more label that was there before. You can check element existence and do not fail by using this element's variable:
/*! Test to determine if the element exists. */ 
var exists: Bool { get }
Each element is backed by a unique query.


XCUIElementQuery

XCUIElementQuery provides access to a specific element. The result of running query is a set of current UI elements visible for the accessibility system.

There is a two main kind of query:
1. Relationship.
2. Filtering.

The relationship queries itself may be one of 3 types.

Descendants returns all elements which are placed below a given element in hierarchy.

let allButtons = app.descendantsMathcingType(.Button) 
let allCellsInTable = table.descendantsMathcingType(.Cell) 
let allMenuItemsInMenu = menu.descendantsMathcingType(.MenuItem)
Or convinetnt

let allButtons = app.buttons 
let allCellsInTable = table.cells 
let allMenuItemsInMenu = menu.meunItems


Children return all elements which are all exact children of a given element in hierarchy.

let childButtons = navBar.childrenMathcingType(.Button) 
let childCells = table.childrenMathcingType(.Cell)


Containment. This type of query helps find elements by describing their descendants. It can be difficult to uniquely identify some elements which cannot have a unique identifiers (like cells), but they can have a unique content.

let cellQuery = cells.containingType(.StaticText, identifier:"Groceries") 


The filters are query which work in conjunction with other queries. Filters use element types and identifiers. Furthermore, you can use predicates to work not only with types and identifiers but with elements' values or do partial evaluation like beginsWith:.., etc.

You can combine queries in chains bind them together similar to UNIX pipes (or monad-way if you know what I'm saying;)). For example, to get all labels in table you could do following:

let allLabelsInTable = app.tables.staticTexts 


Access Elements in Query result

But as you can remember, our goal is to get access to one specific element, not a collection. Queries provides us 3 different way to do that.
- You can access element by accessibility identifier, using subscripting
let labelElement = table.staticTexts["Groceries"] 
- You can access element by index in result set
let button = app.buttons.elementAtIndex(0) 
- If you sure that query returns only one element, you can access it that way
let navigationBar = app.navigationBars.element 


Query evaluation

To be truly successful in UI testing it's very important to understand what exactly happens under the query’s hood. Query evaluation happens not at the creation time, but only when its result needed.

Speaking about elements it's mean that query will be evaluated only when some action on the element will be called (button.tap() for example) or you'll try to read element's value.

Speaking about queries itself it's mean that query will be evaluated only if you try to get its result (for example try to get .allElementsBoundByAccessibilityElement() or get .count() of result elements). Therefore, you get no errors until you try to get a result. Event if a query itself is incorrect.

It should be noted, that query’s result will be updated as soon as any UI changes has occurred, which makes sure that you are working with the most recent element's state independently from how long ago you got a reference on it.

Therefore, if, for example, you've got a reference to a label with some text, and then UI has changed in such a way that another label with same text has appeared, your next access to the reference will make your test fail. Next access will reevaluate the query and it will fail with error of multiple elements with the same identifier.

let someElement = app.staticTexts["Label with text"]
...
// UI shows one more label with the same text: "Label with text"
...
someElement.value // UI Testing Failure: Multiple Mathces Found 


Real using. Couple words about my little experience.

The most astonishing (as and the most simple) thing was the automate test recording. You just place the cursor at a desired place, click the red button and start using the device (or the simulator) and code just appears. You even can run application and start recording with some point you at. You should try it and you'll love it. It's no doubts that all tests will be written in the same way: record some actions and (may be) adjust result code.

Most base scenarios can be easily tested just from scratch. But there are still cases which may be hard to test, for example: custom gestures, actions with delay (the only way I found from scratch is sleep() for some time), etc.

And the test reports are quiet nice. Screenshots are really help to understand what's going on.


Conclusion 

Xcode UI Testing is a very good and convenient developer tool for testing UI in iOS apps. I said developer because you have to understand not only the Xcode IDE itself but be able to write Swift (or Objective-C) code (may be in a simple level but you should) and handle not only simple button-tap situations. 
You should configure tests and work them along with understanding of the app. So being able to test the app you, probably, can enough to write some app too. Or you are an iOS-specific-hard-core-tester). 

But if you are a developer, with unit-test XCTest framework and performance testing introduced earlier you now have all possible tests in one place. We can assume that Apple doesn't stop and will support extensions and watch app UI testing as well as will add features. 

And still - it all possible for iOS9+ only, so, if you need support earlier versions, you should assume that if it works on iOS9 it should work in previous (which you can't be sure) or you should test previous versions some other way (may be using one of this tools).

3 comments:

  1. Thanks for the article.

    You probably want to correct:

    let childButtons = navBar.childrenMethcingType(.Button)
    let childCells = table.childrenMethcingType(.Cell)

    As:

    let childButtons = navBar.childrenMatchingType(.Button)
    let childCells = table.childrenMatchingType(.Cell)

    ReplyDelete
    Replies
    1. You are welcome) Thx for pointing at mistyping

      Delete