19 June 2015

UI Testing in Xcode


In my previous post I did iOS UI-automating tools overview. I start work at the post before the WWDC'15 has started so I've missed one tool introduced at the keynote - Xcode UI Testing. In this post I gonna try to bring justice.

Lets start with key features:

Good news:

+ Brand new native tool for iOS UI automation testing.
+ Support continuous integration with Xcode server
+ Works on simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Test automated recording: you just start record, perform test case on a device (or simulator) once and get a code (on Swift or Objective-C) you could run reproducing that test again and again.
+ Provide detailed test report including steps to reproduce, spent time for each step and screenshots for key moments so you get not only functional tests but aesthetic tests too.
+ Good environmental support: UI tests are integrated in Xcode with all its features including autocompletion, refactoring and others.


Bad news:

- Require iOS 9+ to run tests. Not only SDK, but latest iOS must be installed on testing device/simulator.
- Require device to be configured for development and connected to trusted host running Xcode.
- Require access to accessibility features set from privacy settings.
- Running on Mac OS only: if your QA-team have no Macs (like our do) that may be a problem.
- You could test only your main app: extensions neither watch app testing is not supported.
- Can't find any documentation for the moment.
- Can't find way to test anything outside the app. Can't switch to safari, make URL request and then go back again.

Ok. Now, if you get interested, lets see how does it work.
Before we start: I will assume that you are familiar with concepts of Xcode unit-testing by now.


Essentials

UI testing in Xcode works pretty same as unit-tests does. To begin testing you create a new UI-testing target, set testing app to it (on target's creation wizard) and Xcode configures the rest for you.

Structure of the test classes is same as unit-tests, in fact UI testing class is XCTestCase subclass.

So you have all I-know-this-guy stuff: setUp & tearDown methods are running the same way you are used to, tests methods are start with test- word, XCTAsserts validates stuff as always and you can use all you testing code base if you wish.

Test runs in a separate process along with your application.

Elements are accessed (mostly) by identifiers provided by the accessibility system (as always in all UI testing tools).

Any UI-testing process consist of 4 parts:
1 Access elements.
2 Validate if they exist.
3 Perform actions with those elements.
4 Validate results.

For doing all this UI-testing SDK provide 3 key classes:
1 XCUIApplication.
2 XCUIElement.
3 XCUIElementQuery.
+ set of our well known XCTAsserts.

XCUIApplication

XCUIApplication is a proxy for the app being tested. It is independent from the application because of running in a separate process.

It has not so many methods, just two:
func launch()

which launches your application and
func terminate()
which terminates.

Typically you call the launch() method in your test's setUp method. App launches every time in a new process. If app was running by the time launch() got called, app stops and a new process are started. It makes sure that tests won't affect each other and every single test will have a clean new app running.

In addition to those two method you can set the launching parameters and environment for the app.

XCUIApplication is the root of UI elements tree.



XCUIElement

XCUIElement is a proxy for UI elements.
Elements can be different types: buttons, cells, windows, etc.
Elements has its own identifiers - strings generated by the accessibility system: accessibility identifiers, labels, titles etc.

Elements in application creates a hierarchy - a tree with XCUIApplication as its root (yes, XCUIApplication is an XCUIElement too). Tree are built for current screen and looks like one in the picture (here and further I use pictures from UI Testing in Xcode talk):

When you access an element system automatically validates if it exists. It the element you access does not exist in UI at the time the test will fail. What does not validates by the system is UI state, you should do it yourself with XCTAsserts.

Elements you access to must be identified uniquely. If there are more then one element satisfy a query or if there isn't element at all, test will fail.

Some time it's useful to check if there’s an element and do not fail if there’s not. For example if you tap on a button, some part of UI is updating and now there is no more label that was there before. You can check element existence and do not fail by using this element's variable:
/*! Test to determine if the element exists. */ 
var exists: Bool { get }
Each element is backed by a unique query.


XCUIElementQuery

XCUIElementQuery provides access to a specific element. The result of running query is a set of current UI elements visible for the accessibility system.

There is a two main kind of query:
1. Relationship.
2. Filtering.

The relationship queries itself may be one of 3 types.

Descendants returns all elements which are placed below a given element in hierarchy.

let allButtons = app.descendantsMathcingType(.Button) 
let allCellsInTable = table.descendantsMathcingType(.Cell) 
let allMenuItemsInMenu = menu.descendantsMathcingType(.MenuItem)
Or convinetnt

let allButtons = app.buttons 
let allCellsInTable = table.cells 
let allMenuItemsInMenu = menu.meunItems


Children return all elements which are all exact children of a given element in hierarchy.

let childButtons = navBar.childrenMathcingType(.Button) 
let childCells = table.childrenMathcingType(.Cell)


Containment. This type of query helps find elements by describing their descendants. It can be difficult to uniquely identify some elements which cannot have a unique identifiers (like cells), but they can have a unique content.

let cellQuery = cells.containingType(.StaticText, identifier:"Groceries") 


The filters are query which work in conjunction with other queries. Filters use element types and identifiers. Furthermore, you can use predicates to work not only with types and identifiers but with elements' values or do partial evaluation like beginsWith:.., etc.

You can combine queries in chains bind them together similar to UNIX pipes (or monad-way if you know what I'm saying;)). For example, to get all labels in table you could do following:

let allLabelsInTable = app.tables.staticTexts 


Access Elements in Query result

But as you can remember, our goal is to get access to one specific element, not a collection. Queries provides us 3 different way to do that.
- You can access element by accessibility identifier, using subscripting
let labelElement = table.staticTexts["Groceries"] 
- You can access element by index in result set
let button = app.buttons.elementAtIndex(0) 
- If you sure that query returns only one element, you can access it that way
let navigationBar = app.navigationBars.element 


Query evaluation

To be truly successful in UI testing it's very important to understand what exactly happens under the query’s hood. Query evaluation happens not at the creation time, but only when its result needed.

Speaking about elements it's mean that query will be evaluated only when some action on the element will be called (button.tap() for example) or you'll try to read element's value.

Speaking about queries itself it's mean that query will be evaluated only if you try to get its result (for example try to get .allElementsBoundByAccessibilityElement() or get .count() of result elements). Therefore, you get no errors until you try to get a result. Event if a query itself is incorrect.

It should be noted, that query’s result will be updated as soon as any UI changes has occurred, which makes sure that you are working with the most recent element's state independently from how long ago you got a reference on it.

Therefore, if, for example, you've got a reference to a label with some text, and then UI has changed in such a way that another label with same text has appeared, your next access to the reference will make your test fail. Next access will reevaluate the query and it will fail with error of multiple elements with the same identifier.

let someElement = app.staticTexts["Label with text"]
...
// UI shows one more label with the same text: "Label with text"
...
someElement.value // UI Testing Failure: Multiple Mathces Found 


Real using. Couple words about my little experience.

The most astonishing (as and the most simple) thing was the automate test recording. You just place the cursor at a desired place, click the red button and start using the device (or the simulator) and code just appears. You even can run application and start recording with some point you at. You should try it and you'll love it. It's no doubts that all tests will be written in the same way: record some actions and (may be) adjust result code.

Most base scenarios can be easily tested just from scratch. But there are still cases which may be hard to test, for example: custom gestures, actions with delay (the only way I found from scratch is sleep() for some time), etc.

And the test reports are quiet nice. Screenshots are really help to understand what's going on.


Conclusion 

Xcode UI Testing is a very good and convenient developer tool for testing UI in iOS apps. I said developer because you have to understand not only the Xcode IDE itself but be able to write Swift (or Objective-C) code (may be in a simple level but you should) and handle not only simple button-tap situations. 
You should configure tests and work them along with understanding of the app. So being able to test the app you, probably, can enough to write some app too. Or you are an iOS-specific-hard-core-tester). 

But if you are a developer, with unit-test XCTest framework and performance testing introduced earlier you now have all possible tests in one place. We can assume that Apple doesn't stop and will support extensions and watch app UI testing as well as will add features. 

And still - it all possible for iOS9+ only, so, if you need support earlier versions, you should assume that if it works on iOS9 it should work in previous (which you can't be sure) or you should test previous versions some other way (may be using one of this tools).

11 June 2015

iOS UI-automating tools overview.





There are no doubts, automated test are actually useful during development. 
It makes us sure that every thing is still working after each new line of code we write. It saves us tons of time.

Starting with nothing I was interested in automation of UI testing.

There are a lot of different post, videos and even books about unit-testing. So many explains different tools, techniques, different approaches. So many examples. We have native XCTest, we have Specta, we have Quick (the newest testing framework for Swift and plain old ObjC you should check) and many more.

But it's all about testing business logic. When it comes to testing UI testing, functional testing, testing layout...Huh...Hard to find something.

Some doing hard with unit tests but when it comes to UI - ignores testing at all. Some trying to test UI diving deep into view controllers internal logic, getting controls by making public (just for tests but still...) their private code. Better but still doesn't work for me.

Almost 10 years of iOS, more then 20 years of Mac OS and there isn't better solution for testing then just inject some additional code in each of your view controllers to be able to press a button and check if it's actually pressed?

Possibilities are looks not so wide...At first.

The point is that UI is the business logiс from the user's perspective. And who are doing tests from the user's perspective? QA!
QA-guys are responsible for testing the final UI and behavior. So automating UI testing should help them in first place.
I was just looking for a thing that was laying in a different place! When I've realized that, things just go better!

I found grate post from Harry Hornreich which provides good overview of test automation approaches. Really good post. If you are interested in the topiс, you should read it definitely.

I'll try to list the most popular tools for UI-automation and functional testing and point to its key features.
If you know more tools or more features and applications for listed tools, I would be appreciated if you point me on it.


1. UIAutomation



Native Apple tool for automating UI. It's legitimate to assume that it won't brake with new iOS releases.
Back to 2010 there was a WWDC video called "Automating User Interface Testing with Instruments" describing testing UI using UIAutomation and now days there is a good post making introduction in UIAutomation. Also there is a UIAutomation lesson.

Good news:

+ Native tool for iOS UI automation testing.
+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Test automated recording: you just start record, perform test case on a device once and get a script you could run reproducing that test again and again.

Bad news:

- Test scripts are written in java script: you should know one more language. Not so hard, but new.
- Bad environmental support: it's hard to write tests, has pure autocompletion, no code-documentation, no any SDK - you just write code in notepad-like editor. Some times its even hard to save it.
- Running on Mac OS only: if your QA-team have no Macs (like our do) that may be a problem.
- In fact, I failed to run test recording. 
Seems as hard as I try is not so hard as it gets. ¯\_(ツ)_/¯ 
- Haven't heard any news about UIAutomation since it was mentioned at WWDC in 2010...



Brings Selenium tests to iOS devices. Makes iOS UI automation as simple as web browser's is, as they say... Uses UIAutomation under the hood.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Provide support for Selenium / WebDriver API on iOS device, which allows you to use existing web-testing infrastructure (if any).
+ Scales as hell by Selenium means.
+ Used by Facebook, Google and others where not the last fools works on. Looks like it used in companies with big web-infrastraction.
Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Selenium-tests for iOS, which means that if you don't know it you have to learn it. Test are written on Ruby, Java, Phyton, C#...No Swift or ObjC or even JavaScript from UIAutomation.



Same as iOS-driver Appium is build on Selenium Web Driver API. But it's philosophy a bit wider.
- You should be able to cover by single test all platforms you support (iOS, Android).
- Test should run in native tools, so under the hood it's UIAutomation running iOS device.
- You should not be bounded by a single language. You could use any language you want to write tests.

Sounds good, huh?

The way it works is simple: you have server which drive devices via platforms' native UI automation tools.  You have clients which sends to server tests code written in any language you want. Clients are translating you tests code to server's API.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Cross-platform: single test code for iOS and Android devices.
+ Clients are written in many popular languages (Objective-C, Ruby, Python, Java, JavaScript, PHP, C#, RobotFramework and others).
+ GUI wrappers around Appium server for OS X and Windows.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ Promises very easy installation.
+ Actively supported.
+ Good documentation.

Bad news:

- If you're already using own UIAutomation tests you able to use JavaScript only.




Tool for running iOS automation with Cucumber tests.

Good news:

+ Promises very easy installation, which takes less then 10 minutes.
+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ The only tool I heard which records a video of the tests.
+ Grate community of testers using it.
+ Tests are human-readable. You build some statements with code and then make human-readable tests by combining those statements. Looks good.
+ Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Cucumber-tests for iOS, which means you have to know Ruby.
- The last tool update was 2 years ago.



Another cross-platform open-source and free framework for UIAutomation using cucumber. Developed and maintained by Xamarin. Here you can find a tutorial for iOS Automated Testing With Calabash, Cucumber, and Ruby.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Makes screenshots.
+ Cross-platform: single test code for iOS and Android devices.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ Tests are human-readable. You build some statements with code and then make human-readable tests by combining those statements. Looks good.
+ Supports cloud testing: you could run you app on devices which you can't access physically.
+ Actively supported.
+ While Calabash is completely free, Xamarin provides a number of commercial services centered around Calabash and quality assurance for mobile. You could use it's devices cloud to test your app on a lot of different devices.

Bad news:

- Cucumber-tests for iOS, which means you have to know Ruby.
- Access to devices cloud is not so cheap.



Testing UI from developers perspective can be as much useful as testing business logic. It's always good to know if your changes weren't brake any UI.
It's always good to track thouse bugs you did as soon as it possible not waiting for testers to find them in next iteration or even later (or event users in production).
It's good to have all those benefits come with unit-testing not just for business logic part but for entier application.
So, good news! There are tools not only for QA, but for us, for developers!



6. KIF 


Stands for Keep It Functional. Open-source UI automation framework for Objective-C (compatible with Swift). KIF is the most popular way for functional test automation among iOS developers.
There is a good tutorial for iOS UI Testing with KIF.

Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ KIF made by iOS developers for iOS developers: Objective-C & Swift code.
+ Integrated into XCTests. Developers aren't need to learn any new things - just start write tests.
+ Improving and using by many-many developers every day. Still alive and feels good.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ You can use it not only for testing, but for debug! You don't need repeat steps to reproduce live it for KIF and, when it's done, just stop it and look into the stack, variables - all debug is yours!
+ You have direct access to the code so you could easily put the application into any state right from "inside" of it and not just outside - from the user's perspective.

Bad news:

- Uses Apple's not documented API - Apple could change it at any time.
- Tests are bounded by the application only: it's not possible to do anything outside the application.
- Integrated into unit-tests. You have to turns it off somehow when you don't need it or your tests will take a lot of time.



UI testing consist of two main part: functional testing (which we have told a lot) and esthetic (if you could say so): making sure if UI looks as it should. For second part FBSnapshotTestCase it's a very useful tool. As it comes from the name, FBSnapshotTestCase is snapshot testing tool. Good explanation for it gave Orta Therox in his article in this objc.io issue.

Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Made by iOS developers for iOS developers: Objective-C & Swift code.
+ Integrated into unit tests. Developers aren't need to learn any new things - just start write tests.
+ Improving and using by many developers every day. Still alive and feels good.
+ Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Requires first time configuration: first you make real UI, then you make base snapshot to compare with, then you start testing.


Conclusion


In fact there are a lot of different UI automation tools. All uses different script languages (Ruby, JavaSctript, Phyton, e.t.s). All runs separately from the application and not require tooling application. All of them looks similar with just a little differences. Main differences are script languages, scalability, is it open-source or not, is it popular or not. Choosing tool you must consider in each specific case your own preferences, background, infrastructure and needs, popularity is also not the last point.

As for me, if you are an iOS developer choose is obvious: KIF for functional and FBSnapshotTestCase for visual regression test.

And there is one more thing I need to mention!
One more thing...

Thing was introduced on the latest WWDC'15 and that is XCode UI Testing. It's still to be learned. But it's very-very possible that we, iOS developers, finally found our tool. Not right now, but in near feature with all bugs are fixed)

Stay with me and I'll learn it too.



Where to go next:

02 June 2015

#JustFiguredOut: Local Notifications

It is often a problem to notify user about something. But could we be sure if the user sees those notifications? What if users switched time on the device? What if the device is out of battery just in the fire date? Will user be notified?

I was terribly interested in local notifications behaviour in this situations.

Experiment consisted of two parts:

1. Switching time.


  • Schedule bunch of local notifications.
  • Switch time on the device to some time after scheduled notifications' fire date.
Result: If time was switched less then 15- minutes after the fire date, all scheduled notifications are fired immediately. If switch time to 15 minutes after the fire date and more - notifications are never shown (even if you switch time back again and wait until the fire dafe to come).

Inability to fire notification after returning back to normal time comes from system: it drops off notifications' next fire date after the fire date has come and not restores it again.

2. Turning off the device.


  • Schedule bunch of local notifications.
  • Turn off the device.
  • Wait for fire date to pass.
  • Turn on the device.
Result: all scheduled notifications were appeared fired on devices block screen but (as with switching time) only if the device was turned on not more then 15 minutes after the fire date.

So, in iOS 8.3 and earlier (may be in future too) local notifications scheduled in the system are fired all at once as soon as the fire date is passed.

It works if you switch time on the device manually (not so often case) or  the next time the device is turned on if it was off by the fire date (possible case), but just in 15-minutes interval after the fire date.

But! There is one big difference: if a local notification fires as it should, system automatically removes it from scheduledNotifications list. If it's NOT fired as it should - the notification remains in the list with next file date = null.

Consider the results in your development and use local notifications wisely.