11 June 2015

iOS UI-automating tools overview.





There are no doubts, automated test are actually useful during development. 
It makes us sure that every thing is still working after each new line of code we write. It saves us tons of time.

Starting with nothing I was interested in automation of UI testing.

There are a lot of different post, videos and even books about unit-testing. So many explains different tools, techniques, different approaches. So many examples. We have native XCTest, we have Specta, we have Quick (the newest testing framework for Swift and plain old ObjC you should check) and many more.

But it's all about testing business logic. When it comes to testing UI testing, functional testing, testing layout...Huh...Hard to find something.

Some doing hard with unit tests but when it comes to UI - ignores testing at all. Some trying to test UI diving deep into view controllers internal logic, getting controls by making public (just for tests but still...) their private code. Better but still doesn't work for me.

Almost 10 years of iOS, more then 20 years of Mac OS and there isn't better solution for testing then just inject some additional code in each of your view controllers to be able to press a button and check if it's actually pressed?

Possibilities are looks not so wide...At first.

The point is that UI is the business logiс from the user's perspective. And who are doing tests from the user's perspective? QA!
QA-guys are responsible for testing the final UI and behavior. So automating UI testing should help them in first place.
I was just looking for a thing that was laying in a different place! When I've realized that, things just go better!

I found grate post from Harry Hornreich which provides good overview of test automation approaches. Really good post. If you are interested in the topiс, you should read it definitely.

I'll try to list the most popular tools for UI-automation and functional testing and point to its key features.
If you know more tools or more features and applications for listed tools, I would be appreciated if you point me on it.


1. UIAutomation



Native Apple tool for automating UI. It's legitimate to assume that it won't brake with new iOS releases.
Back to 2010 there was a WWDC video called "Automating User Interface Testing with Instruments" describing testing UI using UIAutomation and now days there is a good post making introduction in UIAutomation. Also there is a UIAutomation lesson.

Good news:

+ Native tool for iOS UI automation testing.
+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Test automated recording: you just start record, perform test case on a device once and get a script you could run reproducing that test again and again.

Bad news:

- Test scripts are written in java script: you should know one more language. Not so hard, but new.
- Bad environmental support: it's hard to write tests, has pure autocompletion, no code-documentation, no any SDK - you just write code in notepad-like editor. Some times its even hard to save it.
- Running on Mac OS only: if your QA-team have no Macs (like our do) that may be a problem.
- In fact, I failed to run test recording. 
Seems as hard as I try is not so hard as it gets. ¯\_(ツ)_/¯ 
- Haven't heard any news about UIAutomation since it was mentioned at WWDC in 2010...



Brings Selenium tests to iOS devices. Makes iOS UI automation as simple as web browser's is, as they say... Uses UIAutomation under the hood.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Provide support for Selenium / WebDriver API on iOS device, which allows you to use existing web-testing infrastructure (if any).
+ Scales as hell by Selenium means.
+ Used by Facebook, Google and others where not the last fools works on. Looks like it used in companies with big web-infrastraction.
Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Selenium-tests for iOS, which means that if you don't know it you have to learn it. Test are written on Ruby, Java, Phyton, C#...No Swift or ObjC or even JavaScript from UIAutomation.



Same as iOS-driver Appium is build on Selenium Web Driver API. But it's philosophy a bit wider.
- You should be able to cover by single test all platforms you support (iOS, Android).
- Test should run in native tools, so under the hood it's UIAutomation running iOS device.
- You should not be bounded by a single language. You could use any language you want to write tests.

Sounds good, huh?

The way it works is simple: you have server which drive devices via platforms' native UI automation tools.  You have clients which sends to server tests code written in any language you want. Clients are translating you tests code to server's API.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Cross-platform: single test code for iOS and Android devices.
+ Clients are written in many popular languages (Objective-C, Ruby, Python, Java, JavaScript, PHP, C#, RobotFramework and others).
+ GUI wrappers around Appium server for OS X and Windows.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ Promises very easy installation.
+ Actively supported.
+ Good documentation.

Bad news:

- If you're already using own UIAutomation tests you able to use JavaScript only.




Tool for running iOS automation with Cucumber tests.

Good news:

+ Promises very easy installation, which takes less then 10 minutes.
+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ The only tool I heard which records a video of the tests.
+ Grate community of testers using it.
+ Tests are human-readable. You build some statements with code and then make human-readable tests by combining those statements. Looks good.
+ Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Cucumber-tests for iOS, which means you have to know Ruby.
- The last tool update was 2 years ago.



Another cross-platform open-source and free framework for UIAutomation using cucumber. Developed and maintained by Xamarin. Here you can find a tutorial for iOS Automated Testing With Calabash, Cucumber, and Ruby.



Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Separated from unit tests: you not have to wait for it finish while running unit tests.
+ Not bounded by testing application only: using it you could go from the application to settings app or safari and then get back again.
+ Makes screenshots.
+ Cross-platform: single test code for iOS and Android devices.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ Tests are human-readable. You build some statements with code and then make human-readable tests by combining those statements. Looks good.
+ Supports cloud testing: you could run you app on devices which you can't access physically.
+ Actively supported.
+ While Calabash is completely free, Xamarin provides a number of commercial services centered around Calabash and quality assurance for mobile. You could use it's devices cloud to test your app on a lot of different devices.

Bad news:

- Cucumber-tests for iOS, which means you have to know Ruby.
- Access to devices cloud is not so cheap.



Testing UI from developers perspective can be as much useful as testing business logic. It's always good to know if your changes weren't brake any UI.
It's always good to track thouse bugs you did as soon as it possible not waiting for testers to find them in next iteration or even later (or event users in production).
It's good to have all those benefits come with unit-testing not just for business logic part but for entier application.
So, good news! There are tools not only for QA, but for us, for developers!



6. KIF 


Stands for Keep It Functional. Open-source UI automation framework for Objective-C (compatible with Swift). KIF is the most popular way for functional test automation among iOS developers.
There is a good tutorial for iOS UI Testing with KIF.

Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ KIF made by iOS developers for iOS developers: Objective-C & Swift code.
+ Integrated into XCTests. Developers aren't need to learn any new things - just start write tests.
+ Improving and using by many-many developers every day. Still alive and feels good.
+ Open-source - you can easily learn how it works and make own changes if need so.
+ You can use it not only for testing, but for debug! You don't need repeat steps to reproduce live it for KIF and, when it's done, just stop it and look into the stack, variables - all debug is yours!
+ You have direct access to the code so you could easily put the application into any state right from "inside" of it and not just outside - from the user's perspective.

Bad news:

- Uses Apple's not documented API - Apple could change it at any time.
- Tests are bounded by the application only: it's not possible to do anything outside the application.
- Integrated into unit-tests. You have to turns it off somehow when you don't need it or your tests will take a lot of time.



UI testing consist of two main part: functional testing (which we have told a lot) and esthetic (if you could say so): making sure if UI looks as it should. For second part FBSnapshotTestCase it's a very useful tool. As it comes from the name, FBSnapshotTestCase is snapshot testing tool. Good explanation for it gave Orta Therox in his article in this objc.io issue.

Good news:

+ Support continuous integration, Jenkins
+ Works for simulator and actual device.
+ There is no need for tooling application itself. You're testing the same build you'll ship.
+ Made by iOS developers for iOS developers: Objective-C & Swift code.
+ Integrated into unit tests. Developers aren't need to learn any new things - just start write tests.
+ Improving and using by many developers every day. Still alive and feels good.
+ Open-source - you can easily learn how it works and make own changes if need so.

Bad news:

- Requires first time configuration: first you make real UI, then you make base snapshot to compare with, then you start testing.


Conclusion


In fact there are a lot of different UI automation tools. All uses different script languages (Ruby, JavaSctript, Phyton, e.t.s). All runs separately from the application and not require tooling application. All of them looks similar with just a little differences. Main differences are script languages, scalability, is it open-source or not, is it popular or not. Choosing tool you must consider in each specific case your own preferences, background, infrastructure and needs, popularity is also not the last point.

As for me, if you are an iOS developer choose is obvious: KIF for functional and FBSnapshotTestCase for visual regression test.

And there is one more thing I need to mention!
One more thing...

Thing was introduced on the latest WWDC'15 and that is XCode UI Testing. It's still to be learned. But it's very-very possible that we, iOS developers, finally found our tool. Not right now, but in near feature with all bugs are fixed)

Stay with me and I'll learn it too.



Where to go next:

1 comment:

  1. Consider ZAPTEST (www.zaptest.com). It uses innovative visual approach to object recognition. It can automate testing any GUI-based application on iOS, Android, Mac, Linux, Windows regardless of technology used to build the app. You can build test using JavaScript or VBScript. It's commercial tool, but has free version.

    ReplyDelete