Monday, January 6, 2014

Testing on Android (Part 2): Functional Tests

Here's part two of my series on Android testing.  This time, I'm going to wade into the huge number of functional testing frameworks that exist.  The differentiation I made between unit testing and functional testing is that the functional tests operate the actual device (as in, you could watch it running through a test).
So Many Frameworks

As you can see in my spreadsheet, there are a ton of functional testing frameworks out there.  I don't have any inclination to test all of them.  Here's what I haven't tried myself, for various reasons:
  • Non-Java - if my code and unit tests are going to be in Java, I might as well make my functional tests in Java as well.
  • Recorder-based testing - While this might be good for a less technical tester, I'd prefer exact control over my tests.
  • Superseded or discontinued - Some frameworks are old and have been surpassed by more modern frameworks.  Others have been discontinued.
This leaves just a few frameworks to investigate.

Robotium

I was already somewhat familiar with Robotium.  It runs off of the official Android testing framework, but adds the capabilities needed to actually run through an entire app.

It's been around for years, and so it's fairly stable.  It's also straightforward; you use the Solo class to run through your app, one step at a time.  It requires compiling along with the app's source, which could be a negative for some.  Overall, it's a solid functional testing solution.

uiautomator

UiAutomator is a more recent offering from Google.  I found it to be lacking and difficult to use.

First, it depends on more recent APIs, so it can only be used on Jelly Bean and above - you can't use it to test backwards compatibility.  Second, the deployment is more complex than other frameworks.  There's no simple script to put the code you wrote onto a device - you have to write a script yourself because it involves a few dynamic commands (depending on how you named your classes).

Third, it just feels like it's not quite there.  For example, simply launching my app from the test was a struggle.  The suggested start is to begin on your home page then page towards your app to launch it, except the code for doing so is broken.  Instead I had to use some hacky code which used "am start" to launch my Activity.

It does have a few unique advantages.  First, it can be written entirely black-box; you don't need access to the source code.  You use uiautomatorviewer to examine how an app is written and write tests based on that.  Second, it runs on the entire OS, so you can really test cross-app interaction.  If you need either of these, then I might check it out; but for my purposes this is a non-starter.

Espresso

Espresso is the latest toolkit from Google, and it's so awesome I have to wonder if the reason uiautomator is having issues is because everyone was actually working on Espresso.

It's got a different paradigm from other testing frameworks.  It starts with onView() or onData() with Hamcrest matchers to find your View/AdapterView.  Once found, you can then either perform an action or check some assertions.

The reason for the onView()/onData() setup is that Espresso examines the UI thread to know when processing is done.  A common problem with functional testing on Android is not knowing when a View will show up, so you end up with a lot of sleeps in your tests.  With Espresso you skip all of that, so it's blazing fast!

It's not all sunshine and roses.  It doesn't handle animations well (in fact, the expected setup is to turn all your animations off with dev opts).  I also found initial understanding harder, having to learn how to use Hamcrest to match my Views.  But once I got past those, the testing was so fast and stable that it blew me away.

Appium

I had started to try out Appium, as it seemed to present another way of functional testing, but ultimately gave up.

Maybe it was because I tried uiautomator, robotium, and Espresso before getting to Appium, but the setup was just overwhelming.  There was no "hello, world" sample I found that could take me from zero to testing.  There was a lot of documentation I found that listed out about five things I had to install on my machine (and have play together nicely) before I could start.

If my entire job was automation testing, or I knew selenium very well already, then I might consider Appium because then the time investment might pay off.  But as an Android developer who wants to dabble with testing on the side, I don't want to have to deal with getting this environment setup (not to mention the pain I'd be putting others through to replicate my work).

Conclusion

I would take my thoughts with a grain of salt; I haven't used any of these long enough to really get into the nitty gritty.  That said, I think my future functional tests will be written in Espresso; its design paradigm makes for focused, solid testing.  Plus, its basis on the Looper to determine when to continue makes it lightyears faster than the other frameworks could hope to be.  I hope it continues to be supported by Google because I like it.

14 comments:

  1. Hi Daniel, Thanks for those articles. Can I also mention a library of my own that can be used both for unit and functional testing. It's name is BoundBox (https://github.com/stephanenicolas/boundbox). It simply breaks encapsulation for testing, thus allowing to invoke private/protected methods, accessing the inner test of an object to assert the result of a test, accessing elements of UI to manipulate them, and even injecting mocked dependencies.
    Also, Jérôme Van Der Linden and I, have been working on a project to setup many test related technologies : Quality Tools for Android (https://github.com/stephanenicolas/Quality-Tools-for-Android).

    ReplyDelete
    Replies
    1. Looks nice, added it to my spreadsheet.

      Delete
  2. Thanks..this post if helpful

    ReplyDelete
  3. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Mobile App Development Company
    Android app Development Company
    ios app development Company
    Mobile App Development Companies

    ReplyDelete
  4. I wondered upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    Mobile App Development Company
    Mobile App Development Companies

    ReplyDelete
  5. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Android App Development Company

    ReplyDelete
  6. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    Web Design Development Company
    Web design Company in Chennai
    Web development Company in Chennai

    ReplyDelete
  7. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  8. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Fitness SMS
    Salon SMS
    Investor Relation SMS

    ReplyDelete
  9. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    iOS App Development Company

    ReplyDelete
  10. Good and Interesting article ... thanks for sharing your valuable information and time for us... keep rocks and sharing...

    Android Training in Chennai

    ReplyDelete
  11. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Texting API
    Text message marketing
    Digital Mobile Marketing
    Sms API
    Sms marketing

    ReplyDelete