Monday, November 26, 2012

Automation feedback - when to think about parallelization

I've designed a few automated testing systems now. A frequent anti-patten I've seen is using UI automation tools such as selenium to drive acceptance tests. One of the big pains is that as a project grows this system becomes more and more cumbersome, requires a lot of maitenance (to the point the cost of keeping them is questionable) and slows down feedback. The main goals to aim for are:

  1.  Feasible to implement in terms of development skills
  2.  Low cost in terms of time and maitenenace effort
  3.  Effective and timely feedback

There are a couple of options I've come across

  • Run a nightly build of UI tests (feedback once every 24 hours.)
    • I tried this on a project for Google. It can work with small teams with limited checkins (~10 per day) where functionality is well defined and it is easy to blame revision numbers
  • Parallelization - Ideally, each test runs concurrently on its own environment. 
    • Problems I've faced:
      • Can be dependant on Hardware if VMs aren't an option
      • Usually BS trying to get corporate IT departments to give you what you need
      • May miss bugs that would occur naturally when 2 users are concurrently using system that would be caught by running them on the same environment
  • Continuous run
    • The test suite just constantly builds and run tests
      • Requires an extremely robust test suite or very basic site
      • Can make debugging difficult because tracking the run against the build is very rapid
  • Test prioritization
    • Label tests as core and have them run very frequently (every checkin)
    • Run tests labelled regression in a seperate pipeline
      • This is usually very feasible
What do I recommend? Don't run acceptance tests at all through the UI. Use a proper domain model so it runs as a headless check and have general smoke tests. However, js heavy websites where business logic is happening a in the user interface means this isnt always feasible. I'd probably choose parallelization since it strikes well in terms of feasibility, cost and feedback.

Thursday, November 8, 2012

Agile Team Health Tips

A lot of teams get sick working in enclosed areas. Here are a few tips I've started to promote to keep a team up

  • If the team is testing against mobile, make sure the phones are wiped down every day
  • If someone feels sick, quarantine them immediately. Work from home for a day and see if its an infection.
  • pair with seperate keyboards / mice
  • Avoid eating while you work
  • Sustainable pace, don't be overworked and have your immune system down