- Asynchronous integration with external systems
- In my previous case, we had an external system that outputted emails that needed to be checked for content. This could take between 1 second and 3 mins to complete and was of extremely high business value.
- Solved by having an AssertWithRetry that would check every every 500 ms and sleep until it timed out
- Problem: Delayed feedback cycle. Many test runs had multiple waits and could delay the test suite
- Lead to perception that tests could not be trusted by dev team
- "Flaky test" vs. "Legitimate bug" fights with devs.
- Frequent quarantining of tests without enough investigation. Client + Devs unwilling to accept time it took for debugging.
- Solved with re-run
- Edge Case testing through UI
- Can be solved with high level of parallelization. Not always an option
- Zero Failed test before check-in
- Redundancy in Specflow Steps
- Test IE first vs. Firefox?
Tuesday, October 30, 2012
Acceptance testing through the UI layer
This is a subject I have been thinking about for a few weeks now. On a recent project I was working on we ran an automated test suite that handled all acceptance tests through specflow + webdriver. I'm not sure I would attempt this again but find myself inheriting a system that sounds like it is based on a lot of the same principles. Here are a few challenges I faced that I hope to avoid: