| 1 | * Test Anti-Patterns |
| 2 | |
| 3 | OK, there are a few patterns that have been found over and over in the |
| 4 | testing code base which makes the tests flaky. Here is an incomplete |
| 5 | list. Don't do that. |
| 6 | |
| 7 | 1) Using pidof to wait for a background application (by name) to |
| 8 | disappear. |
| 9 | |
| 10 | Why is it flaky ? |
| 11 | |
| 12 | The application may be delayed after being forked, but not executed |
| 13 | yet. Therefore, pidof will not find it. Use "wait" instead. |
| 14 | |
| 15 | 2) Using sleep as delay-based optimistic synchronization technique. |
| 16 | |
| 17 | Why is it flaky ? |
| 18 | |
| 19 | Everything that needs to happen before/after other things need to |
| 20 | be explicitly synchronized using e.g. a file (used as a flag). |
| 21 | Sleep is just an indicator of a minimum arbitrary delay, but |
| 22 | machine load and scheduling can actually mess up the real delay |
| 23 | between applications. Use explicit synchronization points. Never |
| 24 | sleep. |
| 25 | |
| 26 | 3) Using killall on a background application. |
| 27 | |
| 28 | Why is it flaky ? |
| 29 | |
| 30 | Similarly to pidof, killall may run before the background application |
| 31 | executes, thus failing to find it. Store the application PID after it |
| 32 | it launched in background into a temporary variable for later use |
| 33 | by kill and wait. |
| 34 | |
| 35 | 4) Using wait ${!} to wait for completion of many background |
| 36 | applications. |
| 37 | |
| 38 | Why is it flaky ? |
| 39 | |
| 40 | It just waits for the last application put in background. Use |
| 41 | "wait" to wait for all background applications. |
| 42 | |
| 43 | 5) Forgetting wait at the end (or error return path) of a test phase |
| 44 | that has background applications. |
| 45 | |
| 46 | Why is it flaky ? |
| 47 | |
| 48 | Those application may interact with the following testing phases, |
| 49 | thus skewing the results. |
| 50 | |
| 51 | 6) Not grepping into the entire code base for similar patterns. |
| 52 | |
| 53 | When you find a problematic coding pattern, chances are it appears |
| 54 | elsewhere in the testing code base. Please fix it everywhere! |
| 55 | |
| 56 | 7) Introducing a utility abstraction without changing all open coded |
| 57 | similar code path. |
| 58 | |
| 59 | When an abstraction for e.g. starting and stopping the session daemon |
| 60 | is introduced as a utility (e.g. utils.sh), future changes will |
| 61 | assume that all the testing code base is using this abstraction. |
| 62 | Leaving a few custom open-coded sites of duplicated code around is a |
| 63 | good way to make it a pain to update the abstraction in the future. |