[lttng-tools.git] / tests / README

* Test Anti-Patterns

OK, there are a few patterns that have been found over and over in the
testing code base which makes the tests flaky. Here is an incomplete
list. Don't do that.

1) Using pidof to wait for a background application (by name) to
   disappear.

   Why is it flaky ?

   The application may be delayed after being forked, but not executed
   yet. Therefore, pidof will not find it. Use "wait" instead.

2) Using sleep as delay-based optimistic synchronization technique.

   Why is it flaky ?

   Everything that needs to happen before/after other things need to
   be explicitly synchronized using e.g. a file (used as a flag).
   Sleep is just an indicator of a minimum arbitrary delay, but
   machine load and scheduling can actually mess up the real delay
   between applications. Use explicit synchronization points. Never
   sleep.

3) Using killall on a background application.

   Why is it flaky ?

   Similarly to pidof, killall may run before the background application
   executes, thus failing to find it. Store the application PID after it
   it launched in background into a temporary variable for later use
   by kill and wait.

4) Using wait ${!} to wait for completion of many background
   applications.

   Why is it flaky ?

   It just waits for the last application put in background. Use
   "wait" to wait for all background applications.

5) Forgetting wait at the end (or error return path) of a test phase
   that has background applications.

   Why is it flaky ?

   Those application may interact with the following testing phases,
   thus skewing the results.

6) Not grepping into the entire code base for similar patterns.

   When you find a problematic coding pattern, chances are it appears
   elsewhere in the testing code base. Please fix it everywhere!

7) Introducing a utility abstraction without changing all open coded
   similar code path.

   When an abstraction for e.g. starting and stopping the session daemon
   is introduced as a utility (e.g. utils.sh), future changes will
   assume that all the testing code base is using this abstraction.
   Leaving a few custom open-coded sites of duplicated code around is a
   good way to make it a pain to update the abstraction in the future.
Commit	Line	Data
94dab75b MD	1	* Test Anti-Patterns
	2
	3	OK, there are a few patterns that have been found over and over in the
	4	testing code base which makes the tests flaky. Here is an incomplete
	5	list. Don't do that.
	6
	7	1) Using pidof to wait for a background application (by name) to
	8	disappear.
	9
	10	Why is it flaky ?
	11
	12	The application may be delayed after being forked, but not executed
	13	yet. Therefore, pidof will not find it. Use "wait" instead.
	14
	15	2) Using sleep as delay-based optimistic synchronization technique.
	16
	17	Why is it flaky ?
	18
	19	Everything that needs to happen before/after other things need to
	20	be explicitly synchronized using e.g. a file (used as a flag).
	21	Sleep is just an indicator of a minimum arbitrary delay, but
	22	machine load and scheduling can actually mess up the real delay
	23	between applications. Use explicit synchronization points. Never
	24	sleep.
	25
	26	3) Using killall on a background application.
	27
	28	Why is it flaky ?
	29
	30	Similarly to pidof, killall may run before the background application
	31	executes, thus failing to find it. Store the application PID after it
	32	it launched in background into a temporary variable for later use
	33	by kill and wait.
	34
	35	4) Using wait ${!} to wait for completion of many background
	36	applications.
	37
	38	Why is it flaky ?
	39
	40	It just waits for the last application put in background. Use
	41	"wait" to wait for all background applications.
	42
	43	5) Forgetting wait at the end (or error return path) of a test phase
	44	that has background applications.
	45
	46	Why is it flaky ?
	47
	48	Those application may interact with the following testing phases,
	49	thus skewing the results.
	50
	51	6) Not grepping into the entire code base for similar patterns.
	52
	53	When you find a problematic coding pattern, chances are it appears
	54	elsewhere in the testing code base. Please fix it everywhere!
	55
	56	7) Introducing a utility abstraction without changing all open coded
	57	similar code path.
	58
	59	When an abstraction for e.g. starting and stopping the session daemon
	60	is introduced as a utility (e.g. utils.sh), future changes will
	61	assume that all the testing code base is using this abstraction.
	62	Leaving a few custom open-coded sites of duplicated code around is a
	63	good way to make it a pain to update the abstraction in the future.