1 // Show ToC at a specific location for a GitHub rendering
10 // This is to mimic what GitHub does so that anchors work in an offline
22 image:https://img.shields.io/pypi/v/normand.svg?label=Latest%20version[link="https://pypi.python.org/pypi/normand"]
25 _**Normand**_ is a text-to-binary processor with its own language.
27 This package offers both a portable {py3} module and a command-line
30 WARNING: This version of Normand is 0.1, meaning both the Normand
31 language and the module/CLI interface aren't stable.
34 // ToC location for a GitHub rendering
40 The purpose of Normand is to consume human-readable text representing
41 bytes and to produce the corresponding binary data.
45 Consider the following Normand input:
48 4f 55 32 bb $167 fe %10100111 a9 $-32
51 The generated nine bytes are:
54 4f 55 32 bb a7 fe a7 a9 e0
58 As you can see in the last example, the fundamental unit of the Normand
59 language is the _byte_. The order in which you list bytes will be the
60 order of the generated data.
62 The Normand language is more than simple lists of bytes, though. Its
65 Comments, including a bunch of insignificant symbols which may improve readability::
70 ff bb %1101:0010 # This is a comment
71 78 29 af $192 # This too # 99 $-80
72 fe80::6257:18ff:fea3:4229
74 10839636-5d65-4a68-8e6a-21608ddf7258
80 ff bb d2 78 29 af c0 99 b0 fe 80 62 57 18 ff fe
81 a3 42 29 60 57 18 a3 42 29 10 83 96 36 5d 65 4a
82 68 8e 6a 21 60 8d df 72 58
85 Hexadecimal, decimal, and binary byte constants::
90 aa bb $247 $-89 %0011_0010 %11.01= 10/10
99 UTF-8, UTF-16, and UTF-32 literal strings::
105 u16le"stress\nverdict 🤣"
111 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 00 73 00 74 ┆ hello world!•s•t
112 00 72 00 65 00 73 00 73 00 0a 00 76 00 65 00 72 ┆ •r•e•s•s•••v•e•r
113 00 64 00 69 00 63 00 74 00 20 00 3e d8 23 dd ┆ •d•i•c•t• •>•#•
116 Labels: special variables holding the offset where they're defined::
119 <beg> b2 52 e3 bc 91 05
120 $100 $50 <chair> 33 9f fe
127 5e 65 {tower = 47} c6 7f f2 c4
128 44 {hurl = tower - 14} b5 {tower = hurl} 26 2d
131 The value of a variable assignment is the evaluation of a valid {py3}
132 expression which may include label and variable names.
134 Value encoding with a specific length (8{nbsp}bits to 64{nbsp}bits) and byte order::
140 {be} 67 <lbl> 44 $178 {(end - lbl) * 8 + strength : 16} $99 <end>
147 67 44 b2 00 2c 63 37 f8 ff ff
150 The encoded value is the evaluation of a valid {py3} expression which
151 may include label and variable names.
158 aa bb * 5 cc "yeah\0" * 8
164 aa bb bb bb bb bb cc 79 65 61 68 00 79 65 61 68 ┆ •••••••yeah.yeah
165 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
166 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ yeah•yeah•yeah•
170 Multilevel grouping::
175 ff ((aa bb "zoom" cc) * 5) * 3 $-34 * 4
181 ff aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa ┆ •••zoom•••zoom••
182 bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a ┆ •zoom•••zoom•••z
183 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f ┆ oom•••zoom•••zoo
184 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc ┆ m•••zoom•••zoom•
185 aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb ┆ ••zoom•••zoom•••
186 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f ┆ zoom•••zoom•••zo
187 6f 6d cc aa bb 7a 6f 6f 6d cc de de de de ┆ om•••zoom•••••
190 Precise error reporting::
193 /tmp/meow.normand:10:24 - Expecting a bit (`0` or `1`).
197 /tmp/meow.normand:32:6 - Unexpected character `k`.
201 /tmp/meow.normand:24:19 - Unknown variable/label name `meow` in expression `(meow - 45) // 8`.
205 /tmp/meow.normand:18:9 - Value 315 is outside the 8-bit range when evaluating expression `end - ICITTE` at byte offset 45.
208 You can use Normand to track data source files in your favorite VCS
209 instead of raw binary files. The binary files that Normand generates can
210 be used to test file format decoding, including malformatted data, for
211 example, as well as for education.
213 See <<learn-normand>> to explore all the Normand features.
217 Normand requires Python ≥ 3.4.
222 $ python3 -m pip install --user normand
226 https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-to-the-user-site[Installing to the User Site]
227 to learn more about a user site installation.
231 Normand has a single module file, `normand.py`, which you can copy as is
232 to your project to use it (both the <<python-3-api,`normand.parse()`>>
233 function and the <<command-line-tool,command-line tool>>).
235 `normand.py` has _no external dependencies_, but if you're using
236 Python{nbsp}3.4, you'll need a local copy of the standard `typing`
242 A Normand text input is a sequence of items which represent a sequence
245 [[state]] During the processing of items to data, Normand relies on a
250 |State variable |Description |Initial value: <<python-3-api,{py3} API>> |Initial value: <<command-line-tool,CLI>>
252 |[[cur-offset]] Current offset
254 The current offset has an effect on the value of
255 <<label,labels>> and of the special `ICITTE` name in <<value,value>> and
256 <<variable-assignment,variable assignment>> expression evaluation.
258 Each generated byte increments the current offset.
260 A <<current-offset-setting,current offset setting>> may change the
262 |`init_offset` parameter of the `parse()` function.
265 |[[cur-bo]] Current byte order
267 The current byte order has an effect on the encoding of <<value,values>>.
269 A <<current-byte-order-setting,current byte order setting>> may change
270 the current byte order.
271 |`init_byte_order` parameter of the `parse()` function.
272 |`--byte-order` option.
275 |Mapping of label names to integral values.
276 |`init_labels` parameter of the `parse()` function.
277 |One or more `--label` options.
279 |<<variable-assignment,Variables>>
280 |Mapping of variable names to integral values.
281 |`init_variables` parameter of the `parse()` function.
282 |One or more `--var` options.
285 The available items are:
287 * A <<byte-constant,constant integer>> representing a single byte.
289 * A <<literal-string,literal string>> representing a sequence of bytes
290 encoding UTF-8, UTF-16, or UTF-32 data.
292 * A <<current-byte-order-setting,current byte order setting>> (big or
295 * A <<value,{py3} expression to be evaluated>> as an unsigned or signed
296 integer to be encoded on one or more bytes using the current byte
299 * A <<current-offset-setting,current offset setting>>.
301 * A <<label,label>>, that is, a named constant holding the current
304 This is similar to an assembly label.
306 * A <<variable-assignment,variable assignment>> associating a name to
307 the integral result of an evaluated {py3} expression.
309 * A <<group,group>>, that is, a scoped sequence of items.
311 Moreover, you can <<repetition,repeat>> any item above, except an offset
312 or a label, a given number of times. This is called a repetition.
314 A Normand comment may exist:
316 * Between items, possibly within a group.
317 * Between the nibbles of a constant hexadecimal byte.
318 * Between the bits of a constant binary byte.
319 * Between the last item and the ``pass:[*]`` character of a repetition,
320 and between that ``pass:[*]`` character and the following number.
322 A comment is anything between two ``pass:[#]`` characters on the same
323 line, or from ``pass:[#]`` until the end of the line. Whitespaces and
324 the following symbol characters are also considered comments where a
328 ! @ / \ ? & : ; . , + [ ] _ = | -
331 The latter serve to improve readability so that you may write, for
332 example, a MAC address or a UUID as is.
334 You can test the examples of this section with the `normand`
335 <<command-line-tool,command-line tool>> as such:
338 $ normand file | hexdump -C
341 where `file` is the name of a file containing the Normand input.
345 A _byte constant_ represents a single byte.
350 Two consecutive hexits.
353 A decimal number after the `$` prefix.
356 Eight bits after the `%` prefix.
376 $192 %1100/0011 $ -77
390 58f64689-6316-4d55-8a1a-04cada366172
391 fe80::6257:18ff:fea3:4229
397 58 f6 46 89 63 16 4d 55 8a 1a 04 ca da 36 61 72 ┆ X•F•c•MU•••••6ar
398 fe 80 62 57 18 ff fe a3 42 29 ┆ ••bW••••B)
406 %01110011 %01100001 %01101100 %01110101 %01110100
412 73 61 6c 75 74 ┆ salut
418 A _literal string_ represents the UTF-8-, UTF-16-, or UTF-32-encoded
421 The string to encode isn't implicitly null-terminated: use `\0` at the
422 end of the string to add a null character.
426 . **Optional**: one of the following encodings instead of UTF-8:
436 . The ``pass:["]`` prefix.
438 . A sequence of zero or more characters, possibly containing escape
441 An escape sequence is the ``\`` character followed by one of:
447 `b`:: Backspace (U+0008)
448 `e`:: Escape (U+001B)
449 `f`:: Form feed (U+000C)
450 `n`:: End of line (U+000A)
451 `r`:: Carriage return (U+000D)
452 `t`:: Character tabulation (U+0009)
453 `v`:: Line tabulation (U+000B)
454 ``\``:: Reverse solidus (U+005C)
455 ``pass:["]``:: Quotation mark (U+0022)
458 . The ``pass:["]`` suffix.
464 "coucou tout le monde!"
470 63 6f 75 63 6f 75 20 74 6f 75 74 20 6c 65 20 6d ┆ coucou tout le m
471 6f 6e 64 65 21 ┆ onde!
479 u16le"I am not young enough to know everything."
485 49 00 20 00 61 00 6d 00 20 00 6e 00 6f 00 74 00 ┆ I• •a•m• •n•o•t•
486 20 00 79 00 6f 00 75 00 6e 00 67 00 20 00 65 00 ┆ •y•o•u•n•g• •e•
487 6e 00 6f 00 75 00 67 00 68 00 20 00 74 00 6f 00 ┆ n•o•u•g•h• •t•o•
488 20 00 6b 00 6e 00 6f 00 77 00 20 00 65 00 76 00 ┆ •k•n•o•w• •e•v•
489 65 00 72 00 79 00 74 00 68 00 69 00 6e 00 67 00 ┆ e•r•y•t•h•i•n•g•
498 u32be "\"illusion is the first\nof all pleasures\" 🦉"
504 00 00 00 22 00 00 00 69 00 00 00 6c 00 00 00 6c ┆ •••"•••i•••l•••l
505 00 00 00 75 00 00 00 73 00 00 00 69 00 00 00 6f ┆ •••u•••s•••i•••o
506 00 00 00 6e 00 00 00 20 00 00 00 69 00 00 00 73 ┆ •••n••• •••i•••s
507 00 00 00 20 00 00 00 74 00 00 00 68 00 00 00 65 ┆ ••• •••t•••h•••e
508 00 00 00 20 00 00 00 66 00 00 00 69 00 00 00 72 ┆ ••• •••f•••i•••r
509 00 00 00 73 00 00 00 74 00 00 00 0a 00 00 00 6f ┆ •••s•••t•••••••o
510 00 00 00 66 00 00 00 20 00 00 00 61 00 00 00 6c ┆ •••f••• •••a•••l
511 00 00 00 6c 00 00 00 20 00 00 00 70 00 00 00 6c ┆ •••l••• •••p•••l
512 00 00 00 65 00 00 00 61 00 00 00 73 00 00 00 75 ┆ •••e•••a•••s•••u
513 00 00 00 72 00 00 00 65 00 00 00 73 00 00 00 22 ┆ •••r•••e•••s•••"
514 00 00 00 20 00 01 f9 89 ┆ ••• ••••
518 === Current byte order setting
520 This special item sets the <<cur-bo,_current byte order_>>.
522 The two accepted forms are:
525 ``pass:[{be}]``:: Set the current byte order to big endian.
526 ``pass:[{le}]``:: Set the current byte order to little endian.
530 A _value_ represents a fixed number of bytes encoding an unsigned or
531 signed integer which is the result of evaluating a {py3} expression
532 using the <<cur-bo,current byte order>>.
534 For a value at some source location{nbsp}__**L**__, its {py3} expression
535 may contain the name of any accessible <<label,label>>, including the
536 name of a label defined after{nbsp}__**L**__, as well as the name of any
537 <<variable-assignment,variable>> known at{nbsp}__**L**__.
539 An accessible label is either:
541 * Outside of the current <<group,group>>.
542 * Within the same immediate group (not within a nested group).
544 In the {py3} expression of a value, the value of the special name
545 `ICITTE` is the <<cur-offset,current offset>> (before encoding the
550 . The ``pass:[{]`` prefix.
552 . A valid {py3} expression.
556 . An encoding length in bits amongst `8`, `16`, `24`, `32`, `40`,
557 `48`, `56`, and `64`.
582 # String length in bits
583 {8 * (str_end - str_beg) : 16}
594 00 60 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 ┆ •`hello world!
602 {20 - ICITTE : 8} * 10
608 14 13 12 11 10 0f 0e 0d 0c 0b
612 === Current offset setting
614 This special item sets the <<cur-offset,_current offset_>>.
616 A current offset setting is:
620 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
621 which is the new current offset.
630 <0x61> {ICITTE : 8} * 8
636 00 01 02 03 04 05 06 07 61 62 63 64 65 66 67 68 ┆ ••••••••abcdefgh
644 aa bb cc dd <meow> ee ff
645 <12> 11 22 33 <mix> 44 55
652 aa bb cc dd ee ff 11 22 33 44 55 04 0f ┆ •••••••"3DU••
658 A _label_ associates a name to the <<cur-offset,current offset>>.
660 All the labels of a whole Normand input must have unique names.
662 A label may not share the name of a <<variable-assignment,variable>>
665 A label name may not be `ICITTE` (see <<value>> and
666 <<variable-assignment>> to learn more).
672 . A valid {py3} name which is not `ICITTE`.
676 === Variable assignment
678 A _variable assignment_ associates a name to the integral result of an
679 evaluated {py3} expression.
681 For a variable assignment at some source location{nbsp}__**L**__, its
682 {py3} expression may contain the name of any accessible <<label,label>>,
683 including the name of a label defined after{nbsp}__**L**__, as well as
684 the name of any variable known at{nbsp}__**L**__.
686 An accessible label is either:
688 * Outside of the current <<group,group>>.
689 * Within the same immediate group (not within a nested group).
691 A variable name may not be `ICITTE` (see <<value>> and
692 <<variable-assignment>> to learn more).
694 In the {py3} expression of a variable assignment, the special name
695 `ICITTE` is the <<cur-offset,current offset>>.
699 . The ``pass:[{]`` prefix.
701 . A valid {py3} name which is not `ICITTE`.
705 . A valid {py3} expression.
714 {meow = 42} 11 22 {meow:8} 33 {meow = ICITTE + 17}
715 "yooo" {meow + mix : 16}
721 11 22 2a 33 79 6f 6f 6f 7a 00 ┆ •"*3yoooz•
727 A _group_ is a scoped sequence of items.
729 The <<label,labels>> within a group aren't visible outside of it.
731 The main purpose of a group is to <<repetition,repeat>> more than a
738 . Zero or more items.
746 ((aa bb cc) dd () ee) "leclerc"
752 aa bb cc dd ee 6c 65 63 6c 65 72 63 ┆ •••••leclerc
760 ((aa bb cc) * 3 dd ee) * 5
766 aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa bb
767 cc aa bb cc dd ee aa bb cc aa bb cc aa bb cc dd
768 ee aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa
779 <str_beg> u16le"sébastien diaz" <str_end>
780 {ICITTE - str_beg : 8}
781 {(end - str_beg) * 5 : 24}
789 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
790 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 e0 ┆ n• •d•i•a•z•••••
791 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
792 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 40 ┆ n• •d•i•a•z••••@
793 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
794 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 00 a0 ┆ n• •d•i•a•z•••••
800 A _repetition_ represents the bytes of an item repeated a given number
807 . The ``pass:[*]`` character.
809 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
810 which is the number of times to repeat the previous item.
816 {end - ICITTE - 1 : 8} * 0x100 <end>
822 ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
823 ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
824 df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
825 cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
826 bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
827 af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
828 9f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
829 8f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
830 7f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
831 6f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
832 5f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
833 4f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
834 3f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
835 2f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
836 1f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
837 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
843 If you <<install-normand,installed>> the `normand` package, then you
844 can use the `normand` command-line tool:
847 $ normand <<< '"ma gang de malades"' | hexdump -C
851 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
855 If you copy the `normand.py` module to your own project, then you can
856 run the module itself:
859 $ python3 -m normand <<< '"ma gang de malades"' | hexdump -C
863 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
867 Without a path argument, the `normand` tool reads from the standard
870 The `normand` tool prints the generated binary data to the standard
873 Various options control the initial <<state,state>> of the processor:
874 use the `--help` option to learn more.
878 The whole `normand` package/module API is:
882 class ByteOrder(enum.Enum):
890 VarsT = typing.Dict[str, int]
896 def line_no(self) -> int:
901 def col_no(self) -> int:
905 class ParseError(RuntimeError):
906 # Source text location.
908 def text_loc(self) -> TextLoc:
915 def data(self) -> bytearray:
918 # Updated variable values.
920 def variables(self) -> VarsT:
923 # Updated main group label values.
925 def labels(self) -> VarsT:
930 def offset(self) -> int:
935 def byte_order(self) -> typing.Optional[int]:
938 def parse(normand: str,
939 init_variables: typing.Optional[VarsT] = None,
940 init_labels: typing.Optional[VarsT] = None,
941 init_offset: int = 0,
942 init_byte_order: typing.Optional[ByteOrder] = None) -> ParseResult:
946 The `normand` parameter is the actual <<learn-normand,Normand input>>
947 while the other parameters control the initial <<state,state>>.
949 The `parse()` function raises a `ParseError` instance should it fail to
950 parse the `normand` string for any reason.