[deliverable/linux.git] / Documentation / networking / pktgen.txt



                  HOWTO for the linux packet generator
                  ------------------------------------

Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
or as a module.  A module is preferred; modprobe pktgen if needed.  Once
running, pktgen creates a thread for each CPU with affinity to that CPU.
Monitoring and controlling is done via /proc.  It is easiest to select a
suitable sample script and configure that.

On a dual CPU:

ps aux | grep pkt
root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [pktgen/0]
root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [pktgen/1]


For monitoring and control pktgen creates:
	/proc/net/pktgen/pgctrl
	/proc/net/pktgen/kpktgend_X
        /proc/net/pktgen/ethX


Tuning NIC for max performance
==============================

The default NIC settings are (likely) not tuned for pktgen's artificial
overload type of benchmarking, as this could hurt the normal use-case.

Specifically increasing the TX ring buffer in the NIC:
 # ethtool -G ethX tx 1024

A larger TX ring can improve pktgen's performance, while it can hurt
in the general case, 1) because the TX ring buffer might get larger
than the CPU's L1/L2 cache, 2) because it allows more queueing in the
NIC HW layer (which is bad for bufferbloat).

One should hesitate to conclude that packets/descriptors in the HW
TX ring cause delay.  Drivers usually delay cleaning up the
ring-buffers for various performance reasons, and packets stalling
the TX ring might just be waiting for cleanup.

This cleanup issue is specifically the case for the driver ixgbe
(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
and the cleanup interval is affected by the ethtool --coalesce setting
of parameter "rx-usecs".

For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
 # ethtool -C ethX rx-usecs 30


Kernel threads
==============
Pktgen creates a thread for each CPU with affinity to that CPU.
Which is controlled through procfile /proc/net/pktgen/kpktgend_X.

Example: /proc/net/pktgen/kpktgend_0

 Running:
 Stopped: eth4@0
 Result: OK: add_device=eth4@0

Most important are the devices assigned to the thread.

The two basic thread commands are:
 * add_device DEVICE@NAME -- adds a single device
 * rem_device_all         -- remove all associated devices

When adding a device to a thread, a corresponding procfile is created
which is used for configuring this device. Thus, device names need to
be unique.

To support adding the same device to multiple threads, which is useful
with multi queue NICs, the device naming scheme is extended with "@":
 device@something

The part after "@" can be anything, but it is custom to use the thread
number.

Viewing devices
===============

The Params section holds configured information.  The Current section
holds running statistics.  The Result is printed after a run or after
interruption.  Example:

/proc/net/pktgen/eth4@0

 Params: count 100000  min_pkt_size: 60  max_pkt_size: 60
     frags: 0  delay: 0  clone_skb: 64  ifname: eth4@0
     flows: 0 flowlen: 0
     queue_map_min: 0  queue_map_max: 0
     dst_min: 192.168.81.2  dst_max:
     src_min:   src_max:
     src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
     udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
     src_mac_count: 0  dst_mac_count: 0
     Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_CPU
 Current:
     pkts-sofar: 100000  errors: 0
     started: 623913381008us  stopped: 623913396439us idle: 25us
     seq_num: 100001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
     cur_saddr: 192.168.8.3  cur_daddr: 192.168.81.2
     cur_udp_dst: 9  cur_udp_src: 42
     cur_queue_map: 0
     flows: 0
 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
  6480562pps 3110Mb/sec (3110669760bps) errors: 0


Configuring devices
===================
This is done via the /proc interface, and most easily done via pgset
as defined in the sample scripts.

Examples:

 pgset "clone_skb 1"     sets the number of copies of the same packet
 pgset "clone_skb 0"     use single SKB for all transmits
 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
                         packet and update HW tx queue tail pointer once.
                         "burst 1" is the default
 pgset "pkt_size 9014"   sets packet size to 9014
 pgset "frags 5"         packet will consist of 5 fragments
 pgset "count 200000"    sets number of packets to send, set to zero
                         for continuous sends until explicitly stopped.

 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds

 pgset "dst 10.0.0.1"    sets IP destination address
                         (BEWARE! This generator is very aggressive!)

 pgset "dst_min 10.0.0.1"            Same as dst
 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
 pgset "src_max 10.0.0.254"          Set the maximum source IP.
 pgset "dst6 fec0::1"     IPV6 destination address
 pgset "src6 fec0::2"     IPV6 source address
 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address

 pgset "queue_map_min 0" Sets the min value of tx queue interval
 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
                         To select queue 1 of a given device,
                         use queue_map_min=1 and queue_map_max=1

 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
                         The 'minimum' MAC is what you set with srcmac.

 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
                         The 'minimum' MAC is what you set with dstmac.

 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
                         are: IPSRC_RND # IP source is random (between min/max)
                              IPDST_RND # IP destination is random
                              UDPSRC_RND, UDPDST_RND,
                              MACSRC_RND, MACDST_RND
                              TXSIZE_RND, IPV6,
                              MPLS_RND, VID_RND, SVID_RND
                              FLOW_SEQ,
                              QUEUE_MAP_RND # queue map random
                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
                              UDPCSUM,
                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
                              NODE_ALLOC # node specific memory allocation
                              NO_TIMESTAMP # disable timestamping

 pgset spi SPI_VALUE     Set specific SA used to transform packet.

 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
                         cycle through the port range.

 pgset "udp_src_max 9"   set UDP source port max.
 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
                         cycle through the port range.
 pgset "udp_dst_max 9"   set UDP destination port max.

 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
                                         outer label=16,middle label=32,
					 inner label=0 (IPv4 NULL)) Note that
					 there must be no spaces between the
					 arguments. Leading zeros are required.
					 Do not set the bottom of stack bit,
					 that's done automatically. If you do
					 set the bottom of stack bit, that
					 indicates that you want to randomly
					 generate that address and the flag
					 MPLS_RND will be turned on. You
					 can have any mix of random and fixed
					 labels in the label stack.

 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)

 pgset "vlan_id 77"       set VLAN ID 0-4095
 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)

 pgset "svlan_id 22"      set SVLAN ID 0-4095
 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)

 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
 pgset "svlan 9999"       > 4095 remove svlan tag


 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)

 pgset stop    	          aborts injection. Also, ^C aborts generator.

 pgset "rate 300M"        set rate to 300 Mb/s
 pgset "ratep 1000000"    set rate to 1Mpps

 pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
				  Works with "burst" but not with "clone_skb".
				  Default xmit_mode is "start_xmit".

Sample scripts
==============

A collection of tutorial scripts and helpers for pktgen is in the
samples/pktgen directory. The helper parameters.sh file support easy
and consistent parameter parsing across the sample scripts.

Usage example and help:
 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2

Usage: ./pktgen_sample01_simple.sh [-vx] -i ethX
  -i : ($DEV)       output interface/device (required)
  -s : ($PKT_SIZE)  packet size
  -d : ($DEST_IP)   destination IP
  -m : ($DST_MAC)   destination MAC-addr
  -t : ($THREADS)   threads to start
  -c : ($SKB_CLONE) SKB clones send before alloc new SKB
  -b : ($BURST)     HW level bursting of SKBs
  -v : ($VERBOSE)   verbose
  -x : ($DEBUG)     debug

The global variables being set are also listed.  E.g. the required
interface/device parameter "-i" sets variable $DEV.  Copy the
pktgen_sampleXX scripts and modify them to fit your own needs.

The old scripts:

pktgen.conf-1-2                  # 1 CPU 2 dev
pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS 
pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.


Interrupt affinity
===================
Note that when adding devices to a specific CPU it is a good idea to
also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
to the same CPU.  This reduces cache bouncing when freeing skbs.

Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
to the running threads CPU (directly from smp_processor_id()).

Enable IPsec
============
Default IPsec transformation with ESP encapsulation plus transport mode
can be enabled by simply setting:

pgset "flag IPSEC"
pgset "flows 1"

To avoid breaking existing testbed scripts for using AH type and tunnel mode,
you can use "pgset spi SPI_VALUE" to specify which transformation mode
to employ.


Current commands and configuration options
==========================================

** Pgcontrol commands:

start
stop
reset

** Thread commands:

add_device
rem_device_all


** Device commands:

count
clone_skb
burst
debug

frags
delay

src_mac_count
dst_mac_count

pkt_size
min_pkt_size
max_pkt_size

queue_map_min
queue_map_max
skb_priority

tos           (ipv4)
traffic_class (ipv6)

mpls

udp_src_min
udp_src_max

udp_dst_min
udp_dst_max

node

flag
  IPSRC_RND
  IPDST_RND
  UDPSRC_RND
  UDPDST_RND
  MACSRC_RND
  MACDST_RND
  TXSIZE_RND
  IPV6
  MPLS_RND
  VID_RND
  SVID_RND
  FLOW_SEQ
  QUEUE_MAP_RND
  QUEUE_MAP_CPU
  UDPCSUM
  IPSEC
  NODE_ALLOC
  NO_TIMESTAMP

spi (ipsec)

dst_min
dst_max

src_min
src_max

dst_mac
src_mac

clear_counters

src6
dst6
dst6_max
dst6_min

flows
flowlen

rate
ratep

xmit_mode <start_xmit|netif_receive>

vlan_cfi
vlan_id
vlan_p

svlan_cfi
svlan_id
svlan_p


References:
ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/

Paper from Linux-Kongress in Erlangen 2004.
ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf

Thanks to:
Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
Stephen Hemminger, Andi Kleen, Dave Miller and many others.


Good luck with the linux net-development.
Commit	Line	Data
	1
	2
	3	HOWTO for the linux packet generator
	4	------------------------------------
	5
	6	Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
	7	or as a module. A module is preferred; modprobe pktgen if needed. Once
	8	running, pktgen creates a thread for each CPU with affinity to that CPU.
	9	Monitoring and controlling is done via /proc. It is easiest to select a
	10	suitable sample script and configure that.
	11
	12	On a dual CPU:
	13
	14	ps aux \| grep pkt
	15	root 129 0.3 0.0 0 0 ? SW 2003 523:20 [pktgen/0]
	16	root 130 0.3 0.0 0 0 ? SW 2003 509:50 [pktgen/1]
	17
	18
	19	For monitoring and control pktgen creates:
	20	/proc/net/pktgen/pgctrl
	21	/proc/net/pktgen/kpktgend_X
	22	/proc/net/pktgen/ethX
	23
	24
	25	Tuning NIC for max performance
	26	==============================
	27
	28	The default NIC settings are (likely) not tuned for pktgen's artificial
	29	overload type of benchmarking, as this could hurt the normal use-case.
	30
	31	Specifically increasing the TX ring buffer in the NIC:
	32	# ethtool -G ethX tx 1024
	33
	34	A larger TX ring can improve pktgen's performance, while it can hurt
	35	in the general case, 1) because the TX ring buffer might get larger
	36	than the CPU's L1/L2 cache, 2) because it allows more queueing in the
	37	NIC HW layer (which is bad for bufferbloat).
	38
	39	One should hesitate to conclude that packets/descriptors in the HW
	40	TX ring cause delay. Drivers usually delay cleaning up the
	41	ring-buffers for various performance reasons, and packets stalling
	42	the TX ring might just be waiting for cleanup.
	43
	44	This cleanup issue is specifically the case for the driver ixgbe
	45	(Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups,
	46	and the cleanup interval is affected by the ethtool --coalesce setting
	47	of parameter "rx-usecs".
	48
	49	For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
	50	# ethtool -C ethX rx-usecs 30
	51
	52
	53	Kernel threads
	54	==============
	55	Pktgen creates a thread for each CPU with affinity to that CPU.
	56	Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
	57
	58	Example: /proc/net/pktgen/kpktgend_0
	59
	60	Running:
	61	Stopped: eth4@0
	62	Result: OK: add_device=eth4@0
	63
	64	Most important are the devices assigned to the thread.
	65
	66	The two basic thread commands are:
	67	* add_device DEVICE@NAME -- adds a single device
	68	* rem_device_all -- remove all associated devices
	69
	70	When adding a device to a thread, a corresponding procfile is created
	71	which is used for configuring this device. Thus, device names need to
	72	be unique.
	73
	74	To support adding the same device to multiple threads, which is useful
	75	with multi queue NICs, the device naming scheme is extended with "@":
	76	device@something
	77
	78	The part after "@" can be anything, but it is custom to use the thread
	79	number.
	80
	81	Viewing devices
	82	===============
	83
	84	The Params section holds configured information. The Current section
	85	holds running statistics. The Result is printed after a run or after
	86	interruption. Example:
	87
	88	/proc/net/pktgen/eth4@0
	89
	90	Params: count 100000 min_pkt_size: 60 max_pkt_size: 60
	91	frags: 0 delay: 0 clone_skb: 64 ifname: eth4@0
	92	flows: 0 flowlen: 0
	93	queue_map_min: 0 queue_map_max: 0
	94	dst_min: 192.168.81.2 dst_max:
	95	src_min: src_max:
	96	src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
	97	udp_src_min: 9 udp_src_max: 109 udp_dst_min: 9 udp_dst_max: 9
	98	src_mac_count: 0 dst_mac_count: 0
	99	Flags: UDPSRC_RND NO_TIMESTAMP QUEUE_MAP_CPU
	100	Current:
	101	pkts-sofar: 100000 errors: 0
	102	started: 623913381008us stopped: 623913396439us idle: 25us
	103	seq_num: 100001 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
	104	cur_saddr: 192.168.8.3 cur_daddr: 192.168.81.2
	105	cur_udp_dst: 9 cur_udp_src: 42
	106	cur_queue_map: 0
	107	flows: 0
	108	Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
	109	6480562pps 3110Mb/sec (3110669760bps) errors: 0
	110
	111
	112	Configuring devices
	113	===================
	114	This is done via the /proc interface, and most easily done via pgset
	115	as defined in the sample scripts.
	116
	117	Examples:
	118
	119	pgset "clone_skb 1" sets the number of copies of the same packet
	120	pgset "clone_skb 0" use single SKB for all transmits
	121	pgset "burst 8" uses xmit_more API to queue 8 copies of the same
	122	packet and update HW tx queue tail pointer once.
	123	"burst 1" is the default
	124	pgset "pkt_size 9014" sets packet size to 9014
	125	pgset "frags 5" packet will consist of 5 fragments
	126	pgset "count 200000" sets number of packets to send, set to zero
	127	for continuous sends until explicitly stopped.
	128
	129	pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds
	130
	131	pgset "dst 10.0.0.1" sets IP destination address
	132	(BEWARE! This generator is very aggressive!)
	133
	134	pgset "dst_min 10.0.0.1" Same as dst
	135	pgset "dst_max 10.0.0.254" Set the maximum destination IP.
	136	pgset "src_min 10.0.0.1" Set the minimum (or only) source IP.
	137	pgset "src_max 10.0.0.254" Set the maximum source IP.
	138	pgset "dst6 fec0::1" IPV6 destination address
	139	pgset "src6 fec0::2" IPV6 source address
	140	pgset "dstmac 00:00:00:00:00:00" sets MAC destination address
	141	pgset "srcmac 00:00:00:00:00:00" sets MAC source address
	142
	143	pgset "queue_map_min 0" Sets the min value of tx queue interval
	144	pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
	145	To select queue 1 of a given device,
	146	use queue_map_min=1 and queue_map_max=1
	147
	148	pgset "src_mac_count 1" Sets the number of MACs we'll range through.
	149	The 'minimum' MAC is what you set with srcmac.
	150
	151	pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
	152	The 'minimum' MAC is what you set with dstmac.
	153
	154	pgset "flag [name]" Set a flag to determine behaviour. Current flags
	155	are: IPSRC_RND # IP source is random (between min/max)
	156	IPDST_RND # IP destination is random
	157	UDPSRC_RND, UDPDST_RND,
	158	MACSRC_RND, MACDST_RND
	159	TXSIZE_RND, IPV6,
	160	MPLS_RND, VID_RND, SVID_RND
	161	FLOW_SEQ,
	162	QUEUE_MAP_RND # queue map random
	163	QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
	164	UDPCSUM,
	165	IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
	166	NODE_ALLOC # node specific memory allocation
	167	NO_TIMESTAMP # disable timestamping
	168
	169	pgset spi SPI_VALUE Set specific SA used to transform packet.
	170
	171	pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then
	172	cycle through the port range.
	173
	174	pgset "udp_src_max 9" set UDP source port max.
	175	pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then
	176	cycle through the port range.
	177	pgset "udp_dst_max 9" set UDP destination port max.
	178
	179	pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
	180	outer label=16,middle label=32,
	181	inner label=0 (IPv4 NULL)) Note that
	182	there must be no spaces between the
	183	arguments. Leading zeros are required.
	184	Do not set the bottom of stack bit,
	185	that's done automatically. If you do
	186	set the bottom of stack bit, that
	187	indicates that you want to randomly
	188	generate that address and the flag
	189	MPLS_RND will be turned on. You
	190	can have any mix of random and fixed
	191	labels in the label stack.
	192
	193	pgset "mpls 0" turn off mpls (or any invalid argument works too!)
	194
	195	pgset "vlan_id 77" set VLAN ID 0-4095
	196	pgset "vlan_p 3" set priority bit 0-7 (default 0)
	197	pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0)
	198
	199	pgset "svlan_id 22" set SVLAN ID 0-4095
	200	pgset "svlan_p 3" set priority bit 0-7 (default 0)
	201	pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0)
	202
	203	pgset "vlan_id 9999" > 4095 remove vlan and svlan tags
	204	pgset "svlan 9999" > 4095 remove svlan tag
	205
	206
	207	pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
	208	pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
	209
	210	pgset stop aborts injection. Also, ^C aborts generator.
	211
	212	pgset "rate 300M" set rate to 300 Mb/s
	213	pgset "ratep 1000000" set rate to 1Mpps
	214
	215	pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb()
	216	Works with "burst" but not with "clone_skb".
	217	Default xmit_mode is "start_xmit".
	218
	219	Sample scripts
	220	==============
	221
	222	A collection of tutorial scripts and helpers for pktgen is in the
	223	samples/pktgen directory. The helper parameters.sh file support easy
	224	and consistent parameter parsing across the sample scripts.
	225
	226	Usage example and help:
	227	./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
	228
	229	Usage: ./pktgen_sample01_simple.sh [-vx] -i ethX
	230	-i : ($DEV) output interface/device (required)
	231	-s : ($PKT_SIZE) packet size
	232	-d : ($DEST_IP) destination IP
	233	-m : ($DST_MAC) destination MAC-addr
	234	-t : ($THREADS) threads to start
	235	-c : ($SKB_CLONE) SKB clones send before alloc new SKB
	236	-b : ($BURST) HW level bursting of SKBs
	237	-v : ($VERBOSE) verbose
	238	-x : ($DEBUG) debug
	239
	240	The global variables being set are also listed. E.g. the required
	241	interface/device parameter "-i" sets variable $DEV. Copy the
	242	pktgen_sampleXX scripts and modify them to fit your own needs.
	243
	244	The old scripts:
	245
	246	pktgen.conf-1-2 # 1 CPU 2 dev
	247	pktgen.conf-1-1-rdos # 1 CPU 1 dev w. route DoS
	248	pktgen.conf-1-1-ip6 # 1 CPU 1 dev ipv6
	249	pktgen.conf-1-1-ip6-rdos # 1 CPU 1 dev ipv6 w. route DoS
	250	pktgen.conf-1-1-flows # 1 CPU 1 dev multiple flows.
	251
	252
	253	Interrupt affinity
	254	===================
	255	Note that when adding devices to a specific CPU it is a good idea to
	256	also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
	257	to the same CPU. This reduces cache bouncing when freeing skbs.
	258
	259	Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
	260	to the running threads CPU (directly from smp_processor_id()).
	261
	262	Enable IPsec
	263	============
	264	Default IPsec transformation with ESP encapsulation plus transport mode
	265	can be enabled by simply setting:
	266
	267	pgset "flag IPSEC"
	268	pgset "flows 1"
	269
	270	To avoid breaking existing testbed scripts for using AH type and tunnel mode,
	271	you can use "pgset spi SPI_VALUE" to specify which transformation mode
	272	to employ.
	273
	274
	275	Current commands and configuration options
	276	==========================================
	277
	278	** Pgcontrol commands:
	279
	280	start
	281	stop
	282	reset
	283
	284	** Thread commands:
	285
	286	add_device
	287	rem_device_all
	288
	289
	290	** Device commands:
	291
	292	count
	293	clone_skb
	294	burst
	295	debug
	296
	297	frags
	298	delay
	299
	300	src_mac_count
	301	dst_mac_count
	302
	303	pkt_size
	304	min_pkt_size
	305	max_pkt_size
	306
	307	queue_map_min
	308	queue_map_max
	309	skb_priority
	310
	311	tos (ipv4)
	312	traffic_class (ipv6)
	313
	314	mpls
	315
	316	udp_src_min
	317	udp_src_max
	318
	319	udp_dst_min
	320	udp_dst_max
	321
	322	node
	323
	324	flag
	325	IPSRC_RND
	326	IPDST_RND
	327	UDPSRC_RND
	328	UDPDST_RND
	329	MACSRC_RND
	330	MACDST_RND
	331	TXSIZE_RND
	332	IPV6
	333	MPLS_RND
	334	VID_RND
	335	SVID_RND
	336	FLOW_SEQ
	337	QUEUE_MAP_RND
	338	QUEUE_MAP_CPU
	339	UDPCSUM
	340	IPSEC
	341	NODE_ALLOC
	342	NO_TIMESTAMP
	343
	344	spi (ipsec)
	345
	346	dst_min
	347	dst_max
	348
	349	src_min
	350	src_max
	351
	352	dst_mac
	353	src_mac
	354
	355	clear_counters
	356
	357	src6
	358	dst6
	359	dst6_max
	360	dst6_min
	361
	362	flows
	363	flowlen
	364
	365	rate
	366	ratep
	367
	368	xmit_mode <start_xmit\|netif_receive>
	369
	370	vlan_cfi
	371	vlan_id
	372	vlan_p
	373
	374	svlan_cfi
	375	svlan_id
	376	svlan_p
	377
	378
	379	References:
	380	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
	381	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
	382
	383	Paper from Linux-Kongress in Erlangen 2004.
	384	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
	385
	386	Thanks to:
	387	Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek
	388	Stephen Hemminger, Andi Kleen, Dave Miller and many others.
	389
	390
	391	Good luck with the linux net-development.