3 1) TCM Userspace Design
7 d) Implementation overview
13 g) Other contingencies
14 2) Writing a user pass-through handler
15 a) Discovering and configuring TCMU uio devices
16 b) Waiting for events on the device(s)
17 c) Managing the command ring
18 3) Command filtering and pass_level
25 TCM is another name for LIO, an in-kernel iSCSI target (server).
26 Existing TCM targets run in the kernel. TCMU (TCM in Userspace)
27 allows userspace programs to be written which act as iSCSI targets.
28 This document describes the design.
30 The existing kernel provides modules for different SCSI transport
31 protocols. TCM also modularizes the data storage. There are existing
32 modules for file, block device, RAM or using another SCSI device as
33 storage. These are called "backstores" or "storage engines". These
34 built-in modules are implemented entirely as kernel code.
38 In addition to modularizing the transport protocol used for carrying
39 SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes
40 the actual data storage as well. These are referred to as "backstores"
41 or "storage engines". The target comes with backstores that allow a
42 file, a block device, RAM, or another SCSI device to be used for the
43 local storage needed for the exported SCSI LUN. Like the rest of LIO,
44 these are implemented entirely as kernel code.
46 These backstores cover the most common use cases, but not all. One new
47 use case that other non-kernel target solutions, such as tgt, are able
48 to support is using Gluster's GLFS or Ceph's RBD as a backstore. The
49 target then serves as a translator, allowing initiators to store data
50 in these non-traditional networked storage systems, while still only
51 using standard protocols themselves.
53 If the target is a userspace process, supporting these is easy. tgt,
54 for example, needs only a small adapter module for each, because the
55 modules just use the available userspace libraries for RBD and GLFS.
57 Adding support for these backstores in LIO is considerably more
58 difficult, because LIO is entirely kernel code. Instead of undertaking
59 the significant work to port the GLFS or RBD APIs and protocols to the
60 kernel, another approach is to create a userspace pass-through
61 backstore for LIO, "TCMU".
66 In addition to allowing relatively easy support for RBD and GLFS, TCMU
67 will also allow easier development of new backstores. TCMU combines
68 with the LIO loopback fabric to become something similar to FUSE
69 (Filesystem in Userspace), but at the SCSI layer instead of the
70 filesystem layer. A SUSE, if you will.
72 The disadvantage is there are more distinct components to configure, and
73 potentially to malfunction. This is unavoidable, but hopefully not
74 fatal if we're careful to keep things as simple as possible.
78 - Good performance: high throughput, low latency
79 - Cleanly handle if userspace:
84 - Allow future flexibility in user & kernel implementations
85 - Be reasonably memory-efficient
86 - Simple to configure & run
87 - Simple to write a userspace backend
90 Implementation overview:
92 The core of the TCMU interface is a memory region that is shared
93 between kernel and userspace. Within this region is: a control area
94 (mailbox); a lockless producer/consumer circular buffer for commands
95 to be passed up, and status returned; and an in/out data buffer area.
97 TCMU uses the pre-existing UIO subsystem. UIO allows device driver
98 development in userspace, and this is conceptually very close to the
99 TCMU use case, except instead of a physical device, TCMU implements a
100 memory-mapped layout designed for SCSI commands. Using UIO also
101 benefits TCMU by handling device introspection (e.g. a way for
102 userspace to determine how large the shared region is) and signaling
103 mechanisms in both directions.
105 There are no embedded pointers in the memory region. Everything is
106 expressed as an offset from the region's starting address. This allows
107 the ring to still work if the user process dies and is restarted with
108 the region mapped at a different virtual address.
110 See target_core_user.h for the struct definitions.
114 The mailbox is always at the start of the shared memory region, and
115 contains a version, details about the starting offset and size of the
116 command ring, and head and tail pointers to be used by the kernel and
117 userspace (respectively) to put commands on the ring, and indicate
118 when the commands are completed.
120 version - 1 (userspace should abort if otherwise)
121 flags - none yet defined.
122 cmdr_off - The offset of the start of the command ring from the start
123 of the memory region, to account for the mailbox size.
124 cmdr_size - The size of the command ring. This does *not* need to be a
126 cmd_head - Modified by the kernel to indicate when a command has been
128 cmd_tail - Modified by userspace to indicate when it has completed
129 processing of a command.
133 Commands are placed on the ring by the kernel incrementing
134 mailbox.cmd_head by the size of the command, modulo cmdr_size, and
135 then signaling userspace via uio_event_notify(). Once the command is
136 completed, userspace updates mailbox.cmd_tail in the same way and
137 signals the kernel via a 4-byte write(). When cmd_head equals
138 cmd_tail, the ring is empty -- no commands are currently waiting to be
139 processed by userspace.
141 TCMU commands are 8-byte aligned. They start with a common header
142 containing "len_op", a 32-bit value that stores the length, as well as
143 the opcode in the lowest unused bits. It also contains cmd_id and
144 flags fields for setting by the kernel (kflags) and userspace
147 Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.
149 When the opcode is CMD, the entry in the command ring is a struct
150 tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via
151 tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the
152 overall shared memory region, not the entry. The data in/out buffers
153 are accessible via tht req.iov[] array. iov_cnt contains the number of
154 entries in iov[] needed to describe either the Data-In or Data-Out
155 buffers. For bidirectional commands, iov_cnt specifies how many iovec
156 entries cover the Data-Out area, and iov_bidi_count specifies how many
157 iovec entries immediately after that in iov[] cover the Data-In
158 area. Just like other fields, iov.iov_base is an offset from the start
161 When completing a command, userspace sets rsp.scsi_status, and
162 rsp.sense_buffer if necessary. Userspace then increments
163 mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the
164 kernel via the UIO method, a 4-byte write to the file descriptor.
166 When the opcode is PAD, userspace only updates cmd_tail as above --
167 it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry
168 is contiguous within the command ring.)
170 More opcodes may be added in the future. If userspace encounters an
171 opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in
172 hdr.uflags, update cmd_tail, and proceed with processing additional
177 This is shared-memory space after the command ring. The organization
178 of this area is not defined in the TCMU interface, and userspace
179 should access only the parts referenced by pending iovs.
184 Other devices may be using UIO besides TCMU. Unrelated user processes
185 may also be handling different sets of TCMU devices. TCMU userspace
186 processes must find their devices by scanning sysfs
187 class/uio/uio*/name. For TCMU devices, these names will be of the
190 tcm-user/<hba_num>/<device_name>/<subtype>/<path>
192 where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num>
193 and <device_name> allow userspace to find the device's path in the
194 kernel target's configfs tree. Assuming the usual mount point, it is
197 /sys/kernel/config/target/core/user_<hba_num>/<device_name>
199 This location contains attributes such as "hw_block_size", that
200 userspace needs to know for correct operation.
202 <subtype> will be a userspace-process-unique string to identify the
203 TCMU device as expecting to be backed by a certain handler, and <path>
204 will be an additional handler-specific string for the user process to
205 configure the device, if needed. The name cannot contain ':', due to
208 For all devices so discovered, the user handler opens /dev/uioX and
211 mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
213 where size must be equal to the value read from
214 /sys/class/uio/uioX/maps/map0/size.
219 If a new device is added or removed, a notification will be broadcast
220 over netlink, using a generic netlink family name of "TCM-USER" and a
221 multicast group named "config". This will include the UIO name as
222 described in the previous section, as well as the UIO minor
223 number. This should allow userspace to identify both the UIO device and
224 the LIO device, so that after determining the device is supported
225 (based on subtype) it can take the appropriate action.
230 Userspace handler process never attaches:
232 - TCMU will post commands, and then abort them after a timeout period
235 Userspace handler process is killed:
237 - It is still possible to restart and re-connect to TCMU
238 devices. Command ring is preserved. However, after the timeout period,
239 the kernel will abort pending tasks.
241 Userspace handler process hangs:
243 - The kernel will abort pending tasks after a timeout period.
245 Userspace handler process is malicious:
247 - The process can trivially break the handling of devices it controls,
248 but should not be able to access kernel memory outside its shared
252 Writing a user pass-through handler (with example code)
253 -------------------------------------------------------
255 A user process handing a TCMU device must support the following:
257 a) Discovering and configuring TCMU uio devices
258 b) Waiting for events on the device(s)
259 c) Managing the command ring: Parsing operations and commands,
260 performing work as needed, setting response fields (scsi_status and
261 possibly sense_buffer), updating cmd_tail, and notifying the kernel
262 that work has been finished
264 First, consider instead writing a plugin for tcmu-runner. tcmu-runner
265 implements all of this, and provides a higher-level API for plugin
268 TCMU is designed so that multiple unrelated processes can manage TCMU
269 devices separately. All handlers should make sure to only open their
270 devices, based opon a known subtype string.
272 a) Discovering and configuring TCMU UIO devices:
274 (error checking omitted for brevity)
278 unsigned long long map_len;
281 fd = open("/sys/class/uio/uio0/name", O_RDONLY);
282 ret = read(fd, buf, sizeof(buf));
284 buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
286 /* we only want uio devices whose name is a format we expect */
287 if (strncmp(buf, "tcm-user", 8))
290 /* Further checking for subtype also needed here */
292 fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY);
293 ret = read(fd, buf, sizeof(buf));
295 str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
297 map_len = strtoull(buf, NULL, 0);
299 dev_fd = open("/dev/uio0", O_RDWR);
300 map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0);
303 b) Waiting for events on the device(s)
308 int ret = read(dev_fd, buf, 4); /* will block */
310 handle_device_events(dev_fd, map);
314 c) Managing the command ring
316 #include <linux/target_core_user.h>
318 int handle_device_events(int fd, void *map)
320 struct tcmu_mailbox *mb = map;
321 struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
322 int did_some_work = 0;
324 /* Process events from cmd ring until we catch up with cmd_head */
325 while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) {
327 if (tcmu_hdr_get_op(&ent->hdr) == TCMU_OP_CMD) {
328 uint8_t *cdb = (void *)mb + ent->req.cdb_off;
331 /* Handle command here. */
332 printf("SCSI opcode: 0x%x\n", cdb[0]);
334 /* Set response fields */
336 ent->rsp.scsi_status = SCSI_NO_SENSE;
338 /* Also fill in rsp->sense_buffer here */
339 ent->rsp.scsi_status = SCSI_CHECK_CONDITION;
343 /* Do nothing for PAD entries */
346 /* update cmd_tail */
347 mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size;
348 ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
352 /* Notify the kernel that work has been finished */
363 Command filtering and pass_level
364 --------------------------------
366 TCMU supports a "pass_level" option with valid values of 0 or 1. When
367 the value is 0 (the default), nearly all SCSI commands received for
368 the device are passed through to the handler. This allows maximum
369 flexibility but increases the amount of code required by the handler,
370 to support all mandatory SCSI commands. If pass_level is set to 1,
371 then only IO-related commands are presented, and the rest are handled
372 by LIO's in-kernel command emulation. The commands presented at level
373 1 include all versions of:
388 Please be careful to return codes as defined by the SCSI
389 specifications. These are different than some values defined in the
390 scsi/scsi.h include file. For example, CHECK CONDITION's status code