Commit | Line | Data |
---|---|---|
c6b4fcba JT |
1 | Introduction |
2 | ============ | |
3 | ||
4 | dm-cache is a device mapper target written by Joe Thornber, Heinz | |
5 | Mauelshagen, and Mike Snitzer. | |
6 | ||
7 | It aims to improve performance of a block device (eg, a spindle) by | |
8 | dynamically migrating some of its data to a faster, smaller device | |
9 | (eg, an SSD). | |
10 | ||
11 | This device-mapper solution allows us to insert this caching at | |
12 | different levels of the dm stack, for instance above the data device for | |
13 | a thin-provisioning pool. Caching solutions that are integrated more | |
14 | closely with the virtual memory system should give better performance. | |
15 | ||
16 | The target reuses the metadata library used in the thin-provisioning | |
17 | library. | |
18 | ||
19 | The decision as to what data to migrate and when is left to a plug-in | |
20 | policy module. Several of these have been written as we experiment, | |
21 | and we hope other people will contribute others for specific io | |
22 | scenarios (eg. a vm image server). | |
23 | ||
24 | Glossary | |
25 | ======== | |
26 | ||
27 | Migration - Movement of the primary copy of a logical block from one | |
28 | device to the other. | |
29 | Promotion - Migration from slow device to fast device. | |
30 | Demotion - Migration from fast device to slow device. | |
31 | ||
32 | The origin device always contains a copy of the logical block, which | |
33 | may be out of date or kept in sync with the copy on the cache device | |
34 | (depending on policy). | |
35 | ||
36 | Design | |
37 | ====== | |
38 | ||
39 | Sub-devices | |
40 | ----------- | |
41 | ||
42 | The target is constructed by passing three devices to it (along with | |
43 | other parameters detailed later): | |
44 | ||
45 | 1. An origin device - the big, slow one. | |
46 | ||
47 | 2. A cache device - the small, fast one. | |
48 | ||
49 | 3. A small metadata device - records which blocks are in the cache, | |
50 | which are dirty, and extra hints for use by the policy object. | |
51 | This information could be put on the cache device, but having it | |
52 | separate allows the volume manager to configure it differently, | |
66bb2644 MS |
53 | e.g. as a mirror for extra robustness. This metadata device may only |
54 | be used by a single cache device. | |
c6b4fcba JT |
55 | |
56 | Fixed block size | |
57 | ---------------- | |
58 | ||
59 | The origin is divided up into blocks of a fixed size. This block size | |
60 | is configurable when you first create the cache. Typically we've been | |
05473044 MS |
61 | using block sizes of 256KB - 1024KB. The block size must be between 64 |
62 | (32KB) and 2097152 (1GB) and a multiple of 64 (32KB). | |
c6b4fcba JT |
63 | |
64 | Having a fixed block size simplifies the target a lot. But it is | |
65 | something of a compromise. For instance, a small part of a block may be | |
66 | getting hit a lot, yet the whole block will be promoted to the cache. | |
67 | So large block sizes are bad because they waste cache space. And small | |
68 | block sizes are bad because they increase the amount of metadata (both | |
69 | in core and on disk). | |
70 | ||
2ee57d58 JT |
71 | Cache operating modes |
72 | --------------------- | |
c6b4fcba | 73 | |
2ee57d58 JT |
74 | The cache has three operating modes: writeback, writethrough and |
75 | passthrough. | |
c6b4fcba JT |
76 | |
77 | If writeback, the default, is selected then a write to a block that is | |
78 | cached will go only to the cache and the block will be marked dirty in | |
79 | the metadata. | |
80 | ||
81 | If writethrough is selected then a write to a cached block will not | |
82 | complete until it has hit both the origin and cache devices. Clean | |
83 | blocks should remain clean. | |
84 | ||
2ee57d58 JT |
85 | If passthrough is selected, useful when the cache contents are not known |
86 | to be coherent with the origin device, then all reads are served from | |
87 | the origin device (all reads miss the cache) and all writes are | |
88 | forwarded to the origin device; additionally, write hits cause cache | |
89 | block invalidates. Passthrough mode allows a cache device to be | |
90 | activated without having to worry about coherency. Coherency that | |
91 | exists is maintained, although the cache will gradually cool as writes | |
92 | take place. If the coherency of the cache can later be verified, or | |
93 | established, the cache device can can be transitioned to writethrough or | |
94 | writeback mode while still warm. Otherwise, the cache contents can be | |
95 | discarded prior to transitioning to the desired operating mode. | |
96 | ||
c6b4fcba JT |
97 | A simple cleaner policy is provided, which will clean (write back) all |
98 | dirty blocks in a cache. Useful for decommissioning a cache. | |
99 | ||
100 | Migration throttling | |
101 | -------------------- | |
102 | ||
103 | Migrating data between the origin and cache device uses bandwidth. | |
104 | The user can set a throttle to prevent more than a certain amount of | |
f884ab15 | 105 | migration occurring at any one time. Currently we're not taking any |
c6b4fcba JT |
106 | account of normal io traffic going to the devices. More work needs |
107 | doing here to avoid migrating during those peak io moments. | |
108 | ||
109 | For the time being, a message "migration_threshold <#sectors>" | |
110 | can be used to set the maximum number of sectors being migrated, | |
111 | the default being 204800 sectors (or 100MB). | |
112 | ||
113 | Updating on-disk metadata | |
114 | ------------------------- | |
115 | ||
116 | On-disk metadata is committed every time a REQ_SYNC or REQ_FUA bio is | |
117 | written. If no such requests are made then commits will occur every | |
118 | second. This means the cache behaves like a physical disk that has a | |
119 | write cache (the same is true of the thin-provisioning target). If | |
120 | power is lost you may lose some recent writes. The metadata should | |
121 | always be consistent in spite of any crash. | |
122 | ||
123 | The 'dirty' state for a cache block changes far too frequently for us | |
124 | to keep updating it on the fly. So we treat it as a hint. In normal | |
125 | operation it will be written when the dm device is suspended. If the | |
126 | system crashes all cache blocks will be assumed dirty when restarted. | |
127 | ||
128 | Per-block policy hints | |
129 | ---------------------- | |
130 | ||
131 | Policy plug-ins can store a chunk of data per cache block. It's up to | |
132 | the policy how big this chunk is, but it should be kept small. Like the | |
133 | dirty flags this data is lost if there's a crash so a safe fallback | |
134 | value should always be possible. | |
135 | ||
136 | For instance, the 'mq' policy, which is currently the default policy, | |
137 | uses this facility to store the hit count of the cache blocks. If | |
138 | there's a crash this information will be lost, which means the cache | |
139 | may be less efficient until those hit counts are regenerated. | |
140 | ||
141 | Policy hints affect performance, not correctness. | |
142 | ||
143 | Policy messaging | |
144 | ---------------- | |
145 | ||
146 | Policies will have different tunables, specific to each one, so we | |
147 | need a generic way of getting and setting these. Device-mapper | |
148 | messages are used. Refer to cache-policies.txt. | |
149 | ||
150 | Discard bitset resolution | |
151 | ------------------------- | |
152 | ||
153 | We can avoid copying data during migration if we know the block has | |
154 | been discarded. A prime example of this is when mkfs discards the | |
155 | whole block device. We store a bitset tracking the discard state of | |
156 | blocks. However, we allow this bitset to have a different block size | |
157 | from the cache blocks. This is because we need to track the discard | |
158 | state for all of the origin device (compare with the dirty bitset | |
159 | which is just for the smaller cache device). | |
160 | ||
161 | Target interface | |
162 | ================ | |
163 | ||
164 | Constructor | |
165 | ----------- | |
166 | ||
167 | cache <metadata dev> <cache dev> <origin dev> <block size> | |
168 | <#feature args> [<feature arg>]* | |
169 | <policy> <#policy args> [policy args]* | |
170 | ||
171 | metadata dev : fast device holding the persistent metadata | |
172 | cache dev : fast device holding cached data blocks | |
173 | origin dev : slow device holding original data blocks | |
174 | block size : cache unit size in sectors | |
175 | ||
176 | #feature args : number of feature arguments passed | |
177 | feature args : writethrough. (The default is writeback.) | |
178 | ||
179 | policy : the replacement policy to use | |
180 | #policy args : an even number of arguments corresponding to | |
181 | key/value pairs passed to the policy | |
182 | policy args : key/value pairs passed to the policy | |
183 | E.g. 'sequential_threshold 1024' | |
184 | See cache-policies.txt for details. | |
185 | ||
186 | Optional feature arguments are: | |
187 | writethrough : write through caching that prohibits cache block | |
188 | content from being different from origin block content. | |
189 | Without this argument, the default behaviour is to write | |
190 | back cache block contents later for performance reasons, | |
191 | so they may differ from the corresponding origin blocks. | |
192 | ||
193 | A policy called 'default' is always registered. This is an alias for | |
194 | the policy we currently think is giving best all round performance. | |
195 | ||
196 | As the default policy could vary between kernels, if you are relying on | |
197 | the characteristics of a specific policy, always request it by name. | |
198 | ||
199 | Status | |
200 | ------ | |
201 | ||
202 | <#used metadata blocks>/<#total metadata blocks> <#read hits> <#read misses> | |
203 | <#write hits> <#write misses> <#demotions> <#promotions> <#blocks in cache> | |
204 | <#dirty> <#features> <features>* <#core args> <core args>* <#policy args> | |
205 | <policy args>* | |
206 | ||
207 | #used metadata blocks : Number of metadata blocks used | |
208 | #total metadata blocks : Total number of metadata blocks | |
209 | #read hits : Number of times a READ bio has been mapped | |
210 | to the cache | |
211 | #read misses : Number of times a READ bio has been mapped | |
212 | to the origin | |
213 | #write hits : Number of times a WRITE bio has been mapped | |
214 | to the cache | |
215 | #write misses : Number of times a WRITE bio has been | |
216 | mapped to the origin | |
217 | #demotions : Number of times a block has been removed | |
218 | from the cache | |
219 | #promotions : Number of times a block has been moved to | |
220 | the cache | |
221 | #blocks in cache : Number of blocks resident in the cache | |
222 | #dirty : Number of blocks in the cache that differ | |
223 | from the origin | |
224 | #feature args : Number of feature args to follow | |
225 | feature args : 'writethrough' (optional) | |
226 | #core args : Number of core arguments (must be even) | |
227 | core args : Key/value pairs for tuning the core | |
228 | e.g. migration_threshold | |
229 | #policy args : Number of policy arguments to follow (must be even) | |
230 | policy args : Key/value pairs | |
231 | e.g. 'sequential_threshold 1024 | |
232 | ||
233 | Messages | |
234 | -------- | |
235 | ||
236 | Policies will have different tunables, specific to each one, so we | |
237 | need a generic way of getting and setting these. Device-mapper | |
238 | messages are used. (A sysfs interface would also be possible.) | |
239 | ||
240 | The message format is: | |
241 | ||
242 | <key> <value> | |
243 | ||
244 | E.g. | |
245 | dmsetup message my_cache 0 sequential_threshold 1024 | |
246 | ||
65790ff9 JT |
247 | |
248 | Invalidation is removing an entry from the cache without writing it | |
249 | back. Cache blocks can be invalidated via the invalidate_cblocks | |
250 | message, which takes an arbitrary number of cblock ranges. | |
251 | ||
252 | invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* | |
253 | ||
254 | E.g. | |
255 | dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 | |
256 | ||
c6b4fcba JT |
257 | Examples |
258 | ======== | |
259 | ||
260 | The test suite can be found here: | |
261 | ||
65790ff9 | 262 | https://github.com/jthornber/device-mapper-test-suite |
c6b4fcba JT |
263 | |
264 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
265 | /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' | |
266 | dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ | |
267 | /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ | |
268 | mq 4 sequential_threshold 1024 random_threshold 8' |