Commit | Line | Data |
---|---|---|
8d497515 CY |
1 | An Intel MIC X100 device is a PCIe form factor add-in coprocessor |
2 | card based on the Intel Many Integrated Core (MIC) architecture | |
3 | that runs a Linux OS. It is a PCIe endpoint in a platform and therefore | |
4 | implements the three required standard address spaces i.e. configuration, | |
5 | memory and I/O. The host OS loads a device driver as is typical for | |
6 | PCIe devices. The card itself runs a bootstrap after reset that | |
af190494 DC |
7 | transfers control to the card OS downloaded from the host driver. The |
8 | host driver supports OSPM suspend and resume operations. It shuts down | |
9 | the card during suspend and reboots the card OS during resume. | |
8d497515 CY |
10 | The card OS as shipped by Intel is a Linux kernel with modifications |
11 | for the X100 devices. | |
12 | ||
13 | Since it is a PCIe card, it does not have the ability to host hardware | |
14 | devices for networking, storage and console. We provide these devices | |
15 | on X100 coprocessors thus enabling a self-bootable equivalent environment | |
16 | for applications. A key benefit of our solution is that it leverages | |
17 | the standard virtio framework for network, disk and console devices, | |
18 | though in our case the virtio framework is used across a PCIe bus. | |
19 | ||
a8035843 SY |
20 | MIC PCIe card has a dma controller with 8 channels. These channels are |
21 | shared between the host s/w and the card s/w. 0 to 3 are used by host | |
22 | and 4 to 7 by card. As the dma device doesn't show up as PCIe device, | |
23 | a virtual bus called mic bus is created and virtual dma devices are | |
24 | created on it by the host/card drivers. On host the channels are private | |
25 | and used only by the host driver to transfer data for the virtio devices. | |
26 | ||
7df20f2d SD |
27 | The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a |
28 | low level communications API across PCIe currently implemented for MIC. | |
29 | More details are available at scif_overview.txt. | |
30 | ||
f4a66c20 AD |
31 | The Coprocessor State Management (COSM) driver on the host allows for |
32 | boot, shutdown and reset of Intel MIC devices. It communicates with a COSM | |
33 | "client" driver on the MIC cards over SCIF to perform these functions. | |
34 | ||
8d497515 CY |
35 | Here is a block diagram of the various components described above. The |
36 | virtio backends are situated on the host rather than the card given better | |
37 | single threaded performance for the host compared to MIC, the ability of | |
38 | the host to initiate DMA's to/from the card using the MIC DMA engine and | |
39 | the fact that the virtio block storage backend can only be on the host. | |
40 | ||
a8035843 SY |
41 | | |
42 | +----------+ | +----------+ | |
43 | | Card OS | | | Host OS | | |
44 | +----------+ | +----------+ | |
45 | | | |
46 | +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ | |
47 | | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio | | |
48 | | Net | |Console | |Block | | |Net | |Console | |Block | | |
49 | | Driver| |Driver | |Driver| | |backend | |backend | |backend | | |
50 | +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ | |
51 | | | | | | | | | |
52 | | | | |User | | | | |
53 | | | | |------|------------|---------|------- | |
54 | +-------------------+ |Kernel +--------------------------+ | |
55 | | | | Virtio over PCIe IOCTLs | | |
56 | | | +--------------------------+ | |
57 | +-----------+ | | | +-----------+ | |
f4a66c20 AD |
58 | | MIC DMA | | +------+ | +------+ +------+ | | MIC DMA | |
59 | | Driver | | | SCIF | | | SCIF | | COSM | | | Driver | | |
60 | +-----------+ | +------+ | +------+ +--+---+ | +-----------+ | |
61 | | | | | | | | | | |
62 | +---------------+ | +------+ | +--+---+ +--+---+ | +----------------+ | |
63 | |MIC virtual Bus| | |SCIF | | |SCIF | | COSM | | |MIC virtual Bus | | |
64 | +---------------+ | |HW Bus| | |HW Bus| | Bus | | +----------------+ | |
65 | | | +------+ | +--+---+ +------+ | | | |
66 | | | | | | | | | | |
67 | | +-----------+---+ | | | +---------------+ | | |
68 | | |Intel MIC | | | | |Intel MIC | | | |
69 | +---|Card Driver | | | | |Host Driver | | | |
70 | +------------+--------+ | +----+---------------+-----+ | |
71 | | | | | |
a8035843 SY |
72 | +-------------------------------------------------------------+ |
73 | | | | |
74 | | PCIe Bus | | |
75 | +-------------------------------------------------------------+ |