-----Original Message-----
From: acrn-dev@... <acrn-dev@...> On
Behalf Of Conghui Chen
Sent: Wednesday, July 20, 2022 3:53 PM
To: acrn-dev@...
Cc: Chen, Conghui <conghui.chen@...>
Subject: [acrn-dev] [PATCH 0/2] hv: support fastio

The original IO request flow is as below:
+-----------------------------+-------------------------------+----------------------------------+
| Service VM vCPU 0 | Service VM vCPU x | User VM vCPU y
|
+-----------------------------+-------------------------------+---------
+-----------------------------+-------------------------------+---------
+-----------------------------+-------------------------------+---------
+-----------------------------+-------------------------------+--------
| | |Trap with MMIO/PIO
access |
| | | |
| | |- Fill an I/O request
|
| | |- set state to PENDING
|
| | |- Fire upcall to Service
VM vCPU 0|
| | |- Wait for COMPLETE |
+-----------------------------+-------------------------------+----------------------------------+
| | | |
| HSM: | | |
| | | |
| - Scan for pending requests | | (1) |
| - Set State to PROCESSING | | |
| - Assign requests to clients| | |
+-----------------------------+-------------------------------+----------------------------------+
| | | |
| | Client(in kernel/userspace): | |
| | - Scan for assigned requests | (2)
|
| | - Handle to requests | |
| | - Set state to COMPLETE | |
| | - Notify the hypervisor | |
+-----------------------------+-------------------------------+----------------------------------+
| | Hypervisor: | (3) |
| | - resume User VM vCPU y |
|
+-----------------------------+-------------------------------+----------------------------------+
| | | Hypervisor: |
| | | - set state to FREE in vCPU y |
| | | - VMEnter
|
+-----------------------------+-------------------------------+----------------------------------+

From the above I/O request process flow, the User VM vCPU y has to wait for
(1)(2)(3) until the I/O request is set to COMPLETE.
For most of READ and WRITE operations, it is meaningful. But for the special
'NOTIFY' access, which do not carry any valid data, has no request on the
return value. So, we can return the User vCPU earlier.

These kinds of IO are given a new name, fastio.


The new process flow like this:
|-----------------------------+---------------------------------+-----------------------------------+
| Service VM vCPU 0 | Service VM vCPU x |User VM vCPU y
|
+-----------------------------+---------------------------------+-----------------------------------+
| | |- Trap with
MMIO/PIO access |
| | |- Fill an I/O request
|
| | |- set state to
PENDING |
| | |- Fire upcall to
Service VM vCPU 0 |
| | |- VMEnter
|
+-----------------------------+---------------------------------+-----------------------------------+
| | | |
| HSM: | | |
| - Scan for pending requests | | |
| - Set state to PROCESSING | | |
| - Assign request to clients | | |
+-----------------------------+---------------------------------+-----------------------------------+
| | | |
| | Client(kernel): | |
| | - Scan for ioeventfd with same | |
| | IO base | |
| | - signal event | |
| | - Set stat to COMPLETE | |
+-----------------------------+---------------------------------+-----------------------------------+

Notes:
1. With ioeventfd, the IOReq can complete in kernel instead of userspace, this
will reduce IO latency.
2. As we only has one I/O request slot for each User VM's vCPU, so when the
second IO comes, it needs
to wait for the completion of the previous one, otherwise, the IOReq
structure would be overwritten.
3. Besides, 'completion_polling' flag is set for the fastio. With this flag, kernel
would not invoke
'HC_NOTIFY_REQUEST_FINISH' hypercall, so the next IO should wait the pre
IO's completion in polling
mode.

So the process becomes more complicated

|-----------------------------+---------------------------------+-----------------------------------+
| Service VM vCPU 0 | Service VM vCPU x |User VM vCPU y
|
+-----------------------------+---------------------------------+-----------------------------------+
| | |- Trap with
MMIO/PIO access |
| | |- Fill an I/O request
|
| | |- set state to
PENDING |
| | |- Fire upcall to
Service VM vCPU 0 |
| | |- VMEnter
|
+-----------------------------+---------------------------------+-----------------------------------+
| | | |
| HSM: | | |
| - Scan for pending requests | | |
| - Set state to PROCESSING | | |
| - Assign request to clients | | |
+-----------------------------+---------------------------------+-----------------------------------+
| | |- Trap with NEW MMIO/PIO access |
| | Client(kernel): |- Waiting for prev I/O complete
|
| | - Scan for ioeventfd with same | |
| | IO base | |
| | - signal event | |
| | - Set stat to COMPLETE | |
+-----------------------------+---------------------------------+-----------------------------------+
| | |- update prev I/O status to
FREE |
| | |- Fill the new I/O request
|
| | |- set state to PENDING
|
| | |- Fire upcall to Service VM vCPU 0 |
| | |- VM enter |
+-----------------------------+---------------------------------+-----------------------------------+
| HSM: | | |
| ... | | |
+-----------------------------+---------------------------------+-----------------------------------|

TODO:

To be faster, better to have an mechanism to record the fastio access, and
release the I/O request slot as soon as possible.


Conghui (2):
hv: add hypercall for fastio
hv: support fastio

hypervisor/arch/x86/guest/vmcall.c | 4 +
hypervisor/common/hypercall.c | 17 +++
hypervisor/dm/io_req.c | 130 ++++++++++++++++++---
hypervisor/include/arch/x86/asm/guest/vm.h | 4 +
hypervisor/include/common/hypercall.h | 29 +++++
hypervisor/include/dm/io_req.h | 11 ++
hypervisor/include/public/acrn_hv_defs.h | 2 +
7 files changed, 182 insertions(+), 15 deletions(-)

--
2.25.1