|
Chen, Zide
Hi Dubravko,
You are very welcome! Additional to Shuo's suggestion, here are some advices:
1) X2APIC. I added this log but didn't see it from your log. That's why I claimed that the guest didn't switch to x2apic.
pr_err("vm%d vcpu%d switched to X2APIC new %x", vlapic->vm->vm_id, vlapic->vcpu->vcpu_id, new);
BTW, Hypervisor passthru the CPUID X2APIC capability (CPUID.01.ECX[21]) to the guest and it's up to the guest to enable X2APIC.
2) Linux logs
Hypervisor provides a virtual UART port to Linux and expects Linux to detect it. You may double check the BASE/IRQ macro in your hypervisor code, and try to explicitly tell Linux with this info thru the bootargs?
scenarios/sdc/vm_configurations.c
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] = {
{
.load_order = SOS_VM,
.name = "ACRN SOS VM",
.vuart[0] = {
.type = VUART_LEGACY_PIO,
.addr.port_base = SOS_COM1_BASE, // 0x3F8
.irq = SOS_COM1_IRQ, // 4
},
}
}
3) VCPU BLOCKED state.
This may be OK. HV could block the VCPU when it'handling guest's IO requests. Or it's never been woken up by SIPI. As I mentioned previously, I didn't see such SIPI sequences on other boards before and can't say it actually works or not. I'd suggest to use "MAXCPUS=1" to force the guest boot with one single CPU only.
4) HV console
So far so good. You may try other commands which may further indicate the guest's status.
ACRN:\>int
IRQ VECTOR CPU0 CPU1 CPU2 CPU3
0 0x20 28 0 0 0
1 0x21 0 0 0 0
2 0x22 0 0 0 0
3 0x23 0 0 0 0
4 0x24 0 0 0 0
5 0x25 0 0 0 0
6 0x26 0 0 0 0
7 0x27 0 0 0 0
8 0x28 0 0 0 1
9 0x29 0 5718 0 0
10 0x2A 0 0 0 0
11 0x2B 0 0 0 0
12 0x2C 0 0 0 0
13 0x2D 0 0 0 0
14 0x2E 0 0 0 0
15 0x2F 0 0 0 0
16 0x34 0 0 0 0
17 0x35 0 0 0 0
18 0x31 0 0 0 0
120 0x30 0 0 0 0
121 0x32 0 0 539127 0
122 0x33 0 0 0 46
123 0x36 0 0 21117 0
124 0x37 0 1061 0 0
125 0x38 0 0 34354 0
252 0xF4 0 0 0 0
253 0xF2 0 0 0 0
254 0xF0 99624 55478 75017 72695
255 0xEF 4761444 2072151 2408597 2532948
ACRN:\>pt
VM TYPE IRQ VEC DEST TM PIN VPIN BDF VBDF
0 IOAPIC 0 0x20 0x0 edge 2 2 0:0.0 0:0.0
0 IOAPIC 9 0x29 0x0 level 9 9 0:0.0 0:0.0
0 IOAPIC 18 0x31 0x0 level 18 18 0:0.0 0:0.0
0 IOAPIC 15 0x2F 0x0 edge 15 15 0:0.0 0:0.0
0 IOAPIC 14 0x2E 0x0 level 14 14 0:0.0 0:0.0
0 IOAPIC 13 0x2D 0x0 edge 13 13 0:0.0 0:0.0
0 IOAPIC 12 0x2C 0x0 edge 12 12 0:0.0 0:0.0
0 IOAPIC 11 0x2B 0x0 edge 11 11 0:0.0 0:0.0
0 IOAPIC 10 0x2A 0x0 edge 10 10 0:0.0 0:0.0
0 IOAPIC 8 0x28 0x0 edge 8 8 0:0.0 0:0.0
0 IOAPIC 7 0x27 0x0 edge 7 7 0:0.0 0:0.0
0 IOAPIC 6 0x26 0x0 edge 6 6 0:0.0 0:0.0
0 IOAPIC 5 0x25 0x0 edge 5 5 0:0.0 0:0.0
0 IOAPIC 1 0x21 0x0 edge 1 1 0:0.0 0:0.0
0 MSI 121 0x32 0x2 edge 255 255 0:2.0 0:2.0
0 MSI 122 0x33 0x0 edge 255 255 0:16.0 0:16.0
0 IOAPIC 16 0x34 0x0 level 16 16 0:0.0 0:0.0
0 IOAPIC 17 0x35 0x0 level 17 17 0:0.0 0:0.0
0 MSI 123 0x36 0x0 edge 255 255 0:17.0 0:17.0
0 MSI 124 0x37 0x0 edge 255 255 0:14.0 0:14.0
0 MSI 125 0x38 0x0 edge 255 255 0:1f.6 0:1f.6
Best Regards,
Zide
toggle quoted messageShow quoted text
On 2/14/20 7:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote: Hi Zide,
1) X2APIC
I see that it's not mentioned anywhere in ACRN log (and I suppose it should be), but it seems to be enabled somehow in non-hypervised Linux:
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $ sudo dmesg | grep -i x2apic
[ 0.165205] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.167422] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.167426] x2apic enabled
[ 0.167448] Switched APIC routing to cluster x2apic.
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $
Our BIOS does not have a GUI setting to manually enable it, and we couldn't find where to explicitly enable it in source code. It's mentioned at many places; though not in DMAR table.
Can we conclude from the log above that it's ok, or we still need to find a way to enable it in ACRN or BIOS?
2) Linux logs
I'm working on this and will report later.
By the way, I think my last logs might be inconsistent with earlier ones (with endless SIPI messages) because earlier I might have in a hurry just pressed enter in the boot menu to choose whichever option was selected, instead of selecting ACRN. Now I'm more careful.
Removing 'quiet' parameter didn't help.
3) Naked HV
... is working. With your previous acrn.efi and bzImage from around Wednesday noon Central European Time, after serial boot log stops, I can indeed switch to ACRN console with ctrl+space. If I switch back to SOS, nothing happens. I can only return back to ACRN console. Here are some printouts:
[13133696us][cpu=0][vm0:vcpu0][sev=3][seq=322]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13146659us][cpu=0][vm0:vcpu0][sev=3][seq=323]:vcpu1 paused, new state: 4
[13154097us][cpu=0][vm0:vcpu0][sev=3][seq=324]:vcpu1 reset
[13160078us][cpu=0][vm0:vcpu0][sev=3][seq=325]:vlapic: vlapic_reset 1693 vm0 vcpu1 initial lapic_id: 2000000
[13170960us][cpu=0][vm0:vcpu0][sev=3][seq=326]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13183985us][cpu=0][vm0:vcpu0][sev=3][seq=327]:vcpu2 paused, new state: 4
[13191429us][cpu=0][vm0:vcpu0][sev=3][seq=328]:vcpu2 reset
[13197408us][cpu=0][vm0:vcpu0][sev=3][seq=329]:vlapic: vlapic_reset 1693 vm0 vcpu2 initial lapic_id: 4000000
[13208295us][cpu=0][vm0:vcpu0][sev=3][seq=330]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13221324us][cpu=0][vm0:vcpu0][sev=3][seq=331]:vcpu3 paused, new state: 4
[13228769us][cpu=0][vm0:vcpu0][sev=3][seq=332]:vcpu3 reset
[13234748us][cpu=0][vm0:vcpu0][sev=3][seq=333]:vlapic: vlapic_reset 1693 vm0 vcpu3 initial lapic_id: 6000000
[13255651us][cpu=0][vm0:vcpu0][sev=3][seq=334]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13268597us][cpu=0][vm0:vcpu0][sev=3][seq=335]:vlapic: Start Secondary VCPU1 for VM[0]...
[13277605us][cpu=0][vm0:vcpu0][sev=3][seq=336]:vcpu1 scheduled on pcpu1 vcpu->state 1
[13286228us][cpu=0][vm0:vcpu0][sev=3][seq=337]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13299256us][cpu=0][vm0:vcpu0][sev=3][seq=338]:vlapic: Start Secondary VCPU2 for VM[0]...
[13308274us][cpu=0][vm0:vcpu0][sev=3][seq=339]:vcpu2 scheduled on pcpu2 vcpu->state 1
[13316902us][cpu=0][vm0:vcpu0][sev=3][seq=340]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13329919us][cpu=0][vm0:vcpu0][sev=3][seq=341]:vlapic: Start Secondary VCPU3 for VM[0]...
[13338926us][cpu=0][vm0:vcpu0][sev=3][seq=342]:vcpu3 scheduled on pcpu3 vcpu->state 1
[13347768us][cpu=0][vm0:vcpu0][sev=3][seq=343]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
[13360707us][cpu=0][vm0:vcpu0][sev=3][seq=344]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
[13373751us][cpu=0][vm0:vcpu0][sev=3][seq=345]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
/* ctrl+space */
---Entering ACRN SHELL---
ACRN:\>
ACRN:\>version
HV version 1.6-unstable-2020-02-11 15:28:54-4990d8574159 DBG (daily tag: acrn-2020w03.4-140000p) build by zide
API version 1.0
ACRN:\>vm_list
VM_UUID VM_ID VM_NAME VM_STATE
================================ ===== ================================ ========
dbbbd4347a574216a12c2201f1ab0240 0 ACRN SOS VM Started
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
===== ======= ======= ========= ========== ==========
0 0 0 PRIMARY Running RUNNING
0 1 1 SECONDARY Running BLOCKED
0 2 2 SECONDARY Running BLOCKED
0 3 3 SECONDARY Running BLOCKED
ACRN:\>vioapic
Error: Invalid parameters.
ACRN:\>vioapic 0
PIN VEC DM DEST TM DELM IRR MASK
0 0x0 phys 0x0 edge 0 0 1
1 0x0 phys 0x0 edge 0 0 1
2 0x0 phys 0x0 edge 0 0 1
3 0x0 phys 0x0 edge 0 0 1
4 0x0 phys 0x0 edge 0 0 1
5 0x0 phys 0x0 edge 0 0 1
6 0x0 phys 0x0 edge 0 0 1
7 0x0 phys 0x0 edge 0 0 1
8 0x0 phys 0x0 edge 0 0 1
9 0x0 phys 0x0 edge 0 0 1
10 0x0 phys 0x0 edge 0 0 1
11 0x0 phys 0x0 edge 0 0 1
12 0x0 phys 0x0 edge 0 0 1
13 0x0 phys 0x0 edge 0 0 1
14 0x0 phys 0x0 edge 0 0 1
15 0x0 phys 0x0 edge 0 0 1
16 0x0 phys 0x0 edge 0 0 1
17 0x0 phys 0x0 edge 0 0 1
18 0x0 phys 0x0 edge 0 0 1
19 0x0 phys 0x0 edge 0 0 1
20 0x0 phys 0x0 edge 0 0 1
21 0x0 phys 0x0 edge 0 0 1
22 0x0 phys 0x0 edge 0 0 1
23 0x0 phys 0x0 edge 0 0 1
24 0x0 phys 0x0 edge 0 0 1
25 0x0 phys 0x0 edge 0 0 1
26 0x0 phys 0x0 edge 0 0 1
27 0x0 phys 0x0 edge 0 0 1
28 0x0 phys 0x0 edge 0 0 1
29 0x0 phys 0x0 edge 0 0 1
30 0x0 phys 0x0 edge 0 0 1
31 0x0 phys 0x0 edge 0 0 1
32 0x0 phys 0x0 edge 0 0 1
33 0x0 phys 0x0 edge 0 0 1
34 0x0 phys 0x0 edge 0 0 1
35 0x0 phys 0x0 edge 0 0 1
36 0x0 phys 0x0 edge 0 0 1
37 0x0 phys 0x0 edge 0 0 1
38 0x0 phys 0x0 edge 0 0 1
39 0x0 phys 0x0 edge 0 0 1
40 0x0 phys 0x0 edge 0 0 1
41 0x0 phys 0x0 edge 0 0 1
42 0x0 phys 0x0 edge 0 0 1
43 0x0 phys 0x0 edge 0 0 1
44 0x0 phys 0x0 edge 0 0 1
45 0x0 phys 0x0 edge 0 0 1
46 0x0 phys 0x0 edge 0 0 1
47 0x0 phys 0x0 edge 0 0 1
48 0x0 phys 0x0 edge 0 0 1
49 0x0 phys 0x0 edge 0 0 1
50 0x0 phys 0x0 edge 0 0 1
51 0x0 phys 0x0 edge 0 0 1
52 0x0 phys 0x0 edge 0 0 1
53 0x0 phys 0x0 edge 0 0 1
54 0x0 phys 0x0 edge 0 0 1
55 0x0 phys 0x0 edge 0 0 1
56 0x0 phys 0x0 edge 0 0 1
57 0x0 phys 0x0 edge 0 0 1
58 0x0 phys 0x0 edge 0 0 1
59 0x0 phys 0x0 edge 0 0 1
60 0x0 phys 0x0 edge 0 0 1
61 0x0 phys 0x0 edge 0 0 1
62 0x0 phys 0x0 edge 0 0 1
63 0x0 phys 0x0 edge 0 0 1
64 0x0 phys 0x0 edge 0 0 1
65 0x0 phys 0x0 edge 0 0 1
66 0x0 phys 0x0 edge 0 0 1
67 0x0 phys 0x0 edge 0 0 1
68 0x0 phys 0x0 edge 0 0 1
69 0x0 phys 0x0 edge 0 0 1
70 0x0 phys 0x0 edge 0 0 1
71 0x0 phys 0x0 edge 0 0 1
72 0x0 phys 0x0 edge 0 0 1
73 0x0 phys 0x0 edge 0 0 1
74 0x0 phys 0x0 edge 0 0 1
75 0x0 phys 0x0 edge 0 0 1
76 0x0 phys 0x0 edge 0 0 1
77 0x0 phys 0x0 edge 0 0 1
78 0x0 phys 0x0 edge 0 0 1
79 0x0 phys 0x0 edge 0 0 1
80 0x0 phys 0x0 edge 0 0 1
81 0x0 phys 0x0 edge 0 0 1
82 0x0 phys 0x0 edge 0 0 1
83 0x0 phys 0x0 edge 0 0 1
84 0x0 phys 0x0 edge 0 0 1
85 0x0 phys 0x0 edge 0 0 1
86 0x0 phys 0x0 edge 0 0 1
87 0x0 phys 0x0 edge 0 0 1
88 0x0 phys 0x0 edge 0 0 1
89 0x0 phys 0x0 edge 0 0 1
90 0x0 phys 0x0 edge 0 0 1
91 0x0 phys 0x0 edge 0 0 1
92 0x0 phys 0x0 edge 0 0 1
93 0x0 phys 0x0 edge 0 0 1
94 0x0 phys 0x0 edge 0 0 1
95 0x0 phys 0x0 edge 0 0 1
96 0x0 phys 0x0 edge 0 0 1
97 0x0 phys 0x0 edge 0 0 1
98 0x0 phys 0x0 edge 0 0 1
99 0x0 phys 0x0 edge 0 0 1
100 0x0 phys 0x0 edge 0 0 1
101 0x0 phys 0x0 edge 0 0 1
102 0x0 phys 0x0 edge 0 0 1
103 0x0 phys 0x0 edge 0 0 1
104 0x0 phys 0x0 edge 0 0 1
105 0x0 phys 0x0 edge 0 0 1
106 0x0 phys 0x0 edge 0 0 1
107 0x0 phys 0x0 edge 0 0 1
108 0x0 phys 0x0 edge 0 0 1
109 0x0 phys 0x0 edge 0 0 1
110 0x0 phys 0x0 edge 0 0 1
111 0x0 phys 0x0 edge 0 0 1
112 0x0 phys 0x0 edge 0 0 1
113 0x0 phys 0x0 edge 0 0 1
114 0x0 phys 0x0 edge 0 0 1
115 0x0 phys 0x0 edge 0 0 1
116 0x0 phys 0x0 edge 0 0 1
117 0x0 phys 0x0 edge 0 0 1
118 0x0 phys 0x0 edge 0 0 1
119 0x0 phys 0x0 edge 0 0 1
ACRN:\>
ACRN:\>
ACRN:\>
ACRN:\>
ACRN:\>vm_console 0
----- Entering VM 0 Shell -----
/* no reaction to any regular keys */
/* ctrl+space */
---Entering ACRN SHELL---
ACRN:\>
Thank you again, I would never get this far without help.
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
*Sent:* Wednesday, February 12, 2020 9:04 PM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
1) Enable X2APIC from guest Linux
Actually ACRN checks X2APIC capability and it goes panic w/o it. So the host CPUID must have X2APIC support.
We met one case where the BIOS DMAR setting has some thing like "do not use x2apic", which prevents Linux from enabling
X2APIC. Or you may check why the guest doesn't switch to X2APIC from native Linux dmesg?
2) Need Linux logs to help debugging.
Attached is the ACRN built with the following hack, which directs guest serial prints directly to HV console.
Seems Linux goes to the path of numachip_wakeup_secondary() to send SIPIs to secondary CPUs, which we didn't see before.
It's good to see Linux early pintk t help address the issue.
diff --git a/hypervisor/dm/vuart.c b/hypervisor/dm/vuart.c
index 8771a7d8a15f..be5f0da943b8 100644
--- a/hypervisor/dm/vuart.c
+++ b/hypervisor/dm/vuart.c
@@ -295,6 +295,7 @@ static void write_reg(struct acrn_vuart *vu, uint16_t reg, uint8_t value_u8)
vu->lsr |= LSR_OE;
} else {
fifo_putchar(&vu->txfifo, (char)value_u8);
+ printf("%c", value_u8);
}
vu->thre_int_pending = true;
break;
3) "naked" HV is working?
In the meantime, if you want, you can hack ACRN with the following and it won't launch any VMs and the HV console is
supposed to be working. It's just a way to check that ACRN is "generally" working on the HW and you may play with some HV
console commands.
diff --git a/hypervisor/arch/x86/guest/vm.c b/hypervisor/arch/x86/guest/vm.c
index 0471324be065..53ccb6b9acae 100644
--- a/hypervisor/arch/x86/guest/vm.c
+++ b/hypervisor/arch/x86/guest/vm.c
@@ -873,6 +873,8 @@ void launch_vms(uint16_t pcpu_id)
uint16_t vm_id, bsp_id;
struct acrn_vm_config *vm_config;
+ return;
+
for (vm_id = 0U; vm_id < CONFIG_MAX_VM_NUM; vm_id++) {
vm_config = get_vm_config(vm_id);
if ((vm_config->load_order == SOS_VM) || (vm_config->load_order == PRE_LAUNCHED_VM)) {
Best Regards,
Zide
On 2/12/20 11:05 AM, Zide Chen wrote:
Hi Dubravko,
With my acrn.efi, it switches to guest console by default. Does it switch back to ACRN console by pressing "Ctrl + Space"?
The logs show that X2APIC is not enabled by Linux. Is it possible to enable X2APIC from BIOS?
Without X2APIC, ACRN doesn't work. Though I'm expecting ACRN should go beyond what it is now.
BTW, can you remove "quiet" from the Linux bootargs? This may not really help, but I'm puzzled why it doesn't print any
Linux logs.
From the logs, one big difference with my images (acrn.efi + bzImage) is that now it doesn't repeat the SIPI sequence
(INIT -> Start up -> INIT -> Start up) endless, instead, after launching 3 VCPUs, the guest sends broadcasting SIPI INIT
and Start Up only. This is a good news, but not sure that's because of the new acrn.efi, or the bzImage?
Best Regards,
Zide
On 2/12/20 10:36 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi Zide,
I've checked all the menus and settings in our BIOS, unfortunately I couldn't find anything that looks like the
mentioned setting or similar or related to it.
We are using coreboot, built according to instructions from https://software.intel.com/en-us/firmware/sites/default/files/intelatome3900-0.71-buildinstructions.txt, plus we added
support for our board (the link apparently no longer works). In general it works very well.
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
Lists.Projectacrn.Org <zide.chen=intel.com@...>
*Sent:* Wednesday, February 12, 2020 6:47 PM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
In the meantime, is there something like "AP Thread Idle Manner" in your BIOS? And changing its setting makes any
difference?
Best Regards,
Zide
On 2/12/20 8:02 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hello,
Please find attached the log file and my acrn.conf.
My old BIOS didn't have any settings related to Mwait, so I've installed newer BIOS which has "Monitor Mwait Enable" and
I've ensured it's set to "Disabled". Changing the BIOS seems to have also changed some data in logs, but the overall
behavior is the same.
New acrn.efi seems to work reasonably well. Our custom board is obviously specific, but in general it is similar to
reference Leaf Hill CRB.
After the last printout from the log, everything stops working (there is nothing on the screen attached to the board and
the keyboard doesn't do anything; also typing in the serial terminal doesn't do anything).
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
Lists.Projectacrn.Org <zide.chen=intel.com@...>
*Sent:* Wednesday, February 12, 2020 1:04 AM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
"Spurious vector" could because of external interrupts not registered in hypervisor. This may not be an issue and can be
ignored for now.
For the repeating "Starting VCPU 3" issue, it seems some thing related to SIPI delivery. But nothing is obvious to me and
need to debug it.
I built a Clear Linux kernel with additional logs showing how it's bringing up APs. Could you please try the attached
bzImage?
Also please enable early printk by adding "ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep" to the options in
acrn.conf; And use the attached acrn.efi (I enabled some additional logs and print out Linux logs even before ACRN console
is operational). If acrn.efi doesn't boot, which is highly possibly since it's not built against your custom board, you
may need to apply the attached patch to ACRN hypervisor.
Hopefully the full logs can give us some clues.
Best Regards,
Zide
zide.chen@...
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Shuo,
Thank you, you were right, with your fix ACRN loads the Service OS on my board too.
Now I have the issue that Ethernet doesn't work (while it does in regular Clear Linux), so I can't download and configure the User OS. I'll try downloading, configuring and recompiling https://github.com/clearlinux-pkgs/linux-iot-lts2018
kernel. Is that the right one and do I need to activate some particular settings so it works well with ACRN?
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Shuo A Liu via Lists.Projectacrn.Org <shuo.a.liu=intel.com@...>
Sent: Friday, February 14, 2020 5:48 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I glanced through the discussion and found it might be a similar issue that I hit today with UP2 board.
My problem is that with ACRN enabled the linux kernel cannot bootup APs, without ACRN hypervisor the native kernel works well.
I have a fix that posted on acrn-dev maillist several hours ago. Perhaps you can have a quick try and hope it can help.
I paste it here.
---
On UEFI UP2 board, BIOS startup APs and then they all go into HLT status.
Then Guest OS take over and re-init the APs again. The flows from HV perspective is like:
wait_event(VCPU_EVENT_VIRTUAL_INTERRUPT) -> sleep_thread
-> pause_vcpu(ZOMBIE) -> sleep_thread
-> reset_vcpu
-> launch_vcpu -> wake_vcpu
However, the last wake_vcpu will fail because the cpu event VCPU_EVENT_VIRTUAL_INTERRUPT had not got signaled.
Add signal_event before sleep_thread in pause_vcpu to make sure the vcpu thread is not waiting for event.
Signed-off-by: Shuo A Liu <shuo.a.liu@...>
---
hypervisor/arch/x86/guest/vcpu.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hypervisor/arch/x86/guest/vcpu.c b/hypervisor/arch/x86/guest/vcpu.c
index 6161889..bd2d2d2 100644
--- a/hypervisor/arch/x86/guest/vcpu.c
+++ b/hypervisor/arch/x86/guest/vcpu.c
@@ -701,6 +701,8 @@ void pause_vcpu(struct acrn_vcpu *vcpu, enum vcpu_state new_state)
vcpu->state = new_state;
if (vcpu->prev_state == VCPU_RUNNING) {
+ /* signal_event will wakeup the vcpu's thread if it's in HLT sleep, else do nothing. */
+ signal_event(&vcpu->events[VCPU_EVENT_VIRTUAL_INTERRUPT]);
sleep_thread(&vcpu->thread_obj);
}
if (pcpu_id != get_pcpu_id()) {
--
2.8.3
Thanks
shuo
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Friday, February 14, 2020 23:41
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
I see that it's not mentioned anywhere in ACRN log (and I suppose it should be), but it seems to be enabled somehow in non-hypervised Linux:
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $ sudo dmesg | grep -i x2apic
[ 0.165205] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.167422] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.167426] x2apic enabled
[ 0.167448] Switched APIC routing to cluster x2apic.
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $
Our BIOS does not have a GUI setting to manually enable it, and we couldn't find where to explicitly enable it in source code. It's mentioned at many places; though not in DMAR table.
Can we conclude from the log above that it's ok, or we still need to find a way to enable it in ACRN or BIOS?
I'm working on this and will report later.
By the way, I think my last logs might be inconsistent with earlier ones (with endless SIPI messages) because earlier I might have in a hurry just pressed enter in the boot menu to choose whichever option was selected,
instead of selecting ACRN. Now I'm more careful.
Removing 'quiet' parameter didn't help.
... is working. With your previous acrn.efi and bzImage from around Wednesday noon Central European Time, after serial boot log stops, I can indeed switch to ACRN console with ctrl+space. If I switch back to SOS,
nothing happens. I can only return back to ACRN console. Here are some printouts:
[13133696us][cpu=0][vm0:vcpu0][sev=3][seq=322]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13146659us][cpu=0][vm0:vcpu0][sev=3][seq=323]:vcpu1 paused, new state: 4
[13154097us][cpu=0][vm0:vcpu0][sev=3][seq=324]:vcpu1 reset
[13160078us][cpu=0][vm0:vcpu0][sev=3][seq=325]:vlapic: vlapic_reset 1693 vm0 vcpu1 initial lapic_id: 2000000
[13170960us][cpu=0][vm0:vcpu0][sev=3][seq=326]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13183985us][cpu=0][vm0:vcpu0][sev=3][seq=327]:vcpu2 paused, new state: 4
[13191429us][cpu=0][vm0:vcpu0][sev=3][seq=328]:vcpu2 reset
[13197408us][cpu=0][vm0:vcpu0][sev=3][seq=329]:vlapic: vlapic_reset 1693 vm0 vcpu2 initial lapic_id: 4000000
[13208295us][cpu=0][vm0:vcpu0][sev=3][seq=330]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13221324us][cpu=0][vm0:vcpu0][sev=3][seq=331]:vcpu3 paused, new state: 4
[13228769us][cpu=0][vm0:vcpu0][sev=3][seq=332]:vcpu3 reset
[13234748us][cpu=0][vm0:vcpu0][sev=3][seq=333]:vlapic: vlapic_reset 1693 vm0 vcpu3 initial lapic_id: 6000000
[13255651us][cpu=0][vm0:vcpu0][sev=3][seq=334]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13268597us][cpu=0][vm0:vcpu0][sev=3][seq=335]:vlapic: Start Secondary VCPU1 for VM[0]...
[13277605us][cpu=0][vm0:vcpu0][sev=3][seq=336]:vcpu1 scheduled on pcpu1 vcpu->state 1
[13286228us][cpu=0][vm0:vcpu0][sev=3][seq=337]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13299256us][cpu=0][vm0:vcpu0][sev=3][seq=338]:vlapic: Start Secondary VCPU2 for VM[0]...
[13308274us][cpu=0][vm0:vcpu0][sev=3][seq=339]:vcpu2 scheduled on pcpu2 vcpu->state 1
[13316902us][cpu=0][vm0:vcpu0][sev=3][seq=340]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13329919us][cpu=0][vm0:vcpu0][sev=3][seq=341]:vlapic: Start Secondary VCPU3 for VM[0]...
[13338926us][cpu=0][vm0:vcpu0][sev=3][seq=342]:vcpu3 scheduled on pcpu3 vcpu->state 1
[13347768us][cpu=0][vm0:vcpu0][sev=3][seq=343]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
[13360707us][cpu=0][vm0:vcpu0][sev=3][seq=344]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
[13373751us][cpu=0][vm0:vcpu0][sev=3][seq=345]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
---Entering ACRN SHELL---
ACRN:\>
ACRN:\>version
HV version 1.6-unstable-2020-02-11 15:28:54-4990d8574159 DBG (daily tag: acrn-2020w03.4-140000p) build by zide
VM_UUID VM_ID VM_NAME VM_STATE
================================ ===== ================================ ========
dbbbd4347a574216a12c2201f1ab0240 0 ACRN SOS VM Started
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
===== ======= ======= ========= ========== ==========
0 0 0 PRIMARY Running RUNNING
0 1 1 SECONDARY Running BLOCKED
0 2 2 SECONDARY Running BLOCKED
0 3 3 SECONDARY Running BLOCKED
Error: Invalid parameters.
PIN VEC DM DEST TM DELM IRR MASK
0 0x0 phys 0x0 edge 0 0 1
1 0x0 phys 0x0 edge 0 0 1
2 0x0 phys 0x0 edge 0 0 1
3 0x0 phys 0x0 edge 0 0 1
4 0x0 phys 0x0 edge 0 0 1
5 0x0 phys 0x0 edge 0 0 1
6 0x0 phys 0x0 edge 0 0 1
7 0x0 phys 0x0 edge 0 0 1
8 0x0 phys 0x0 edge 0 0 1
9 0x0 phys 0x0 edge 0 0 1
10 0x0 phys 0x0 edge 0 0 1
11 0x0 phys 0x0 edge 0 0 1
12 0x0 phys 0x0 edge 0 0 1
13 0x0 phys 0x0 edge 0 0 1
14 0x0 phys 0x0 edge 0 0 1
15 0x0 phys 0x0 edge 0 0 1
16 0x0 phys 0x0 edge 0 0 1
17 0x0 phys 0x0 edge 0 0 1
18 0x0 phys 0x0 edge 0 0 1
19 0x0 phys 0x0 edge 0 0 1
20 0x0 phys 0x0 edge 0 0 1
21 0x0 phys 0x0 edge 0 0 1
22 0x0 phys 0x0 edge 0 0 1
23 0x0 phys 0x0 edge 0 0 1
24 0x0 phys 0x0 edge 0 0 1
25 0x0 phys 0x0 edge 0 0 1
26 0x0 phys 0x0 edge 0 0 1
27 0x0 phys 0x0 edge 0 0 1
28 0x0 phys 0x0 edge 0 0 1
29 0x0 phys 0x0 edge 0 0 1
30 0x0 phys 0x0 edge 0 0 1
31 0x0 phys 0x0 edge 0 0 1
32 0x0 phys 0x0 edge 0 0 1
33 0x0 phys 0x0 edge 0 0 1
34 0x0 phys 0x0 edge 0 0 1
35 0x0 phys 0x0 edge 0 0 1
36 0x0 phys 0x0 edge 0 0 1
37 0x0 phys 0x0 edge 0 0 1
38 0x0 phys 0x0 edge 0 0 1
39 0x0 phys 0x0 edge 0 0 1
40 0x0 phys 0x0 edge 0 0 1
41 0x0 phys 0x0 edge 0 0 1
42 0x0 phys 0x0 edge 0 0 1
43 0x0 phys 0x0 edge 0 0 1
44 0x0 phys 0x0 edge 0 0 1
45 0x0 phys 0x0 edge 0 0 1
46 0x0 phys 0x0 edge 0 0 1
47 0x0 phys 0x0 edge 0 0 1
48 0x0 phys 0x0 edge 0 0 1
49 0x0 phys 0x0 edge 0 0 1
50 0x0 phys 0x0 edge 0 0 1
51 0x0 phys 0x0 edge 0 0 1
52 0x0 phys 0x0 edge 0 0 1
53 0x0 phys 0x0 edge 0 0 1
54 0x0 phys 0x0 edge 0 0 1
55 0x0 phys 0x0 edge 0 0 1
56 0x0 phys 0x0 edge 0 0 1
57 0x0 phys 0x0 edge 0 0 1
58 0x0 phys 0x0 edge 0 0 1
59 0x0 phys 0x0 edge 0 0 1
60 0x0 phys 0x0 edge 0 0 1
61 0x0 phys 0x0 edge 0 0 1
62 0x0 phys 0x0 edge 0 0 1
63 0x0 phys 0x0 edge 0 0 1
64 0x0 phys 0x0 edge 0 0 1
65 0x0 phys 0x0 edge 0 0 1
66 0x0 phys 0x0 edge 0 0 1
67 0x0 phys 0x0 edge 0 0 1
68 0x0 phys 0x0 edge 0 0 1
69 0x0 phys 0x0 edge 0 0 1
70 0x0 phys 0x0 edge 0 0 1
71 0x0 phys 0x0 edge 0 0 1
72 0x0 phys 0x0 edge 0 0 1
73 0x0 phys 0x0 edge 0 0 1
74 0x0 phys 0x0 edge 0 0 1
75 0x0 phys 0x0 edge 0 0 1
76 0x0 phys 0x0 edge 0 0 1
77 0x0 phys 0x0 edge 0 0 1
78 0x0 phys 0x0 edge 0 0 1
79 0x0 phys 0x0 edge 0 0 1
80 0x0 phys 0x0 edge 0 0 1
81 0x0 phys 0x0 edge 0 0 1
82 0x0 phys 0x0 edge 0 0 1
83 0x0 phys 0x0 edge 0 0 1
84 0x0 phys 0x0 edge 0 0 1
85 0x0 phys 0x0 edge 0 0 1
86 0x0 phys 0x0 edge 0 0 1
87 0x0 phys 0x0 edge 0 0 1
88 0x0 phys 0x0 edge 0 0 1
89 0x0 phys 0x0 edge 0 0 1
90 0x0 phys 0x0 edge 0 0 1
91 0x0 phys 0x0 edge 0 0 1
92 0x0 phys 0x0 edge 0 0 1
93 0x0 phys 0x0 edge 0 0 1
94 0x0 phys 0x0 edge 0 0 1
95 0x0 phys 0x0 edge 0 0 1
96 0x0 phys 0x0 edge 0 0 1
97 0x0 phys 0x0 edge 0 0 1
98 0x0 phys 0x0 edge 0 0 1
99 0x0 phys 0x0 edge 0 0 1
100 0x0 phys 0x0 edge 0 0 1
101 0x0 phys 0x0 edge 0 0 1
102 0x0 phys 0x0 edge 0 0 1
103 0x0 phys 0x0 edge 0 0 1
104 0x0 phys 0x0 edge 0 0 1
105 0x0 phys 0x0 edge 0 0 1
106 0x0 phys 0x0 edge 0 0 1
107 0x0 phys 0x0 edge 0 0 1
108 0x0 phys 0x0 edge 0 0 1
109 0x0 phys 0x0 edge 0 0 1
110 0x0 phys 0x0 edge 0 0 1
111 0x0 phys 0x0 edge 0 0 1
112 0x0 phys 0x0 edge 0 0 1
113 0x0 phys 0x0 edge 0 0 1
114 0x0 phys 0x0 edge 0 0 1
115 0x0 phys 0x0 edge 0 0 1
116 0x0 phys 0x0 edge 0 0 1
117 0x0 phys 0x0 edge 0 0 1
118 0x0 phys 0x0 edge 0 0 1
119 0x0 phys 0x0 edge 0 0 1
----- Entering VM 0 Shell -----
/* no reaction to any regular keys */
---Entering ACRN SHELL---
ACRN:\>
Thank you again, I would never get this far without help.
 Prima
di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Hi Dubravko,
1) Enable X2APIC from guest Linux
Actually ACRN checks X2APIC capability and it goes panic w/o it. So the host CPUID must have X2APIC support.
We met one case where the BIOS DMAR setting has some thing like "do not use x2apic", which prevents Linux from enabling
X2APIC. Or you may check why the guest doesn't switch to X2APIC from native Linux dmesg?
2) Need Linux logs to help debugging.
Attached is the ACRN built with the following hack, which directs guest serial prints directly to HV console.
Seems Linux goes to the path of numachip_wakeup_secondary() to send SIPIs to secondary CPUs, which we didn't see before.
It's good to see Linux early pintk t help address the issue.
diff --git a/hypervisor/dm/vuart.c b/hypervisor/dm/vuart.c
index 8771a7d8a15f..be5f0da943b8 100644
--- a/hypervisor/dm/vuart.c
+++ b/hypervisor/dm/vuart.c
@@ -295,6 +295,7 @@ static void write_reg(struct acrn_vuart *vu, uint16_t reg, uint8_t value_u8)
vu->lsr |= LSR_OE;
} else {
fifo_putchar(&vu->txfifo, (char)value_u8);
+ printf("%c", value_u8);
}
vu->thre_int_pending = true;
break;
3) "naked" HV is working?
In the meantime, if you want, you can hack ACRN with the following and it won't launch any VMs and the HV console is
supposed to be working. It's just a way to check that ACRN is "generally" working on the HW and you may play with some HV
console commands.
diff --git a/hypervisor/arch/x86/guest/vm.c b/hypervisor/arch/x86/guest/vm.c
index 0471324be065..53ccb6b9acae 100644
--- a/hypervisor/arch/x86/guest/vm.c
+++ b/hypervisor/arch/x86/guest/vm.c
@@ -873,6 +873,8 @@ void launch_vms(uint16_t pcpu_id)
uint16_t vm_id, bsp_id;
struct acrn_vm_config *vm_config;
+ return;
+
for (vm_id = 0U; vm_id < CONFIG_MAX_VM_NUM; vm_id++) {
vm_config = get_vm_config(vm_id);
if ((vm_config->load_order == SOS_VM) || (vm_config->load_order == PRE_LAUNCHED_VM)) {
Best Regards,
Zide
On 2/12/20 11:05 AM, Zide Chen wrote:
> Hi Dubravko,
>
> With my acrn.efi, it switches to guest console by default. Does it switch back to ACRN console by pressing "Ctrl + Space"?
>
> The logs show that X2APIC is not enabled by Linux. Is it possible to enable X2APIC from BIOS?
>
> Without X2APIC, ACRN doesn't work. Though I'm expecting ACRN should go beyond what it is now.
>
> BTW, can you remove "quiet" from the Linux bootargs? This may not really help, but I'm puzzled why it doesn't print any
> Linux logs.
>
> From the logs, one big difference with my images (acrn.efi + bzImage) is that now it doesn't repeat the SIPI sequence
> (INIT -> Start up -> INIT -> Start up) endless, instead, after launching 3 VCPUs, the guest sends broadcasting SIPI INIT
> and Start Up only. This is a good news, but not sure that's because of the new acrn.efi, or the bzImage?
>
> Best Regards,
> Zide
>
> On 2/12/20 10:36 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>> Hi Zide,
>>
>> I've checked all the menus and settings in our BIOS, unfortunately I couldn't find anything that looks like the
>> mentioned setting or similar or related to it.
>>
>> We are using coreboot, built according to instructions from
>>
https://software.intel.com/en-us/firmware/sites/default/files/intelatome3900-0.71-buildinstructions.txt, plus we added
>> support for our board (the link apparently no longer works). In general it works very well.
>>
>> Best regards,
>> Dubravko
>>
>>
>> *Dubravko Moravski*
>> /SW engineering/
>> *Exor Embedded S.r.l.*
>> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>> w: exorint.com <https://exorint.com/>
>>
>>
>>
>>
>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>
>> Privacy <https://www.exorint.com/it/privacy>
>> --------------------------------------------------------------------------------------------------------------------------
>> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
>> Lists.Projectacrn.Org <zide.chen=intel.com@...>
>> *Sent:* Wednesday, February 12, 2020 6:47 PM
>> *To:* acrn-users@... <acrn-users@...>
>> *Subject:* Re: [acrn-users] Getting ACRN to work
>> Hi Dubravko,
>>
>> In the meantime, is there something like "AP Thread Idle Manner" in your BIOS? And changing its setting makes any
>> difference?
>>
>> Best Regards,
>> Zide
>>
>> On 2/12/20 8:02 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>>> Hello,
>>>
>>> Please find attached the log file and my acrn.conf.
>>>
>>> My old BIOS didn't have any settings related to Mwait, so I've installed newer BIOS which has "Monitor Mwait Enable" and
>>> I've ensured it's set to "Disabled". Changing the BIOS seems to have also changed some data in logs, but the overall
>>> behavior is the same.
>>>
>>> New acrn.efi seems to work reasonably well. Our custom board is obviously specific, but in general it is similar to
>>> reference Leaf Hill CRB.
>>>
>>> After the last printout from the log, everything stops working (there is nothing on the screen attached to the board and
>>> the keyboard doesn't do anything; also typing in the serial terminal doesn't do anything).
>>>
>>> Best regards,
>>> Dubravko
>>>
>>>
>>> *Dubravko Moravski*
>>> /SW engineering/
>>> *Exor Embedded S.r.l.*
>>> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
>>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>>> w: exorint.com <https://exorint.com/>
>>>
>>>
>>>
>>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>>
>>> Privacy <https://www.exorint.com/it/privacy>
>>> --------------------------------------------------------------------------------------------------------------------------
>>> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
>>> Lists.Projectacrn.Org <zide.chen=intel.com@...>
>>> *Sent:* Wednesday, February 12, 2020 1:04 AM
>>> *To:* acrn-users@... <acrn-users@...>
>>> *Subject:* Re: [acrn-users] Getting ACRN to work
>>> Hi Dubravko,
>>>
>>> "Spurious vector" could because of external interrupts not registered in hypervisor. This may not be an issue and can be
>>> ignored for now.
>>>
>>> For the repeating "Starting VCPU 3" issue, it seems some thing related to SIPI delivery. But nothing is obvious to me and
>>> need to debug it.
>>>
>>> I built a Clear Linux kernel with additional logs showing how it's bringing up APs. Could you please try the attached
>>> bzImage?
>>>
>>> Also please enable early printk by adding "ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep" to the options in
>>> acrn.conf; And use the attached acrn.efi (I enabled some additional logs and print out Linux logs even before ACRN console
>>> is operational). If acrn.efi doesn't boot, which is highly possibly since it's not built against your custom board, you
>>> may need to apply the attached patch to ACRN hypervisor.
>>>
>>> Hopefully the full logs can give us some clues.
>>>
>>> Best Regards,
>>> Zide
>>>
>>> zide.chen@...
>>>
>>>
>>
>>
>>
>>
|
|
|
That’s great.
For the Ethernet issue, ‘lspci -k’ to check if the device exists in SOS and how about the driver status?
If you are using a customized kernel, make sure the related ethernet driver configured. It may be compiled as module, don’t forget put it into system filesystem(/lib/modules/).
Thanks
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Wednesday, February 19, 2020 22:57
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Thank you, you were right, with your fix ACRN loads the Service OS on my board too.
Now I have the issue that Ethernet doesn't work (while it does in regular Clear Linux), so I can't download and configure the User OS. I'll try downloading, configuring and recompiling https://github.com/clearlinux-pkgs/linux-iot-lts2018
kernel. Is that the right one and do I need to activate some particular settings so it works well with ACRN?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I glanced through the discussion and found it might be a similar issue that I hit today with UP2 board.
My problem is that with ACRN enabled the linux kernel cannot bootup APs, without ACRN hypervisor the native kernel works well.
I have a fix that posted on acrn-dev maillist several hours ago. Perhaps you can have a quick try and hope it can help.
I paste it here.
---
On UEFI UP2 board, BIOS startup APs and then they all go into HLT status.
Then Guest OS take over and re-init the APs again. The flows from HV perspective is like:
wait_event(VCPU_EVENT_VIRTUAL_INTERRUPT) -> sleep_thread
-> pause_vcpu(ZOMBIE) -> sleep_thread
-> reset_vcpu
-> launch_vcpu -> wake_vcpu
However, the last wake_vcpu will fail because the cpu event VCPU_EVENT_VIRTUAL_INTERRUPT had not got signaled.
Add signal_event before sleep_thread in pause_vcpu to make sure the vcpu thread is not waiting for event.
Signed-off-by: Shuo A Liu <shuo.a.liu@...>
---
hypervisor/arch/x86/guest/vcpu.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hypervisor/arch/x86/guest/vcpu.c b/hypervisor/arch/x86/guest/vcpu.c
index 6161889..bd2d2d2 100644
--- a/hypervisor/arch/x86/guest/vcpu.c
+++ b/hypervisor/arch/x86/guest/vcpu.c
@@ -701,6 +701,8 @@ void pause_vcpu(struct acrn_vcpu *vcpu, enum vcpu_state new_state)
vcpu->state = new_state;
if (vcpu->prev_state == VCPU_RUNNING) {
+ /* signal_event will wakeup the vcpu's thread if it's in HLT sleep, else do nothing. */
+ signal_event(&vcpu->events[VCPU_EVENT_VIRTUAL_INTERRUPT]);
sleep_thread(&vcpu->thread_obj);
}
if (pcpu_id != get_pcpu_id()) {
--
2.8.3
Thanks
shuo
I see that it's not mentioned anywhere in ACRN log (and I suppose it should be), but it seems to be enabled somehow in non-hypervised Linux:
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $ sudo dmesg | grep -i x2apic
[ 0.165205] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.167422] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.167426] x2apic enabled
[ 0.167448] Switched APIC routing to cluster x2apic.
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $
Our BIOS does not have a GUI setting to manually enable it, and we couldn't find where to explicitly enable it in source code. It's mentioned at many places; though not in DMAR table.
Can we conclude from the log above that it's ok, or we still need to find a way to enable it in ACRN or BIOS?
I'm working on this and will report later.
By the way, I think my last logs might be inconsistent with earlier ones (with endless SIPI messages) because earlier I might have in a hurry just pressed enter in the boot menu to choose whichever
option was selected, instead of selecting ACRN. Now I'm more careful.
Removing 'quiet' parameter didn't help.
... is working. With your previous acrn.efi and bzImage from around Wednesday noon Central European Time, after serial boot log stops, I can indeed switch to ACRN console with ctrl+space. If I
switch back to SOS, nothing happens. I can only return back to ACRN console. Here are some printouts:
[13133696us][cpu=0][vm0:vcpu0][sev=3][seq=322]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13146659us][cpu=0][vm0:vcpu0][sev=3][seq=323]:vcpu1 paused, new state: 4
[13154097us][cpu=0][vm0:vcpu0][sev=3][seq=324]:vcpu1 reset
[13160078us][cpu=0][vm0:vcpu0][sev=3][seq=325]:vlapic: vlapic_reset 1693 vm0 vcpu1 initial lapic_id: 2000000
[13170960us][cpu=0][vm0:vcpu0][sev=3][seq=326]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13183985us][cpu=0][vm0:vcpu0][sev=3][seq=327]:vcpu2 paused, new state: 4
[13191429us][cpu=0][vm0:vcpu0][sev=3][seq=328]:vcpu2 reset
[13197408us][cpu=0][vm0:vcpu0][sev=3][seq=329]:vlapic: vlapic_reset 1693 vm0 vcpu2 initial lapic_id: 4000000
[13208295us][cpu=0][vm0:vcpu0][sev=3][seq=330]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 500 icr_low c4500 sipi cnt 0 vcpu state 2
[13221324us][cpu=0][vm0:vcpu0][sev=3][seq=331]:vcpu3 paused, new state: 4
[13228769us][cpu=0][vm0:vcpu0][sev=3][seq=332]:vcpu3 reset
[13234748us][cpu=0][vm0:vcpu0][sev=3][seq=333]:vlapic: vlapic_reset 1693 vm0 vcpu3 initial lapic_id: 6000000
[13255651us][cpu=0][vm0:vcpu0][sev=3][seq=334]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13268597us][cpu=0][vm0:vcpu0][sev=3][seq=335]:vlapic: Start Secondary VCPU1 for VM[0]...
[13277605us][cpu=0][vm0:vcpu0][sev=3][seq=336]:vcpu1 scheduled on pcpu1 vcpu->state 1
[13286228us][cpu=0][vm0:vcpu0][sev=3][seq=337]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13299256us][cpu=0][vm0:vcpu0][sev=3][seq=338]:vlapic: Start Secondary VCPU2 for VM[0]...
[13308274us][cpu=0][vm0:vcpu0][sev=3][seq=339]:vcpu2 scheduled on pcpu2 vcpu->state 1
[13316902us][cpu=0][vm0:vcpu0][sev=3][seq=340]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 1 vcpu state 1
[13329919us][cpu=0][vm0:vcpu0][sev=3][seq=341]:vlapic: Start Secondary VCPU3 for VM[0]...
[13338926us][cpu=0][vm0:vcpu0][sev=3][seq=342]:vcpu3 scheduled on pcpu3 vcpu->state 1
[13347768us][cpu=0][vm0:vcpu0][sev=3][seq=343]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
[13360707us][cpu=0][vm0:vcpu0][sev=3][seq=344]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
[13373751us][cpu=0][vm0:vcpu0][sev=3][seq=345]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 0 vcpu state 2
---Entering ACRN SHELL---
ACRN:\>
ACRN:\>version
HV version 1.6-unstable-2020-02-11 15:28:54-4990d8574159 DBG (daily tag: acrn-2020w03.4-140000p) build by zide
VM_UUID VM_ID VM_NAME VM_STATE
================================ ===== ================================ ========
dbbbd4347a574216a12c2201f1ab0240 0 ACRN SOS VM Started
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
===== ======= ======= ========= ========== ==========
0 0 0 PRIMARY Running RUNNING
0 1 1 SECONDARY Running BLOCKED
0 2 2 SECONDARY Running BLOCKED
0 3 3 SECONDARY Running BLOCKED
Error: Invalid parameters.
PIN VEC DM DEST TM DELM IRR MASK
0 0x0 phys 0x0 edge 0 0 1
1 0x0 phys 0x0 edge 0 0 1
2 0x0 phys 0x0 edge 0 0 1
3 0x0 phys 0x0 edge 0 0 1
4 0x0 phys 0x0 edge 0 0 1
5 0x0 phys 0x0 edge 0 0 1
6 0x0 phys 0x0 edge 0 0 1
7 0x0 phys 0x0 edge 0 0 1
8 0x0 phys 0x0 edge 0 0 1
9 0x0 phys 0x0 edge 0 0 1
10 0x0 phys 0x0 edge 0 0 1
11 0x0 phys 0x0 edge 0 0 1
12 0x0 phys 0x0 edge 0 0 1
13 0x0 phys 0x0 edge 0 0 1
14 0x0 phys 0x0 edge 0 0 1
15 0x0 phys 0x0 edge 0 0 1
16 0x0 phys 0x0 edge 0 0 1
17 0x0 phys 0x0 edge 0 0 1
18 0x0 phys 0x0 edge 0 0 1
19 0x0 phys 0x0 edge 0 0 1
20 0x0 phys 0x0 edge 0 0 1
21 0x0 phys 0x0 edge 0 0 1
22 0x0 phys 0x0 edge 0 0 1
23 0x0 phys 0x0 edge 0 0 1
24 0x0 phys 0x0 edge 0 0 1
25 0x0 phys 0x0 edge 0 0 1
26 0x0 phys 0x0 edge 0 0 1
27 0x0 phys 0x0 edge 0 0 1
28 0x0 phys 0x0 edge 0 0 1
29 0x0 phys 0x0 edge 0 0 1
30 0x0 phys 0x0 edge 0 0 1
31 0x0 phys 0x0 edge 0 0 1
32 0x0 phys 0x0 edge 0 0 1
33 0x0 phys 0x0 edge 0 0 1
34 0x0 phys 0x0 edge 0 0 1
35 0x0 phys 0x0 edge 0 0 1
36 0x0 phys 0x0 edge 0 0 1
37 0x0 phys 0x0 edge 0 0 1
38 0x0 phys 0x0 edge 0 0 1
39 0x0 phys 0x0 edge 0 0 1
40 0x0 phys 0x0 edge 0 0 1
41 0x0 phys 0x0 edge 0 0 1
42 0x0 phys 0x0 edge 0 0 1
43 0x0 phys 0x0 edge 0 0 1
44 0x0 phys 0x0 edge 0 0 1
45 0x0 phys 0x0 edge 0 0 1
46 0x0 phys 0x0 edge 0 0 1
47 0x0 phys 0x0 edge 0 0 1
48 0x0 phys 0x0 edge 0 0 1
49 0x0 phys 0x0 edge 0 0 1
50 0x0 phys 0x0 edge 0 0 1
51 0x0 phys 0x0 edge 0 0 1
52 0x0 phys 0x0 edge 0 0 1
53 0x0 phys 0x0 edge 0 0 1
54 0x0 phys 0x0 edge 0 0 1
55 0x0 phys 0x0 edge 0 0 1
56 0x0 phys 0x0 edge 0 0 1
57 0x0 phys 0x0 edge 0 0 1
58 0x0 phys 0x0 edge 0 0 1
59 0x0 phys 0x0 edge 0 0 1
60 0x0 phys 0x0 edge 0 0 1
61 0x0 phys 0x0 edge 0 0 1
62 0x0 phys 0x0 edge 0 0 1
63 0x0 phys 0x0 edge 0 0 1
64 0x0 phys 0x0 edge 0 0 1
65 0x0 phys 0x0 edge 0 0 1
66 0x0 phys 0x0 edge 0 0 1
67 0x0 phys 0x0 edge 0 0 1
68 0x0 phys 0x0 edge 0 0 1
69 0x0 phys 0x0 edge 0 0 1
70 0x0 phys 0x0 edge 0 0 1
71 0x0 phys 0x0 edge 0 0 1
72 0x0 phys 0x0 edge 0 0 1
73 0x0 phys 0x0 edge 0 0 1
74 0x0 phys 0x0 edge 0 0 1
75 0x0 phys 0x0 edge 0 0 1
76 0x0 phys 0x0 edge 0 0 1
77 0x0 phys 0x0 edge 0 0 1
78 0x0 phys 0x0 edge 0 0 1
79 0x0 phys 0x0 edge 0 0 1
80 0x0 phys 0x0 edge 0 0 1
81 0x0 phys 0x0 edge 0 0 1
82 0x0 phys 0x0 edge 0 0 1
83 0x0 phys 0x0 edge 0 0 1
84 0x0 phys 0x0 edge 0 0 1
85 0x0 phys 0x0 edge 0 0 1
86 0x0 phys 0x0 edge 0 0 1
87 0x0 phys 0x0 edge 0 0 1
88 0x0 phys 0x0 edge 0 0 1
89 0x0 phys 0x0 edge 0 0 1
90 0x0 phys 0x0 edge 0 0 1
91 0x0 phys 0x0 edge 0 0 1
92 0x0 phys 0x0 edge 0 0 1
93 0x0 phys 0x0 edge 0 0 1
94 0x0 phys 0x0 edge 0 0 1
95 0x0 phys 0x0 edge 0 0 1
96 0x0 phys 0x0 edge 0 0 1
97 0x0 phys 0x0 edge 0 0 1
98 0x0 phys 0x0 edge 0 0 1
99 0x0 phys 0x0 edge 0 0 1
100 0x0 phys 0x0 edge 0 0 1
101 0x0 phys 0x0 edge 0 0 1
102 0x0 phys 0x0 edge 0 0 1
103 0x0 phys 0x0 edge 0 0 1
104 0x0 phys 0x0 edge 0 0 1
105 0x0 phys 0x0 edge 0 0 1
106 0x0 phys 0x0 edge 0 0 1
107 0x0 phys 0x0 edge 0 0 1
108 0x0 phys 0x0 edge 0 0 1
109 0x0 phys 0x0 edge 0 0 1
110 0x0 phys 0x0 edge 0 0 1
111 0x0 phys 0x0 edge 0 0 1
112 0x0 phys 0x0 edge 0 0 1
113 0x0 phys 0x0 edge 0 0 1
114 0x0 phys 0x0 edge 0 0 1
115 0x0 phys 0x0 edge 0 0 1
116 0x0 phys 0x0 edge 0 0 1
117 0x0 phys 0x0 edge 0 0 1
118 0x0 phys 0x0 edge 0 0 1
119 0x0 phys 0x0 edge 0 0 1
----- Entering VM 0 Shell -----
/* no reaction to any regular keys */
---Entering ACRN SHELL---
ACRN:\>
Thank you again, I would never get this far without help.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
1) Enable X2APIC from guest Linux
Actually ACRN checks X2APIC capability and it goes panic w/o it. So the host CPUID must have X2APIC support.
We met one case where the BIOS DMAR setting has some thing like "do not use x2apic", which prevents Linux from enabling
X2APIC. Or you may check why the guest doesn't switch to X2APIC from native Linux dmesg?
2) Need Linux logs to help debugging.
Attached is the ACRN built with the following hack, which directs guest serial prints directly to HV console.
Seems Linux goes to the path of numachip_wakeup_secondary() to send SIPIs to secondary CPUs, which we didn't see before.
It's good to see Linux early pintk t help address the issue.
diff --git a/hypervisor/dm/vuart.c b/hypervisor/dm/vuart.c
index 8771a7d8a15f..be5f0da943b8 100644
--- a/hypervisor/dm/vuart.c
+++ b/hypervisor/dm/vuart.c
@@ -295,6 +295,7 @@ static void write_reg(struct acrn_vuart *vu, uint16_t reg, uint8_t value_u8)
vu->lsr |= LSR_OE;
} else {
fifo_putchar(&vu->txfifo, (char)value_u8);
+ printf("%c", value_u8);
}
vu->thre_int_pending = true;
break;
3) "naked" HV is working?
In the meantime, if you want, you can hack ACRN with the following and it won't launch any VMs and the HV console is
supposed to be working. It's just a way to check that ACRN is "generally" working on the HW and you may play with some HV
console commands.
diff --git a/hypervisor/arch/x86/guest/vm.c b/hypervisor/arch/x86/guest/vm.c
index 0471324be065..53ccb6b9acae 100644
--- a/hypervisor/arch/x86/guest/vm.c
+++ b/hypervisor/arch/x86/guest/vm.c
@@ -873,6 +873,8 @@ void launch_vms(uint16_t pcpu_id)
uint16_t vm_id, bsp_id;
struct acrn_vm_config *vm_config;
+ return;
+
for (vm_id = 0U; vm_id < CONFIG_MAX_VM_NUM; vm_id++) {
vm_config = get_vm_config(vm_id);
if ((vm_config->load_order == SOS_VM) || (vm_config->load_order == PRE_LAUNCHED_VM)) {
Best Regards,
Zide
On 2/12/20 11:05 AM, Zide Chen wrote:
> Hi Dubravko,
>
> With my acrn.efi, it switches to guest console by default. Does it switch back to ACRN console by pressing "Ctrl + Space"?
>
> The logs show that X2APIC is not enabled by Linux. Is it possible to enable X2APIC from BIOS?
>
> Without X2APIC, ACRN doesn't work. Though I'm expecting ACRN should go beyond what it is now.
>
> BTW, can you remove "quiet" from the Linux bootargs? This may not really help, but I'm puzzled why it doesn't print any
> Linux logs.
>
> From the logs, one big difference with my images (acrn.efi + bzImage) is that now it doesn't repeat the SIPI sequence
> (INIT -> Start up -> INIT -> Start up) endless, instead, after launching 3 VCPUs, the guest sends broadcasting SIPI INIT
> and Start Up only. This is a good news, but not sure that's because of the new acrn.efi, or the bzImage?
>
> Best Regards,
> Zide
>
> On 2/12/20 10:36 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>> Hi Zide,
>>
>> I've checked all the menus and settings in our BIOS, unfortunately I couldn't find anything that looks like the
>> mentioned setting or similar or related to it.
>>
>> We are using coreboot, built according to instructions from
>>
https://software.intel.com/en-us/firmware/sites/default/files/intelatome3900-0.71-buildinstructions.txt, plus we added
>> support for our board (the link apparently no longer works). In general it works very well.
>>
>> Best regards,
>> Dubravko
>>
>>
>> *Dubravko Moravski*
>> /SW engineering/
>> *Exor Embedded S.r.l.*
>> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>> w: exorint.com <https://exorint.com/>
>>
>>
>>
>>
>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>
>> Privacy <https://www.exorint.com/it/privacy>
>> --------------------------------------------------------------------------------------------------------------------------
>> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
>> Lists.Projectacrn.Org <zide.chen=intel.com@...>
>> *Sent:* Wednesday, February 12, 2020 6:47 PM
>> *To:* acrn-users@... <acrn-users@...>
>> *Subject:* Re: [acrn-users] Getting ACRN to work
>> Hi Dubravko,
>>
>> In the meantime, is there something like "AP Thread Idle Manner" in your BIOS? And changing its setting makes any
>> difference?
>>
>> Best Regards,
>> Zide
>>
>> On 2/12/20 8:02 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>>> Hello,
>>>
>>> Please find attached the log file and my acrn.conf.
>>>
>>> My old BIOS didn't have any settings related to Mwait, so I've installed newer BIOS which has "Monitor Mwait Enable" and
>>> I've ensured it's set to "Disabled". Changing the BIOS seems to have also changed some data in logs, but the overall
>>> behavior is the same.
>>>
>>> New acrn.efi seems to work reasonably well. Our custom board is obviously specific, but in general it is similar to
>>> reference Leaf Hill CRB.
>>>
>>> After the last printout from the log, everything stops working (there is nothing on the screen attached to the board and
>>> the keyboard doesn't do anything; also typing in the serial terminal doesn't do anything).
>>>
>>> Best regards,
>>> Dubravko
>>>
>>>
>>> *Dubravko Moravski*
>>> /SW engineering/
>>> *Exor Embedded S.r.l.*
>>> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
>>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>>> w: exorint.com <https://exorint.com/>
>>>
>>>
>>>
>>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>>
>>> Privacy <https://www.exorint.com/it/privacy>
>>> --------------------------------------------------------------------------------------------------------------------------
>>> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
>>> Lists.Projectacrn.Org <zide.chen=intel.com@...>
>>> *Sent:* Wednesday, February 12, 2020 1:04 AM
>>> *To:* acrn-users@... <acrn-users@...>
>>> *Subject:* Re: [acrn-users] Getting ACRN to work
>>> Hi Dubravko,
>>>
>>> "Spurious vector" could because of external interrupts not registered in hypervisor. This may not be an issue and can be
>>> ignored for now.
>>>
>>> For the repeating "Starting VCPU 3" issue, it seems some thing related to SIPI delivery. But nothing is obvious to me and
>>> need to debug it.
>>>
>>> I built a Clear Linux kernel with additional logs showing how it's bringing up APs. Could you please try the attached
>>> bzImage?
>>>
>>> Also please enable early printk by adding "ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep" to the options in
>>> acrn.conf; And use the attached acrn.efi (I enabled some additional logs and print out Linux logs even before ACRN console
>>> is operational). If acrn.efi doesn't boot, which is highly possibly since it's not built against your custom board, you
>>> may need to apply the attached patch to ACRN hypervisor.
>>>
>>> Hopefully the full logs can give us some clues.
>>>
>>> Best Regards,
>>> Zide
>>>
>>> zide.chen@...
>>>
>>>
>>
>>
>>
>>
|
|
|
Chen, Zide
Hi Dubravko, I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc. You may compile your own kernel like this: $ git clone https://github.com/projectacrn/acrn-kernel.git$ cd acrn-kernel $ cp kernel_config_uefi_sos .config $ make menuconfig $ make Best Regards, Zide
On 2/19/20 7:20 AM, Shuo A Liu wrote: That’s great.
For the Ethernet issue, ‘lspci -k’ to check if the device exists in SOS and how about the driver status?
If you are using a customized kernel, make sure the related ethernet driver configured. It may be compiled as module, don’t forget put it into system filesystem(/lib/modules/).
Thanks
*From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Wednesday, February 19, 2020 22:57
*To:* acrn-users@...
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Shuo,
Thank you, you were right, with your fix ACRN loads the Service OS on my board too.
Now I have the issue that Ethernet doesn't work (while it does in regular Clear Linux), so I can't download and configure the User OS. I'll try downloading, configuring and recompiling https://github.com/clearlinux-pkgs/linux-iot-lts2018 kernel. Is that the right one and do I need to activate some particular settings so it works well with ACRN?
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:
+38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
a:
Slavonska avenija, 50, Zagreb, Croatia, 10000
w:
exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
*Sent:* Friday, February 14, 2020 5:48 PM
*To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I glanced through the discussion and found it might be a similar issue that I hit today with UP2 board.
My problem is that with ACRN enabled the linux kernel cannot bootup APs, without ACRN hypervisor the native kernel works well.
I have a fix that posted on acrn-dev maillist several hours ago. Perhaps you can have a quick try and hope it can help.
I paste it here.
---
On UEFI UP2 board, BIOS startup APs and then they all go into HLT status.
Then Guest OS take over and re-init the APs again. The flows from HV perspective is like:
wait_event(VCPU_EVENT_VIRTUAL_INTERRUPT) -> sleep_thread
-> pause_vcpu(ZOMBIE) -> sleep_thread
-> reset_vcpu
-> launch_vcpu -> wake_vcpu
However, the last wake_vcpu will fail because the cpu event VCPU_EVENT_VIRTUAL_INTERRUPT had not got signaled.
Add signal_event before sleep_thread in pause_vcpu to make sure the vcpu thread is not waiting for event.
Signed-off-by: Shuo A Liu <shuo.a.liu@... <mailto:shuo.a.liu@...>>
---
hypervisor/arch/x86/guest/vcpu.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hypervisor/arch/x86/guest/vcpu.c b/hypervisor/arch/x86/guest/vcpu.c
index 6161889..bd2d2d2 100644
--- a/hypervisor/arch/x86/guest/vcpu.c
+++ b/hypervisor/arch/x86/guest/vcpu.c
@@ -701,6 +701,8 @@ void pause_vcpu(struct acrn_vcpu *vcpu, enum vcpu_state new_state)
vcpu->state = new_state;
if (vcpu->prev_state == VCPU_RUNNING) {
+ /* signal_event will wakeup the vcpu's thread if it's in HLT sleep, else do nothing. */
+ signal_event(&vcpu->events[VCPU_EVENT_VIRTUAL_INTERRUPT]);
sleep_thread(&vcpu->thread_obj);
}
if (pcpu_id != get_pcpu_id()) {
--
2.8.3
Thanks
shuo
*From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Friday, February 14, 2020 23:41
*To:* acrn-users@... <mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Zide,
1) X2APIC
I see that it's not mentioned anywhere in ACRN log (and I suppose it should be), but it seems to be enabled somehow in non-hypervised Linux:
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $ sudo dmesg | grep -i x2apic
[ 0.165205] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.167422] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.167426] x2apic enabled
[ 0.167448] Switched APIC routing to cluster x2apic.
clear@clr-bcdcb0fadf6546d7b0984fe80030eee8~ $
Our BIOS does not have a GUI setting to manually enable it, and we couldn't find where to explicitly enable it in source code. It's mentioned at many places; though not in DMAR table.
Can we conclude from the log above that it's ok, or we still need to find a way to enable it in ACRN or BIOS?
2) Linux logs
I'm working on this and will report later.
By the way, I think my last logs might be inconsistent with earlier ones (with endless SIPI messages) because earlier I might have in a hurry just pressed enter in the boot menu to choose whichever option was selected, instead of selecting ACRN. Now I'm more careful.
Removing 'quiet' parameter didn't help.
3) Naked HV
... is working. With your previous acrn.efi and bzImage from around Wednesday noon Central European Time, after serial boot log stops, I can indeed switch to ACRN console with ctrl+space. If I switch back to SOS, nothing happens. I can only return back to ACRN console. Here are some printouts:
[13133696us][cpu=0][vm0:vcpu0][sev=3][seq=322]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13146659us][cpu=0][vm0:vcpu0][sev=3][seq=323]:vcpu1 paused, new state: 4
[13154097us][cpu=0][vm0:vcpu0][sev=3][seq=324]:vcpu1 reset
[13160078us][cpu=0][vm0:vcpu0][sev=3][seq=325]:vlapic: vlapic_reset 1693 vm0 vcpu1 initial lapic_id: 2000000
[13170960us][cpu=0][vm0:vcpu0][sev=3][seq=326]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13183985us][cpu=0][vm0:vcpu0][sev=3][seq=327]:vcpu2 paused, new state: 4
[13191429us][cpu=0][vm0:vcpu0][sev=3][seq=328]:vcpu2 reset
[13197408us][cpu=0][vm0:vcpu0][sev=3][seq=329]:vlapic: vlapic_reset 1693 vm0 vcpu2 initial lapic_id: 4000000
[13208295us][cpu=0][vm0:vcpu0][sev=3][seq=330]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 500 icr_low c4500 sipi cnt 0
vcpu state 2
[13221324us][cpu=0][vm0:vcpu0][sev=3][seq=331]:vcpu3 paused, new state: 4
[13228769us][cpu=0][vm0:vcpu0][sev=3][seq=332]:vcpu3 reset
[13234748us][cpu=0][vm0:vcpu0][sev=3][seq=333]:vlapic: vlapic_reset 1693 vm0 vcpu3 initial lapic_id: 6000000
[13255651us][cpu=0][vm0:vcpu0][sev=3][seq=334]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13268597us][cpu=0][vm0:vcpu0][sev=3][seq=335]:vlapic: Start Secondary VCPU1 for VM[0]...
[13277605us][cpu=0][vm0:vcpu0][sev=3][seq=336]:vcpu1 scheduled on pcpu1 vcpu->state 1
[13286228us][cpu=0][vm0:vcpu0][sev=3][seq=337]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13299256us][cpu=0][vm0:vcpu0][sev=3][seq=338]:vlapic: Start Secondary VCPU2 for VM[0]...
[13308274us][cpu=0][vm0:vcpu0][sev=3][seq=339]:vcpu2 scheduled on pcpu2 vcpu->state 1
[13316902us][cpu=0][vm0:vcpu0][sev=3][seq=340]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 1
vcpu state 1
[13329919us][cpu=0][vm0:vcpu0][sev=3][seq=341]:vlapic: Start Secondary VCPU3 for VM[0]...
[13338926us][cpu=0][vm0:vcpu0][sev=3][seq=342]:vcpu3 scheduled on pcpu3 vcpu->state 1
[13347768us][cpu=0][vm0:vcpu0][sev=3][seq=343]:vlapic: pcpu0 Sending SIPI to vcpu1, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
[13360707us][cpu=0][vm0:vcpu0][sev=3][seq=344]:vlapic: pcpu0 Sending SIPI to vcpu2, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
[13373751us][cpu=0][vm0:vcpu0][sev=3][seq=345]:vlapic: pcpu0 Sending SIPI to vcpu3, mode 600 icr_low c469f sipi cnt 0
vcpu state 2
/* ctrl+space */
---Entering ACRN SHELL---
ACRN:\>
ACRN:\>version
HV version 1.6-unstable-2020-02-11 15:28:54-4990d8574159 DBG (daily tag: acrn-2020w03.4-140000p) build by zide
API version 1.0
ACRN:\>vm_list
VM_UUID VM_ID VM_NAME VM_STATE
================================ ===== ================================ ========
dbbbd4347a574216a12c2201f1ab0240 0 ACRN SOS VM Started
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
===== ======= ======= ========= ========== ==========
0 0 0 PRIMARY Running RUNNING
0 1 1 SECONDARY Running BLOCKED
0 2 2 SECONDARY Running BLOCKED
0 3 3 SECONDARY Running BLOCKED
ACRN:\>vioapic
Error: Invalid parameters.
ACRN:\>vioapic 0
PIN VEC DM DEST TM DELM IRR MASK
0 0x0 phys 0x0 edge 0 0 1
1 0x0 phys 0x0 edge 0 0 1
2 0x0 phys 0x0 edge 0 0 1
3 0x0 phys 0x0 edge 0 0 1
4 0x0 phys 0x0 edge 0 0 1
5 0x0 phys 0x0 edge 0 0 1
6 0x0 phys 0x0 edge 0 0 1
7 0x0 phys 0x0 edge 0 0 1
8 0x0 phys 0x0 edge 0 0 1
9 0x0 phys 0x0 edge 0 0 1
10 0x0 phys 0x0 edge 0 0 1
11 0x0 phys 0x0 edge 0 0 1
12 0x0 phys 0x0 edge 0 0 1
13 0x0 phys 0x0 edge 0 0 1
14 0x0 phys 0x0 edge 0 0 1
15 0x0 phys 0x0 edge 0 0 1
16 0x0 phys 0x0 edge 0 0 1
17 0x0 phys 0x0 edge 0 0 1
18 0x0 phys 0x0 edge 0 0 1
19 0x0 phys 0x0 edge 0 0 1
20 0x0 phys 0x0 edge 0 0 1
21 0x0 phys 0x0 edge 0 0 1
22 0x0 phys 0x0 edge 0 0 1
23 0x0 phys 0x0 edge 0 0 1
24 0x0 phys 0x0 edge 0 0 1
25 0x0 phys 0x0 edge 0 0 1
26 0x0 phys 0x0 edge 0 0 1
27 0x0 phys 0x0 edge 0 0 1
28 0x0 phys 0x0 edge 0 0 1
29 0x0 phys 0x0 edge 0 0 1
30 0x0 phys 0x0 edge 0 0 1
31 0x0 phys 0x0 edge 0 0 1
32 0x0 phys 0x0 edge 0 0 1
33 0x0 phys 0x0 edge 0 0 1
34 0x0 phys 0x0 edge 0 0 1
35 0x0 phys 0x0 edge 0 0 1
36 0x0 phys 0x0 edge 0 0 1
37 0x0 phys 0x0 edge 0 0 1
38 0x0 phys 0x0 edge 0 0 1
39 0x0 phys 0x0 edge 0 0 1
40 0x0 phys 0x0 edge 0 0 1
41 0x0 phys 0x0 edge 0 0 1
42 0x0 phys 0x0 edge 0 0 1
43 0x0 phys 0x0 edge 0 0 1
44 0x0 phys 0x0 edge 0 0 1
45 0x0 phys 0x0 edge 0 0 1
46 0x0 phys 0x0 edge 0 0 1
47 0x0 phys 0x0 edge 0 0 1
48 0x0 phys 0x0 edge 0 0 1
49 0x0 phys 0x0 edge 0 0 1
50 0x0 phys 0x0 edge 0 0 1
51 0x0 phys 0x0 edge 0 0 1
52 0x0 phys 0x0 edge 0 0 1
53 0x0 phys 0x0 edge 0 0 1
54 0x0 phys 0x0 edge 0 0 1
55 0x0 phys 0x0 edge 0 0 1
56 0x0 phys 0x0 edge 0 0 1
57 0x0 phys 0x0 edge 0 0 1
58 0x0 phys 0x0 edge 0 0 1
59 0x0 phys 0x0 edge 0 0 1
60 0x0 phys 0x0 edge 0 0 1
61 0x0 phys 0x0 edge 0 0 1
62 0x0 phys 0x0 edge 0 0 1
63 0x0 phys 0x0 edge 0 0 1
64 0x0 phys 0x0 edge 0 0 1
65 0x0 phys 0x0 edge 0 0 1
66 0x0 phys 0x0 edge 0 0 1
67 0x0 phys 0x0 edge 0 0 1
68 0x0 phys 0x0 edge 0 0 1
69 0x0 phys 0x0 edge 0 0 1
70 0x0 phys 0x0 edge 0 0 1
71 0x0 phys 0x0 edge 0 0 1
72 0x0 phys 0x0 edge 0 0 1
73 0x0 phys 0x0 edge 0 0 1
74 0x0 phys 0x0 edge 0 0 1
75 0x0 phys 0x0 edge 0 0 1
76 0x0 phys 0x0 edge 0 0 1
77 0x0 phys 0x0 edge 0 0 1
78 0x0 phys 0x0 edge 0 0 1
79 0x0 phys 0x0 edge 0 0 1
80 0x0 phys 0x0 edge 0 0 1
81 0x0 phys 0x0 edge 0 0 1
82 0x0 phys 0x0 edge 0 0 1
83 0x0 phys 0x0 edge 0 0 1
84 0x0 phys 0x0 edge 0 0 1
85 0x0 phys 0x0 edge 0 0 1
86 0x0 phys 0x0 edge 0 0 1
87 0x0 phys 0x0 edge 0 0 1
88 0x0 phys 0x0 edge 0 0 1
89 0x0 phys 0x0 edge 0 0 1
90 0x0 phys 0x0 edge 0 0 1
91 0x0 phys 0x0 edge 0 0 1
92 0x0 phys 0x0 edge 0 0 1
93 0x0 phys 0x0 edge 0 0 1
94 0x0 phys 0x0 edge 0 0 1
95 0x0 phys 0x0 edge 0 0 1
96 0x0 phys 0x0 edge 0 0 1
97 0x0 phys 0x0 edge 0 0 1
98 0x0 phys 0x0 edge 0 0 1
99 0x0 phys 0x0 edge 0 0 1
100 0x0 phys 0x0 edge 0 0 1
101 0x0 phys 0x0 edge 0 0 1
102 0x0 phys 0x0 edge 0 0 1
103 0x0 phys 0x0 edge 0 0 1
104 0x0 phys 0x0 edge 0 0 1
105 0x0 phys 0x0 edge 0 0 1
106 0x0 phys 0x0 edge 0 0 1
107 0x0 phys 0x0 edge 0 0 1
108 0x0 phys 0x0 edge 0 0 1
109 0x0 phys 0x0 edge 0 0 1
110 0x0 phys 0x0 edge 0 0 1
111 0x0 phys 0x0 edge 0 0 1
112 0x0 phys 0x0 edge 0 0 1
113 0x0 phys 0x0 edge 0 0 1
114 0x0 phys 0x0 edge 0 0 1
115 0x0 phys 0x0 edge 0 0 1
116 0x0 phys 0x0 edge 0 0 1
117 0x0 phys 0x0 edge 0 0 1
118 0x0 phys 0x0 edge 0 0 1
119 0x0 phys 0x0 edge 0 0 1
ACRN:\>
ACRN:\>
ACRN:\>
ACRN:\>
ACRN:\>vm_console 0
----- Entering VM 0 Shell -----
/* no reaction to any regular keys */
/* ctrl+space */
---Entering ACRN SHELL---
ACRN:\>
Thank you again, I would never get this far without help.
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:
+38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
a:
Slavonska avenija, 50, Zagreb, Croatia, 10000
w:
exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:*acrn-users@... <mailto:acrn-users@...><acrn-users@... <mailto:acrn-users@...>> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@... <mailto:zide.chen=intel.com@...>>
*Sent:* Wednesday, February 12, 2020 9:04 PM
*To:* acrn-users@... <mailto:acrn-users@...><acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
1) Enable X2APIC from guest Linux
Actually ACRN checks X2APIC capability and it goes panic w/o it. So the host CPUID must have X2APIC support.
We met one case where the BIOS DMAR setting has some thing like "do not use x2apic", which prevents Linux from enabling
X2APIC. Or you may check why the guest doesn't switch to X2APIC from native Linux dmesg?
2) Need Linux logs to help debugging.
Attached is the ACRN built with the following hack, which directs guest serial prints directly to HV console.
Seems Linux goes to the path of numachip_wakeup_secondary() to send SIPIs to secondary CPUs, which we didn't see before.
It's good to see Linux early pintk t help address the issue.
diff --git a/hypervisor/dm/vuart.c b/hypervisor/dm/vuart.c
index 8771a7d8a15f..be5f0da943b8 100644
--- a/hypervisor/dm/vuart.c
+++ b/hypervisor/dm/vuart.c
@@ -295,6 +295,7 @@ static void write_reg(struct acrn_vuart *vu, uint16_t reg, uint8_t value_u8)
vu->lsr |= LSR_OE;
} else {
fifo_putchar(&vu->txfifo, (char)value_u8);
+ printf("%c", value_u8);
}
vu->thre_int_pending = true;
break;
3) "naked" HV is working?
In the meantime, if you want, you can hack ACRN with the following and it won't launch any VMs and the HV console is
supposed to be working. It's just a way to check that ACRN is "generally" working on the HW and you may play with some HV
console commands.
diff --git a/hypervisor/arch/x86/guest/vm.c b/hypervisor/arch/x86/guest/vm.c
index 0471324be065..53ccb6b9acae 100644
--- a/hypervisor/arch/x86/guest/vm.c
+++ b/hypervisor/arch/x86/guest/vm.c
@@ -873,6 +873,8 @@ void launch_vms(uint16_t pcpu_id)
uint16_t vm_id, bsp_id;
struct acrn_vm_config *vm_config;
+ return;
+
for (vm_id = 0U; vm_id < CONFIG_MAX_VM_NUM; vm_id++) {
vm_config = get_vm_config(vm_id);
if ((vm_config->load_order == SOS_VM) || (vm_config->load_order == PRE_LAUNCHED_VM)) {
Best Regards,
Zide
On 2/12/20 11:05 AM, Zide Chen wrote:
> Hi Dubravko,
>
> With my acrn.efi, it switches to guest console by default. Does it switch back to ACRN console by pressing "Ctrl + Space"?
>
> The logs show that X2APIC is not enabled by Linux. Is it possible to enable X2APIC from BIOS?
>
> Without X2APIC, ACRN doesn't work. Though I'm expecting ACRN should go beyond what it is now.
>
> BTW, can you remove "quiet" from the Linux bootargs? This may not really help, but I'm puzzled why it doesn't print any
> Linux logs.
>
> From the logs, one big difference with my images (acrn.efi + bzImage) is that now it doesn't repeat the SIPI sequence
> (INIT -> Start up -> INIT -> Start up) endless, instead, after launching 3 VCPUs, the guest sends broadcasting SIPI INIT
> and Start Up only. This is a good news, but not sure that's because of the new acrn.efi, or the bzImage?
>
> Best Regards,
> Zide
>
> On 2/12/20 10:36 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>> Hi Zide,
>>
>> I've checked all the menus and settings in our BIOS, unfortunately I couldn't find anything that looks like the
>> mentioned setting or similar or related to it.
>>
>> We are using coreboot, built according to instructions from
>> https://software.intel.com/en-us/firmware/sites/default/files/intelatome3900-0.71-buildinstructions.txt, plus we added
>> support for our board (the link apparently no longer works). In general it works very well.
>>
>> Best regards,
>> Dubravko
>>
>>
>> *Dubravko Moravski*
>> /SW engineering/
>> *Exor Embedded S.r.l.*
>> p: +38 512455659 <tel:+38 512455659 <tel:+38%20512455659>> m:+38 5915402413 <tel:+38 5915402413
<tel:+38%205915402413>>
>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>> w: exorint.com <https://exorint.com/>
>>
>>
>>
>>
>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>
>> Privacy <https://www.exorint.com/it/privacy>
>> --------------------------------------------------------------------------------------------------------------------------
>> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
<mailto:acrn-users@...>> on behalf of Chen, Zide via
>> Lists.Projectacrn.Org <zide.chen=intel.com@... <mailto:zide.chen=intel.com@...>>
>> *Sent:* Wednesday, February 12, 2020 6:47 PM
>> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
<mailto:acrn-users@...>>
>> *Subject:* Re: [acrn-users] Getting ACRN to work
>> Hi Dubravko,
>>
>> In the meantime, is there something like "AP Thread Idle Manner" in your BIOS? And changing its setting makes any
>> difference?
>>
>> Best Regards,
>> Zide
>>
>> On 2/12/20 8:02 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
>>> Hello,
>>>
>>> Please find attached the log file and my acrn.conf.
>>>
>>> My old BIOS didn't have any settings related to Mwait, so I've installed newer BIOS which has "Monitor Mwait Enable" and
>>> I've ensured it's set to "Disabled". Changing the BIOS seems to have also changed some data in logs, but the overall
>>> behavior is the same.
>>>
>>> New acrn.efi seems to work reasonably well. Our custom board is obviously specific, but in general it is similar to
>>> reference Leaf Hill CRB.
>>>
>>> After the last printout from the log, everything stops working (there is nothing on the screen attached to the board and
>>> the keyboard doesn't do anything; also typing in the serial terminal doesn't do anything).
>>>
>>> Best regards,
>>> Dubravko
>>>
>>>
>>> *Dubravko Moravski*
>>> /SW engineering/
>>> *Exor Embedded S.r.l.*
>>> p: +38 512455659 <tel:+38 512455659 <tel:+38%20512455659>> m:+38 5915402413 <tel:+38 5915402413
<tel:+38%205915402413>>
>>> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
>>> w: exorint.com <https://exorint.com/>
>>>
>>>
>>>
>>> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>>>
>>> Privacy <https://www.exorint.com/it/privacy>
>>>
--------------------------------------------------------------------------------------------------------------------------
>>> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
<mailto:acrn-users@...>> on behalf of Chen, Zide via
>>> Lists.Projectacrn.Org <zide.chen=intel.com@... <mailto:zide.chen=intel.com@...>>
>>> *Sent:* Wednesday, February 12, 2020 1:04 AM
>>> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
<mailto:acrn-users@...>>
>>> *Subject:* Re: [acrn-users] Getting ACRN to work
>>> Hi Dubravko,
>>>
>>> "Spurious vector" could because of external interrupts not registered in hypervisor. This may not be an issue and can be
>>> ignored for now.
>>>
>>> For the repeating "Starting VCPU 3" issue, it seems some thing related to SIPI delivery. But nothing is obvious to me and
>>> need to debug it.
>>>
>>> I built a Clear Linux kernel with additional logs showing how it's bringing up APs. Could you please try the attached
>>> bzImage?
>>>
>>> Also please enable early printk by adding "ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep" to the options in
>>> acrn.conf; And use the attached acrn.efi (I enabled some additional logs and print out Linux logs even before ACRN
console
>>> is operational). If acrn.efi doesn't boot, which is highly possibly since it's not built against your custom board, you
>>> may need to apply the attached patch to ACRN hypervisor.
>>>
>>> Hopefully the full logs can give us some clues.
>>>
>>> Best Regards,
>>> Zide
>>>
>>> zide.chen@... <mailto:zide.chen@...>
>>>
>>>
>>
>>
>>
>>
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Zide,
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
Sent: Wednesday, February 19, 2020 5:27 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Zide,
I've noticed that for Service OS kernel, you've provided a link to a dedicated repository. I've had a meeting today and was asked to ask the experts on the mailing list to please clarify:
- Is the dedicated SOS kernel the only one that will work with ACRN?
- Does UOS also use a special kernel?
- Related to the previous questions: we would like to use Ubuntu for both SOS and UOS, because many of our customers are familiar with it. Is that possible? If it's not possible to simply use such a distribution right away, is there a well-defined API or
some other kind of interface in ACRN-specific kernels, that we could port to other kernels?
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Dubravko Moravski | Exor Embedded S.r.l. via Lists.Projectacrn.Org
<dubravko.moravski=exorembedded.net@...>
Sent: Thursday, February 20, 2020 4:06 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Zide,
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
Best regards,
Dubravko
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing
this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
Sent: Wednesday, February 19, 2020 5:27 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Chen, Zide
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Friday, February 21, 2020 7:49 AM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
I've noticed that for Service OS kernel, you've provided a link to a dedicated repository. I've had a meeting today and was asked to ask the experts on the mailing list to please clarify:
-
Is the dedicated SOS kernel the only one that will work with ACRN?
-
Does UOS also use a special kernel?
-
Related to the previous questions: we would like to use Ubuntu for both SOS and UOS, because many of our customers are familiar with it. Is that possible? If it's not possible to simply use such a distribution right away, is there
a well-defined API or some other kind of interface in ACRN-specific kernels, that we could port to other kernels?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Zide,
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
{
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
u32 value = 0;
if (E1000_REMOVED(hw_addr))
return ~value;
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
hw->hw_addr = NULL;
netdev_err(netdev, "PCIe link lost\n");
}
return value;
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
Password:
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the cursor is visible.
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
Sent: Friday, February 21, 2020 5:41 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
Yes, Service OS kernel can be acrn-kernel only because it has some ACRN specific modules. But UOS kernel doesn’t have such limitation, though normally we build UOS kernel from the same source with different config
(kernel_config_uos).
You can run Ubuntu rootfs + acrn-kernel for SOS:
https://projectacrn.github.io/latest/tutorials/using_ubuntu_as_sos.html
And “real” Ubuntu as UOS:
https://projectacrn.github.io/latest/tutorials/running_ubun_as_user_vm.html
Best Regards,
Zide
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Friday, February 21, 2020 7:49 AM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
I've noticed that for Service OS kernel, you've provided a link to a dedicated repository. I've had a meeting today and was asked to ask the experts on the mailing list to please clarify:
-
Is the dedicated SOS kernel the only one that will work with ACRN?
-
Does UOS also use a special kernel?
-
Related to the previous questions: we would like to use Ubuntu for both SOS and UOS, because many of our customers are familiar with it. Is that possible? If it's not possible to simply use such a distribution right away, is there
a well-defined API or some other kind of interface in ACRN-specific kernels, that we could port to other kernels?
 Prima di
stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
 Prima di
stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@...>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@...>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Thursday, February 27, 2020 02:19
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211
chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I
could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the
cursor is visible.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
Yes, Service OS kernel can be acrn-kernel only because it has some ACRN specific modules. But UOS kernel doesn’t have such limitation, though normally we build UOS kernel from the same source
with different config (kernel_config_uos).
You can run Ubuntu rootfs + acrn-kernel for SOS:
https://projectacrn.github.io/latest/tutorials/using_ubuntu_as_sos.html
And “real” Ubuntu as UOS:
https://projectacrn.github.io/latest/tutorials/running_ubun_as_user_vm.html
Best Regards,
Zide
I've noticed that for Service OS kernel, you've provided a link to a dedicated repository. I've had a meeting today and was asked to ask the experts on the mailing list to please clarify:
-
Is the dedicated SOS kernel the only one that will work with ACRN?
-
Does UOS also use a special kernel?
-
Related to the previous questions: we would like to use Ubuntu for both SOS and UOS, because many of our customers are familiar with it. Is that possible? If it's not possible to simply use such a distribution right away, is there
a well-defined API or some other kind of interface in ACRN-specific kernels, that we could port to other kernels?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Geoffroy Van Cutsem
Hi Dubravko,
About the mouse cursor not being visible, can you try to disable the mouse acceleration (most likely somewhere in ‘gnome-tweaks’)?
Thanks,
Geoffroy
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Wednesday, February 26, 2020 7:19 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211
chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I
could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the
cursor is visible.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
Yes, Service OS kernel can be acrn-kernel only because it has some ACRN specific modules. But UOS kernel doesn’t have such limitation, though normally we build UOS kernel from the same source
with different config (kernel_config_uos).
You can run Ubuntu rootfs + acrn-kernel for SOS:
https://projectacrn.github.io/latest/tutorials/using_ubuntu_as_sos.html
And “real” Ubuntu as UOS:
https://projectacrn.github.io/latest/tutorials/running_ubun_as_user_vm.html
Best Regards,
Zide
I've noticed that for Service OS kernel, you've provided a link to a dedicated repository. I've had a meeting today and was asked to ask the experts on the mailing list to please clarify:
-
Is the dedicated SOS kernel the only one that will work with ACRN?
-
Does UOS also use a special kernel?
-
Related to the previous questions: we would like to use Ubuntu for both SOS and UOS, because many of our customers are familiar with it. Is that possible? If it's not possible to simply use such a distribution right away, is there
a well-defined API or some other kind of interface in ACRN-specific kernels, that we could port to other kernels?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
I've managed to download the kernel as instructed, enable the required Ethernet driver, and compile the kernel.
Today when I wanted to test it, my board has died, for no particular reason. Hardware team thinks it's not repairable. I'll have to wait a couple of days at least until I get another one.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I assume x2apic is working for you now since you can mount rootfs. Then you may treat the Ethernet driver issue just like
it on the native Linux, for example, /lib/modules/ matches kernel version 'uname -r'? driver module exists? etc.
You may compile your own kernel like this:
$ git clone https://github.com/projectacrn/acrn-kernel.git
$ cd acrn-kernel
$ cp kernel_config_uefi_sos .config
$ make menuconfig
$ make
Best Regards,
Zide
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi everyone,
I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical, "PCIe link lost". If
you have any other ideas for me to try, please let me know.
Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything related to hardware
acceleration. If you have any other ideas...
clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
cpu1 online=1
cpu2 online=1
cpu3 online=1
cat: '/sys/class/net/e*/address': No such file or directory
passed gvt-g optargs low_gm 64, high_gm 448, fence 8
SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
pm by vuart node-index = 0
logger: name=console, level=4
logger: name=kmsg, level=3
logger: name=disk, level=5
vm_create: vm1
VHM api version 1.0
vm_setup_memory: size=0x80000000
open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
level 0 free/need pages:0/1 page size:0x200000
level 1 free/need pages:0/2 page size:0x40000000
to reserve more free pages:
to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
now enough free pages are reserved!
try to setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
total_size 0x180000000
mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
mmap 0x80000000@0x7fdbc0000000
touch 2 pages with pagesz 0x40000000
mmap 0x200000@0x7fdcbfe00000
touch 1 pages with pagesz 0x200000
really setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
vm_init_vdevs
polling 38...
Listening 38...
pci init hostbridge
pci init lpc
pci init pci-gvt
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
GVT: init failed
pci pci-gvt init failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
Stop listening 38...
Stop polling 38...
Unable to init vdev (2)
I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in configuring ACRN or UOS, I'll re-check what I have done.
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Shuo A Liu via Lists.Projectacrn.Org <shuo.a.liu=intel.com@...>
Sent: Friday, February 28, 2020 9:41 AM
To: acrn-users@... <acrn-users@...>
Cc: Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui <conghui.chen@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@...>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@...>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Thursday, February 27, 2020 02:19
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I could do much debugging
here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the cursor is visible.
|
|
|
Chen, Zide
Hi Dubravko,
Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same acrn tree to avoid any version compatibility issue.
$ cd acrn-hypervisor
$ make devicemodel
$ scp build/devicemodel/acrn-dm target:/usr/bin
For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue
of initializing that device.
Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
Best Regards,
Zide
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Friday, February 28, 2020 9:57 AM
To: acrn-users@...
Cc: Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui <conghui.chen@...>
Subject: Re: [acrn-users] Getting ACRN to work
I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to the subsequent modifications it wasn't trivial but in the end I've managed
- the behavior however remained identical, "PCIe link lost". If you have any other ideas for me to try, please let me know.
Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the acceleration of the movement of the mouse pointer. I've checked all the
other tweak pages, still I couldn't find anything related to hardware acceleration. If you have any other ideas...
clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
cat: '/sys/class/net/e*/address': No such file or directory
passed gvt-g optargs low_gm 64, high_gm 448, fence 8
SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
pm by vuart node-index = 0
logger: name=console, level=4
logger: name=kmsg, level=3
logger: name=disk, level=5
vm_setup_memory: size=0x80000000
open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
level 0 free/need pages:0/1 page size:0x200000
level 1 free/need pages:0/2 page size:0x40000000
to reserve more free pages:
to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
now enough free pages are reserved!
try to setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
mmap 0x80000000@0x7fdbc0000000
touch 2 pages with pagesz 0x40000000
mmap 0x200000@0x7fdcbfe00000
touch 1 pages with pagesz 0x200000
really setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in configuring ACRN or UOS, I'll re-check what I have done.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@...>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@...>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211
chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I
could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the
cursor is visible.
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Zide,
Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow affect it?
I have changed the launch script like this:
acrn-dm -A -m $mem_size -s 0:0,hostbridge \
-s 5,virtio-console,@stdio:stdio_port \
-s 6,virtio-hyper_dmabuf \
-s 3,virtio-blk,/home/clear/work/uos/uos.img \
-s 4,virtio-net,tap0 \
-s 7,virtio-rnd \
--ovmf /usr/share/acrn/bios/OVMF.fd \
$pm_channel $pm_by_vuart $pm_vuart_node \
$logger_setting \
--mac_seed $mac_seed \
$vm_name
}
Apparently it doesn't like this line, now commented out:
# -s 2,pci-gvt -G "$2" \
This happens if it's enabled:
vm_init_vdevs
polling 38...
Listening 38...
pci init hostbridge
pci init lpc
pci init pci-gvt
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
GVT: init failed
pci pci-gvt init failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
Stop listening 38...
Stop polling 38...
Unable to init vdev (2)
Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display outputs simultaneously and drawing/compositing 2D graphics?
About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
Sent: Saturday, February 29, 2020 9:04 AM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same acrn tree to avoid any version compatibility issue.
$ cd acrn-hypervisor
$ make devicemodel
$ scp build/devicemodel/acrn-dm target:/usr/bin
For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of initializing that
device.
Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
Best Regards,
Zide
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Friday, February 28, 2020 9:57 AM
To: acrn-users@...
Cc: Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui <conghui.chen@...>
Subject: Re: [acrn-users] Getting ACRN to work
I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however
remained identical, "PCIe link lost". If you have any other ideas for me to try, please let me know.
Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the acceleration of the movement of the mouse pointer. I've checked all the other tweak pages,
still I couldn't find anything related to hardware acceleration. If you have any other ideas...
clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
cat: '/sys/class/net/e*/address': No such file or directory
passed gvt-g optargs low_gm 64, high_gm 448, fence 8
SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
pm by vuart node-index = 0
logger: name=console, level=4
logger: name=kmsg, level=3
logger: name=disk, level=5
vm_setup_memory: size=0x80000000
open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
level 0 free/need pages:0/1 page size:0x200000
level 1 free/need pages:0/2 page size:0x40000000
to reserve more free pages:
to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
now enough free pages are reserved!
try to setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
mmap 0x80000000@0x7fdbc0000000
touch 2 pages with pagesz 0x40000000
mmap 0x200000@0x7fdcbfe00000
touch 1 pages with pagesz 0x200000
really setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in configuring ACRN or UOS, I'll re-check what I have done.
 Prima di
stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@...>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@...>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I could do much debugging
here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the cursor is visible.
|
|
|
Geoffroy Van Cutsem
Hi Dubravko,
GVT is the graphics (GPU) sharing capability, you can read more about it here:
https://projectacrn.github.io/latest/developer-guides/GVT-g-porting.html
It’s essentially useful only if you want to share the GPU between different Virtual Machines (VMs). If you don’t need/want to do that, you don’t need it (and can still drive multiple displays from the Service VM). Note that there are kernel
command-line parameters that we should adjust in that case too.
About the mouse issue, my suspicion is that it is something wrong with the GVT parameter (cursor plane not working correctly), if you don’t need GVT, you can try to disable it altogether (in /boot/loader/entries/acrn.conf).
Regards,
Geoffroy
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Tuesday, March 3, 2020 2:42 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly sure why, but Ethernet works too! I'm using the same board and the same kernel
as few days ago. Does acrn-dm somehow affect it?
I have changed the launch script like this:
acrn-dm -A -m $mem_size -s 0:0,hostbridge \
-s 5,virtio-console,@stdio:stdio_port \
-s 6,virtio-hyper_dmabuf \
-s 3,virtio-blk,/home/clear/work/uos/uos.img \
--ovmf /usr/share/acrn/bios/OVMF.fd \
$pm_channel $pm_by_vuart $pm_vuart_node \
Apparently it doesn't like this line, now commented out:
This happens if it's enabled:
vm_init_vdevs
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
Unable to init vdev (2)
Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display outputs simultaneously and drawing/compositing 2D graphics?
About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In acrn, the mouse is still functional, but invisible (it's a plain Logitech
USB mouse, nothing special about it).
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same acrn tree to avoid any version compatibility issue.
$ cd acrn-hypervisor
$ make devicemodel
$ scp build/devicemodel/acrn-dm target:/usr/bin
For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue
of initializing that device.
Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
Best Regards,
Zide
I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to the subsequent modifications it wasn't trivial but in the end I've managed
- the behavior however remained identical, "PCIe link lost". If you have any other ideas for me to try, please let me know.
Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the acceleration of the movement of the mouse pointer. I've checked all the
other tweak pages, still I couldn't find anything related to hardware acceleration. If you have any other ideas...
clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
cat: '/sys/class/net/e*/address': No such file or directory
passed gvt-g optargs low_gm 64, high_gm 448, fence 8
SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
pm by vuart node-index = 0
logger: name=console, level=4
logger: name=kmsg, level=3
logger: name=disk, level=5
vm_setup_memory: size=0x80000000
open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
level 0 free/need pages:0/1 page size:0x200000
level 1 free/need pages:0/2 page size:0x40000000
to reserve more free pages:
to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
now enough free pages are reserved!
try to setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
mmap 0x80000000@0x7fdbc0000000
touch 2 pages with pagesz 0x40000000
mmap 0x200000@0x7fdcbfe00000
touch 1 pages with pagesz 0x200000
really setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in configuring ACRN or UOS, I'll re-check what I have done.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@...>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@...>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the
I211 chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized):
PCIe link lost
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
if (E1000_REMOVED(hw_addr))
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
netdev_err(netdev, "PCIe link lost\n");
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like
I could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and
the cursor is visible.
|
|
|
Chen, Zide
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
toggle quoted messageShow quoted text
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote: Hi Zide,
Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow affect it?
I have changed the launch script like this:
acrn-dm -A -m $mem_size -s 0:0,hostbridge \
-s 5,virtio-console,@stdio:stdio_port \
-s 6,virtio-hyper_dmabuf \
-s 3,virtio-blk,/home/clear/work/uos/uos.img \
-s 4,virtio-net,tap0 \
-s 7,virtio-rnd \
--ovmf /usr/share/acrn/bios/OVMF.fd \
$pm_channel $pm_by_vuart $pm_vuart_node \
$logger_setting \
--mac_seed $mac_seed \
$vm_name
}
Apparently it doesn't like this line, now commented out:
# -s 2,pci-gvt -G "$2" \
This happens if it's enabled:
vm_init_vdevs
polling 38...
Listening 38...
pci init hostbridge
pci init lpc
pci init pci-gvt
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
GVT: init failed
pci pci-gvt init failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
Stop listening 38...
Stop polling 38...
Unable to init vdev (2)
Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display outputs simultaneously and drawing/compositing 2D graphics?
About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
*Sent:* Saturday, February 29, 2020 9:04 AM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same acrn tree to avoid any version compatibility issue.
$ cd acrn-hypervisor
$ make devicemodel
$ scp build/devicemodel/acrn-dm target:/usr/bin
For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of initializing that device.
Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
Best Regards,
Zide
*From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Friday, February 28, 2020 9:57 AM
*To:* acrn-users@...
*Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui <conghui.chen@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi everyone,
I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical, "PCIe link lost". If you have any other ideas for me to try, please let me know.
Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything related to hardware acceleration. If you have any other ideas...
Unfortunately I'm also having issues launching User OS. I've followed instructions from here: https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it up goes well, but in the end it fails to launch:
clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
cpu1 online=1
cpu2 online=1
cpu3 online=1
cat: '/sys/class/net/e*/address': No such file or directory
passed gvt-g optargs low_gm 64, high_gm 448, fence 8
SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
pm by vuart node-index = 0
logger: name=console, level=4
logger: name=kmsg, level=3
logger: name=disk, level=5
vm_create: vm1
VHM api version 1.0
vm_setup_memory: size=0x80000000
open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
level 0 free/need pages:0/1 page size:0x200000
level 1 free/need pages:0/2 page size:0x40000000
to reserve more free pages:
to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
now enough free pages are reserved!
try to setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
total_size 0x180000000
mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
mmap 0x80000000@0x7fdbc0000000
touch 2 pages with pagesz 0x40000000
mmap 0x200000@0x7fdcbfe00000
touch 1 pages with pagesz 0x200000
really setup hugepage with:
level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
vm_init_vdevs
polling 38...
Listening 38...
pci init hostbridge
pci init lpc
pci init pci-gvt
GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
GVT: init failed
pci pci-gvt init failed
mngr_client_new: Failed to accept from fd 38, err: Invalid argument
Stop listening 38...
Stop polling 38...
Unable to init vdev (2)
I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in configuring ACRN or UOS, I'll re-check what I have done.
Best regards,
Dubravko
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:
+38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
a:
Slavonska avenija, 50, Zagreb, Croatia, 10000
w:
exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
--------------------------------------------------------------------------------------------------------------------------
*From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
*Sent:* Friday, February 28, 2020 9:41 AM
*To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>>
*Cc:* Wang, Hongbo <hongbo.wang@... <mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@... <mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... <mailto:fei1.li@...>>; Chen, Conghui <conghui.chen@... <mailto:conghui.chen@...>>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help check if your HV include it?
commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
Author: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
Date: Thu Dec 5 22:51:06 2019 +0800
hv: vpci: trap PCIe ECAM access for SOS
SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
VM trap PCIe ECAM access in DM).
Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
platform hide PCI devices by BIOS to SOS.
Tracked-On: #3475
Signed-off-by: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
You can have a quick try to revert it firstly if it is included.
Thanks
shuo
*From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Thursday, February 27, 2020 02:19
*To:* acrn-users@... <mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Getting ACRN to work
Hi Zide,
Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
The new kernel mostly works, however igb driver doesn't (and we really need the network):
clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
[ 3.708554] calling igb_init_module+0x0/0x4d @ 1
[ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
[ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
[ 7.504456] igb 0000:04:00.0: Invalid MAC Address
[ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
[ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
The "PCIe link lost" message originates in igb_main.c, approx line 740:
u32 igb_rd32(struct e1000_hw *hw, u32 reg)
{
struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
u32 value = 0;
if (E1000_REMOVED(hw_addr))
return ~value;
value = readl(&hw_addr[reg]);
/* reads should not return all F's */
if (!(~value) && (!reg || !(~readl(hw_addr)))) {
struct net_device *netdev = igb->netdev;
hw->hw_addr = NULL;
netdev_err(netdev, "*PCIe link lost*\n");
}
return value;
}
I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible level, looking from the software side of things, so it doesn't look like I could do much debugging here.
In regular Clear Linux, the driver consistently works and there are never any link issues:
clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
Password:
[ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
[ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 4.519304] igb 0000:04:00.0: added PHC on eth0
[ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
[ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
[ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
[ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
[ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with the same file system and GUI and system settings, the mouse works and the cursor is visible.
Best regards,
Dubravko
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Zide and Geoffroy,
Yes, igb i211 driver now works on SOS, and UOS can access the network too, without changing any setting or modifying the kernel. Usually I would suspect some hardware issue, but we haven't seen any issues with PCIe so far on the board, so this is puzzling.
Invisible mouse cursor, it works on regular Clear Linux with the same SOS kernel (that I've built, with igb driver), and the same rootfs (/dev/mmcblk1p3, and we have only one MMC device so it can't be confused with anything else).
"sudo dmesg | grep -i mouse" produces identical output when it's visible and invisible, and nothing looks suspicious.
It looks like I don't need GVT, but just in case it's simple to fix, I've checked dmesg (no mentions of it) and /sys/kernel/ directory. Here, gvt/ subfolder is missing, and UOS launching script complains about that. Do I just need to enable some driver or start
some service to get it?
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Chen, Zide via Lists.Projectacrn.Org <zide.chen=intel.com@...>
Sent: Tuesday, March 3, 2020 5:40 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same
rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only
when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
toggle quoted messageShow quoted text
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
> Hi Zide,
>
> Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly
> sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow
> affect it?
>
> I have changed the launch script like this:
>
> acrn-dm -A -m $mem_size -s 0:0,hostbridge \
> -s 5,virtio-console,@stdio:stdio_port \
> -s 6,virtio-hyper_dmabuf \
> -s 3,virtio-blk,/home/clear/work/uos/uos.img \
> -s 4,virtio-net,tap0 \
> -s 7,virtio-rnd \
> --ovmf /usr/share/acrn/bios/OVMF.fd \
> $pm_channel $pm_by_vuart $pm_vuart_node \
> $logger_setting \
> --mac_seed $mac_seed \
> $vm_name
> }
>
> Apparently it doesn't like this line, now commented out:
>
> # -s 2,pci-gvt -G "$2" \
>
> This happens if it's enabled:
>
> vm_init_vdevs
> polling 38...
> Listening 38...
> pci init hostbridge
> pci init lpc
> pci init pci-gvt
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
> GVT: init failed
> pci pci-gvt init failed
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
> Stop listening 38...
> Stop polling 38...
> Unable to init vdev (2)
>
> Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display
> outputs simultaneously and drawing/compositing 2D graphics?
>
> About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In
> acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
>
> Best regards,
> Dubravko
>
>
> *Dubravko Moravski*
> /SW engineering/
> *Exor Embedded S.r.l.*
> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
> w: exorint.com < https://exorint.com/>
>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy < https://www.exorint.com/it/privacy>
> --------------------------------------------------------------------------------------------------------------------------
> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
> Lists.Projectacrn.Org <zide.chen=intel.com@...>
> *Sent:* Saturday, February 29, 2020 9:04 AM
> *To:* acrn-users@... <acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same
> acrn tree to avoid any version compatibility issue.
>
> $ cd acrn-hypervisor
>
> $ make devicemodel
>
> $ scp build/devicemodel/acrn-dm target:/usr/bin
>
> For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you
> may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of
> initializing that device.
>
> Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems
> not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
>
> Best Regards,
>
> Zide
>
> *From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor
> Embedded S.r.l.
> *Sent:* Friday, February 28, 2020 9:57 AM
> *To:* acrn-users@...
> *Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui
> <conghui.chen@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi everyone,
>
> I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to
> the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical,
> "PCIe link lost". If you have any other ideas for me to try, please let me know.
>
> Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the
> acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything
> related to hardware acceleration. If you have any other ideas...
>
> Unfortunately I'm also having issues launching User OS. I've followed instructions from here:
>
https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it
> up goes well, but in the end it fails to launch:
>
> clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
>
> cpu1 online=1
>
> cpu2 online=1
>
> cpu3 online=1
>
> cat: '/sys/class/net/e*/address': No such file or directory
>
> passed gvt-g optargs low_gm 64, high_gm 448, fence 8
>
> SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
>
> pm by vuart node-index = 0
>
> logger: name=console, level=4
>
> logger: name=kmsg, level=3
>
> logger: name=disk, level=5
>
> vm_create: vm1
>
> VHM api version 1.0
>
> vm_setup_memory: size=0x80000000
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
>
> level 0 free/need pages:0/1 page size:0x200000
>
> level 1 free/need pages:0/2 page size:0x40000000
>
> to reserve more free pages:
>
> to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
>
> to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> now enough free pages are reserved!
>
> try to setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> total_size 0x180000000
>
> mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
>
> mmap 0x80000000@0x7fdbc0000000
>
> touch 2 pages with pagesz 0x40000000
>
> mmap 0x200000@0x7fdcbfe00000
>
> touch 1 pages with pagesz 0x200000
>
> really setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> vm_init_vdevs
>
> polling 38...
>
> Listening 38...
>
> pci init hostbridge
>
> pci init lpc
>
> pci init pci-gvt
>
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
>
> GVT: init failed
>
> pci pci-gvt init failed
>
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
>
> Stop listening 38...
>
> Stop polling 38...
>
> Unable to init vdev (2)
>
> I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in
> configuring ACRN or UOS, I'll re-check what I have done.
>
> Best regards,
>
> Dubravko
>
> *Dubravko Moravski*
>
> /SW engineering/
>
> *Exor Embedded S.r.l.*
>
> p:
>
>
>
> +38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
>
> a:
>
>
>
> Slavonska avenija, 50, Zagreb, Croatia, 10000
>
> w:
>
>
>
> exorint.com < https://exorint.com/>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy < https://www.exorint.com/it/privacy>
>
> --------------------------------------------------------------------------------------------------------------------------
>
> *From:*acrn-users@... < mailto:acrn-users@...> <acrn-users@...
> < mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org
> <shuo.a.liu=intel.com@... < mailto:shuo.a.liu=intel.com@...>>
> *Sent:* Friday, February 28, 2020 9:41 AM
> *To:* acrn-users@... < mailto:acrn-users@...> <acrn-users@...
> < mailto:acrn-users@...>>
> *Cc:* Wang, Hongbo <hongbo.wang@... < mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@...
> < mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... < mailto:fei1.li@...>>; Chen, Conghui
> <conghui.chen@... < mailto:conghui.chen@...>>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help
> check if your HV include it?
>
> commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
>
> Author: Li Fei1 <fei1.li@... < mailto:fei1.li@...>>
>
> Date: Thu Dec 5 22:51:06 2019 +0800
>
> hv: vpci: trap PCIe ECAM access for SOS
>
> SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
>
> access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
>
> VM trap PCIe ECAM access in DM).
>
> Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
>
> platform hide PCI devices by BIOS to SOS.
>
> Tracked-On: #3475
>
> Signed-off-by: Li Fei1 <fei1.li@... < mailto:fei1.li@...>>
>
> You can have a quick try to revert it firstly if it is included.
>
> Thanks
>
> shuo
>
> *From:* acrn-users@... < mailto:acrn-users@...> <acrn-users@...
> < mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
> *Sent:* Thursday, February 27, 2020 02:19
> *To:* acrn-users@... < mailto:acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Zide,
>
> Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
>
> I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the
> kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
> The new kernel mostly works, however igb driver doesn't (and we really need the network):
>
> clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
>
> [ 3.708554] calling igb_init_module+0x0/0x4d @ 1
>
> [ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
>
> [ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
>
> [ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
>
> [ 7.504456] igb 0000:04:00.0: Invalid MAC Address
>
> [ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
>
> [ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
>
> The "PCIe link lost" message originates in igb_main.c, approx line 740:
>
> u32 igb_rd32(struct e1000_hw *hw, u32 reg)
>
> {
>
> struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
>
> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>
> u32 value = 0;
>
> if (E1000_REMOVED(hw_addr))
>
> return ~value;
>
> value = readl(&hw_addr[reg]);
>
> /* reads should not return all F's */
>
> if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>
> struct net_device *netdev = igb->netdev;
>
> hw->hw_addr = NULL;
>
> netdev_err(netdev, "*PCIe link lost*\n");
>
> }
>
> return value;
>
> }
>
> I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible
> level, looking from the software side of things, so it doesn't look like I could do much debugging here.
>
> In regular Clear Linux, the driver consistently works and there are never any link issues:
>
> clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
>
> Password:
>
> [ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
>
> [ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
>
> [ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 4.519304] igb 0000:04:00.0: added PHC on eth0
>
> [ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
>
> [ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
>
> [ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
>
> [ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
>
> [ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
>
> [ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
>
> [ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>
> So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
>
> Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with
> the same file system and GUI and system settings, the mouse works and the cursor is visible.
>
> Best regards,
>
> Dubravko
>
>
|
|
|
Geoffroy Van Cutsem
Can you send your kernel command-line parameters (in the SOS): $ cat /proc/cmdline?
We should also check in the kernel config (/proc/config.gz) in case GVT got inadvertently turned off (CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN) when you reconfigured and recompiled your kernel.
Thanks,
Geoffroy
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Tuesday, March 3, 2020 7:52 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Yes, igb i211 driver now works on SOS, and UOS can access the network too, without changing any setting or modifying the kernel. Usually I would suspect some hardware issue, but we haven't seen
any issues with PCIe so far on the board, so this is puzzling.
Invisible mouse cursor, it works on regular Clear Linux with the same SOS kernel (that I've built, with igb driver), and the same rootfs (/dev/mmcblk1p3, and we have only one MMC device so it can't
be confused with anything else).
"sudo dmesg | grep -i mouse" produces identical output when it's visible and invisible, and nothing looks suspicious.
It looks like I don't need GVT, but just in case it's simple to fix, I've checked dmesg (no mentions of it) and /sys/kernel/ directory. Here, gvt/ subfolder is missing, and UOS launching script
complains about that. Do I just need to enable some driver or start some service to get it?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same
rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only
when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
> Hi Zide,
>
> Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly
> sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow
> affect it?
>
> I have changed the launch script like this:
>
> acrn-dm -A -m $mem_size -s 0:0,hostbridge \
> -s 5,virtio-console,@stdio:stdio_port \
> -s 6,virtio-hyper_dmabuf \
> -s 3,virtio-blk,/home/clear/work/uos/uos.img \
> -s 4,virtio-net,tap0 \
> -s 7,virtio-rnd \
> --ovmf /usr/share/acrn/bios/OVMF.fd \
> $pm_channel $pm_by_vuart $pm_vuart_node \
> $logger_setting \
> --mac_seed $mac_seed \
> $vm_name
> }
>
> Apparently it doesn't like this line, now commented out:
>
> # -s 2,pci-gvt -G "$2" \
>
> This happens if it's enabled:
>
> vm_init_vdevs
> polling 38...
> Listening 38...
> pci init hostbridge
> pci init lpc
> pci init pci-gvt
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
> GVT: init failed
> pci pci-gvt init failed
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
> Stop listening 38...
> Stop polling 38...
> Unable to init vdev (2)
>
> Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display
> outputs simultaneously and drawing/compositing 2D graphics?
>
> About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In
> acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
>
> Best regards,
> Dubravko
>
>
> *Dubravko Moravski*
> /SW engineering/
> *Exor Embedded S.r.l.*
> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
> w: exorint.com <https://exorint.com/>
>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
> --------------------------------------------------------------------------------------------------------------------------
> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
> Lists.Projectacrn.Org <zide.chen=intel.com@...>
> *Sent:* Saturday, February 29, 2020 9:04 AM
> *To:* acrn-users@... <acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same
> acrn tree to avoid any version compatibility issue.
>
> $ cd acrn-hypervisor
>
> $ make devicemodel
>
> $ scp build/devicemodel/acrn-dm target:/usr/bin
>
> For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you
> may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of
> initializing that device.
>
> Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems
> not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
>
> Best Regards,
>
> Zide
>
> *From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor
> Embedded S.r.l.
> *Sent:* Friday, February 28, 2020 9:57 AM
> *To:* acrn-users@...
> *Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui
> <conghui.chen@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi everyone,
>
> I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to
> the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical,
> "PCIe link lost". If you have any other ideas for me to try, please let me know.
>
> Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the
> acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything
> related to hardware acceleration. If you have any other ideas...
>
> Unfortunately I'm also having issues launching User OS. I've followed instructions from here:
>
https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it
> up goes well, but in the end it fails to launch:
>
> clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
>
> cpu1 online=1
>
> cpu2 online=1
>
> cpu3 online=1
>
> cat: '/sys/class/net/e*/address': No such file or directory
>
> passed gvt-g optargs low_gm 64, high_gm 448, fence 8
>
> SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
>
> pm by vuart node-index = 0
>
> logger: name=console, level=4
>
> logger: name=kmsg, level=3
>
> logger: name=disk, level=5
>
> vm_create: vm1
>
> VHM api version 1.0
>
> vm_setup_memory: size=0x80000000
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
>
> level 0 free/need pages:0/1 page size:0x200000
>
> level 1 free/need pages:0/2 page size:0x40000000
>
> to reserve more free pages:
>
> to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
>
> to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> now enough free pages are reserved!
>
> try to setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> total_size 0x180000000
>
> mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
>
> mmap 0x80000000@0x7fdbc0000000
>
> touch 2 pages with pagesz 0x40000000
>
> mmap 0x200000@0x7fdcbfe00000
>
> touch 1 pages with pagesz 0x200000
>
> really setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> vm_init_vdevs
>
> polling 38...
>
> Listening 38...
>
> pci init hostbridge
>
> pci init lpc
>
> pci init pci-gvt
>
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
>
> GVT: init failed
>
> pci pci-gvt init failed
>
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
>
> Stop listening 38...
>
> Stop polling 38...
>
> Unable to init vdev (2)
>
> I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in
> configuring ACRN or UOS, I'll re-check what I have done.
>
> Best regards,
>
> Dubravko
>
> *Dubravko Moravski*
>
> /SW engineering/
>
> *Exor Embedded S.r.l.*
>
> p:
>
>
>
> +38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
>
> a:
>
>
>
> Slavonska avenija, 50, Zagreb, Croatia, 10000
>
> w:
>
>
>
> exorint.com <https://exorint.com/>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
>
> --------------------------------------------------------------------------------------------------------------------------
>
> *From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org
> <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
> *Sent:* Friday, February 28, 2020 9:41 AM
> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>>
> *Cc:* Wang, Hongbo <hongbo.wang@... <mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@...
> <mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... <mailto:fei1.li@...>>; Chen, Conghui
> <conghui.chen@... <mailto:conghui.chen@...>>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help
> check if your HV include it?
>
> commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
>
> Author: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> Date: Thu Dec 5 22:51:06 2019 +0800
>
> hv: vpci: trap PCIe ECAM access for SOS
>
> SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
>
> access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
>
> VM trap PCIe ECAM access in DM).
>
> Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
>
> platform hide PCI devices by BIOS to SOS.
>
> Tracked-On: #3475
>
> Signed-off-by: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> You can have a quick try to revert it firstly if it is included.
>
> Thanks
>
> shuo
>
> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
> *Sent:* Thursday, February 27, 2020 02:19
> *To:* acrn-users@... <mailto:acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Zide,
>
> Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
>
> I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the
> kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
> The new kernel mostly works, however igb driver doesn't (and we really need the network):
>
> clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
>
> [ 3.708554] calling igb_init_module+0x0/0x4d @ 1
>
> [ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
>
> [ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
>
> [ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
>
> [ 7.504456] igb 0000:04:00.0: Invalid MAC Address
>
> [ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
>
> [ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
>
> The "PCIe link lost" message originates in igb_main.c, approx line 740:
>
> u32 igb_rd32(struct e1000_hw *hw, u32 reg)
>
> {
>
> struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
>
> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>
> u32 value = 0;
>
> if (E1000_REMOVED(hw_addr))
>
> return ~value;
>
> value = readl(&hw_addr[reg]);
>
> /* reads should not return all F's */
>
> if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>
> struct net_device *netdev = igb->netdev;
>
> hw->hw_addr = NULL;
>
> netdev_err(netdev, "*PCIe link lost*\n");
>
> }
>
> return value;
>
> }
>
> I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible
> level, looking from the software side of things, so it doesn't look like I could do much debugging here.
>
> In regular Clear Linux, the driver consistently works and there are never any link issues:
>
> clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
>
> Password:
>
> [ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
>
> [ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
>
> [ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 4.519304] igb 0000:04:00.0: added PHC on eth0
>
> [ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
>
> [ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
>
> [ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
>
> [ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
>
> [ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
>
> [ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
>
> [ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>
> So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
>
> Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with
> the same file system and GUI and system settings, the mouse works and the cursor is visible.
>
> Best regards,
>
> Dubravko
>
>
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Geoffroy,
SOS command line parameters are:
initrd=\EFI\org.clearlinux\freestanding-00-intel-ucode.cpio initrd=\EFI\org.clearlinux\freestanding-i915-firmware.cpio.xz root=PARTUUID=ef5fabcd-7773-447d-ad51-0fb2b6522c90 quiet console=tty0 console=ttyS0,115200n8 cryptomgr.notests init=/usr/bin/initra-desktop
initcall_debug intel_iommu=igfx_off kvm-intel.nested=1 no_timer_check noreplace-smp page_alloc.shuffle=1 rcu_nocbs=0-64 rcupdate.rcu_expedited=1 rootfstype=ext4,btrfs,xfs tsc=reliable rw ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep
I've searched for all config parameters containing 915_GVT substring, CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN are on, CONFIG_DRM_I915_GVT_KVMGT is off.
Regard,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Geoffroy Van Cutsem via Lists.Projectacrn.Org <geoffroy.vancutsem=intel.com@...>
Sent: Tuesday, March 3, 2020 10:33 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
Can you send your kernel command-line parameters (in the SOS): $ cat /proc/cmdline?
We should also check in the kernel config (/proc/config.gz) in case GVT got inadvertently turned off (CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN) when you reconfigured and recompiled your kernel.
Thanks,
Geoffroy
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Tuesday, March 3, 2020 7:52 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
Yes, igb i211 driver now works on SOS, and UOS can access the network too, without changing any setting or modifying the kernel. Usually I would suspect some hardware issue, but we haven't seen any issues with PCIe
so far on the board, so this is puzzling.
Invisible mouse cursor, it works on regular Clear Linux with the same SOS kernel (that I've built, with igb driver), and the same rootfs (/dev/mmcblk1p3, and we have only one MMC device so it can't be confused with
anything else).
"sudo dmesg | grep -i mouse" produces identical output when it's visible and invisible, and nothing looks suspicious.
It looks like I don't need GVT, but just in case it's simple to fix, I've checked dmesg (no mentions of it) and /sys/kernel/ directory. Here, gvt/ subfolder is missing, and UOS launching script complains about that.
Do I just need to enable some driver or start some service to get it?
 Prima di
stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same
rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only
when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
> Hi Zide,
>
> Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly
> sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow
> affect it?
>
> I have changed the launch script like this:
>
> acrn-dm -A -m $mem_size -s 0:0,hostbridge \
> -s 5,virtio-console,@stdio:stdio_port \
> -s 6,virtio-hyper_dmabuf \
> -s 3,virtio-blk,/home/clear/work/uos/uos.img \
> -s 4,virtio-net,tap0 \
> -s 7,virtio-rnd \
> --ovmf /usr/share/acrn/bios/OVMF.fd \
> $pm_channel $pm_by_vuart $pm_vuart_node \
> $logger_setting \
> --mac_seed $mac_seed \
> $vm_name
> }
>
> Apparently it doesn't like this line, now commented out:
>
> # -s 2,pci-gvt -G "$2" \
>
> This happens if it's enabled:
>
> vm_init_vdevs
> polling 38...
> Listening 38...
> pci init hostbridge
> pci init lpc
> pci init pci-gvt
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
> GVT: init failed
> pci pci-gvt init failed
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
> Stop listening 38...
> Stop polling 38...
> Unable to init vdev (2)
>
> Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display
> outputs simultaneously and drawing/compositing 2D graphics?
>
> About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In
> acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
>
> Best regards,
> Dubravko
>
>
> *Dubravko Moravski*
> /SW engineering/
> *Exor Embedded S.r.l.*
> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
> w: exorint.com <https://exorint.com/>
>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
> --------------------------------------------------------------------------------------------------------------------------
> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
> Lists.Projectacrn.Org <zide.chen=intel.com@...>
> *Sent:* Saturday, February 29, 2020 9:04 AM
> *To:* acrn-users@... <acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same
> acrn tree to avoid any version compatibility issue.
>
> $ cd acrn-hypervisor
>
> $ make devicemodel
>
> $ scp build/devicemodel/acrn-dm target:/usr/bin
>
> For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you
> may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of
> initializing that device.
>
> Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems
> not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
>
> Best Regards,
>
> Zide
>
> *From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor
> Embedded S.r.l.
> *Sent:* Friday, February 28, 2020 9:57 AM
> *To:* acrn-users@...
> *Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui
> <conghui.chen@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi everyone,
>
> I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to
> the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical,
> "PCIe link lost". If you have any other ideas for me to try, please let me know.
>
> Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the
> acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything
> related to hardware acceleration. If you have any other ideas...
>
> Unfortunately I'm also having issues launching User OS. I've followed instructions from here:
>
https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it
> up goes well, but in the end it fails to launch:
>
> clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
>
> cpu1 online=1
>
> cpu2 online=1
>
> cpu3 online=1
>
> cat: '/sys/class/net/e*/address': No such file or directory
>
> passed gvt-g optargs low_gm 64, high_gm 448, fence 8
>
> SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
>
> pm by vuart node-index = 0
>
> logger: name=console, level=4
>
> logger: name=kmsg, level=3
>
> logger: name=disk, level=5
>
> vm_create: vm1
>
> VHM api version 1.0
>
> vm_setup_memory: size=0x80000000
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
>
> level 0 free/need pages:0/1 page size:0x200000
>
> level 1 free/need pages:0/2 page size:0x40000000
>
> to reserve more free pages:
>
> to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
>
> to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> now enough free pages are reserved!
>
> try to setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> total_size 0x180000000
>
> mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
>
> mmap 0x80000000@0x7fdbc0000000
>
> touch 2 pages with pagesz 0x40000000
>
> mmap 0x200000@0x7fdcbfe00000
>
> touch 1 pages with pagesz 0x200000
>
> really setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> vm_init_vdevs
>
> polling 38...
>
> Listening 38...
>
> pci init hostbridge
>
> pci init lpc
>
> pci init pci-gvt
>
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
>
> GVT: init failed
>
> pci pci-gvt init failed
>
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
>
> Stop listening 38...
>
> Stop polling 38...
>
> Unable to init vdev (2)
>
> I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in
> configuring ACRN or UOS, I'll re-check what I have done.
>
> Best regards,
>
> Dubravko
>
> *Dubravko Moravski*
>
> /SW engineering/
>
> *Exor Embedded S.r.l.*
>
> p:
>
>
>
> +38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
>
> a:
>
>
>
> Slavonska avenija, 50, Zagreb, Croatia, 10000
>
> w:
>
>
>
> exorint.com <https://exorint.com/>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
>
> --------------------------------------------------------------------------------------------------------------------------
>
> *From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org
> <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
> *Sent:* Friday, February 28, 2020 9:41 AM
> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>>
> *Cc:* Wang, Hongbo <hongbo.wang@... <mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@...
> <mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... <mailto:fei1.li@...>>; Chen, Conghui
> <conghui.chen@... <mailto:conghui.chen@...>>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help
> check if your HV include it?
>
> commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
>
> Author: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> Date: Thu Dec 5 22:51:06 2019 +0800
>
> hv: vpci: trap PCIe ECAM access for SOS
>
> SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
>
> access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
>
> VM trap PCIe ECAM access in DM).
>
> Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
>
> platform hide PCI devices by BIOS to SOS.
>
> Tracked-On: #3475
>
> Signed-off-by: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> You can have a quick try to revert it firstly if it is included.
>
> Thanks
>
> shuo
>
> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
> *Sent:* Thursday, February 27, 2020 02:19
> *To:* acrn-users@... <mailto:acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Zide,
>
> Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
>
> I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the
> kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
> The new kernel mostly works, however igb driver doesn't (and we really need the network):
>
> clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
>
> [ 3.708554] calling igb_init_module+0x0/0x4d @ 1
>
> [ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
>
> [ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
>
> [ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
>
> [ 7.504456] igb 0000:04:00.0: Invalid MAC Address
>
> [ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
>
> [ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
>
> The "PCIe link lost" message originates in igb_main.c, approx line 740:
>
> u32 igb_rd32(struct e1000_hw *hw, u32 reg)
>
> {
>
> struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
>
> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>
> u32 value = 0;
>
> if (E1000_REMOVED(hw_addr))
>
> return ~value;
>
> value = readl(&hw_addr[reg]);
>
> /* reads should not return all F's */
>
> if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>
> struct net_device *netdev = igb->netdev;
>
> hw->hw_addr = NULL;
>
> netdev_err(netdev, "*PCIe link lost*\n");
>
> }
>
> return value;
>
> }
>
> I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible
> level, looking from the software side of things, so it doesn't look like I could do much debugging here.
>
> In regular Clear Linux, the driver consistently works and there are never any link issues:
>
> clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
>
> Password:
>
> [ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
>
> [ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
>
> [ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 4.519304] igb 0000:04:00.0: added PHC on eth0
>
> [ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
>
> [ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
>
> [ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
>
> [ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
>
> [ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
>
> [ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
>
> [ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>
> So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
>
> Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with
> the same file system and GUI and system settings, the mouse works and the cursor is visible.
>
> Best regards,
>
> Dubravko
>
>
|
|
|
Geoffroy Van Cutsem
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Wednesday, March 4, 2020 2:14 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
SOS command line parameters are:
initrd=\EFI\org.clearlinux\freestanding-00-intel-ucode.cpio initrd=\EFI\org.clearlinux\freestanding-i915-firmware.cpio.xz root=PARTUUID=ef5fabcd-7773-447d-ad51-0fb2b6522c90 quiet console=tty0 console=ttyS0,115200n8
cryptomgr.notests init=/usr/bin/initra-desktop initcall_debug intel_iommu=igfx_off kvm-intel.nested=1 no_timer_check noreplace-smp page_alloc.shuffle=1 rcu_nocbs=0-64 rcupdate.rcu_expedited=1 rootfstype=ext4,btrfs,xfs tsc=reliable rw ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep
I've searched for all config parameters containing 915_GVT substring, CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN are on, CONFIG_DRM_I915_GVT_KVMGT is off.
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Can you send your kernel command-line parameters (in the SOS): $ cat /proc/cmdline?
We should also check in the kernel config (/proc/config.gz) in case GVT got inadvertently turned off (CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN) when you reconfigured and recompiled your kernel.
Thanks,
Geoffroy
Yes, igb i211 driver now works on SOS, and UOS can access the network too, without changing any setting or modifying the kernel. Usually I would suspect some hardware issue, but we haven't seen
any issues with PCIe so far on the board, so this is puzzling.
Invisible mouse cursor, it works on regular Clear Linux with the same SOS kernel (that I've built, with igb driver), and the same rootfs (/dev/mmcblk1p3, and we have only one MMC device so it
can't be confused with anything else).
"sudo dmesg | grep -i mouse" produces identical output when it's visible and invisible, and nothing looks suspicious.
It looks like I don't need GVT, but just in case it's simple to fix, I've checked dmesg (no mentions of it) and /sys/kernel/ directory. Here, gvt/ subfolder is missing, and UOS launching script
complains about that. Do I just need to enable some driver or start some service to get it?
 Prima di stampare pensa ai costi ambientali.
Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same
rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only
when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
> Hi Zide,
>
> Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly
> sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow
> affect it?
>
> I have changed the launch script like this:
>
> acrn-dm -A -m $mem_size -s 0:0,hostbridge \
> -s 5,virtio-console,@stdio:stdio_port \
> -s 6,virtio-hyper_dmabuf \
> -s 3,virtio-blk,/home/clear/work/uos/uos.img \
> -s 4,virtio-net,tap0 \
> -s 7,virtio-rnd \
> --ovmf /usr/share/acrn/bios/OVMF.fd \
> $pm_channel $pm_by_vuart $pm_vuart_node \
> $logger_setting \
> --mac_seed $mac_seed \
> $vm_name
> }
>
> Apparently it doesn't like this line, now commented out:
>
> # -s 2,pci-gvt -G "$2" \
>
> This happens if it's enabled:
>
> vm_init_vdevs
> polling 38...
> Listening 38...
> pci init hostbridge
> pci init lpc
> pci init pci-gvt
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
> GVT: init failed
> pci pci-gvt init failed
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
> Stop listening 38...
> Stop polling 38...
> Unable to init vdev (2)
>
> Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display
> outputs simultaneously and drawing/compositing 2D graphics?
>
> About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In
> acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
>
> Best regards,
> Dubravko
>
>
> *Dubravko Moravski*
> /SW engineering/
> *Exor Embedded S.r.l.*
> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
> w: exorint.com <https://exorint.com/>
>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
> --------------------------------------------------------------------------------------------------------------------------
> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
> Lists.Projectacrn.Org <zide.chen=intel.com@...>
> *Sent:* Saturday, February 29, 2020 9:04 AM
> *To:* acrn-users@... <acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same
> acrn tree to avoid any version compatibility issue.
>
> $ cd acrn-hypervisor
>
> $ make devicemodel
>
> $ scp build/devicemodel/acrn-dm target:/usr/bin
>
> For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you
> may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of
> initializing that device.
>
> Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems
> not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
>
> Best Regards,
>
> Zide
>
> *From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor
> Embedded S.r.l.
> *Sent:* Friday, February 28, 2020 9:57 AM
> *To:* acrn-users@...
> *Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui
> <conghui.chen@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi everyone,
>
> I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to
> the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical,
> "PCIe link lost". If you have any other ideas for me to try, please let me know.
>
> Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the
> acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything
> related to hardware acceleration. If you have any other ideas...
>
> Unfortunately I'm also having issues launching User OS. I've followed instructions from here:
>
https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it
> up goes well, but in the end it fails to launch:
>
> clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
>
> cpu1 online=1
>
> cpu2 online=1
>
> cpu3 online=1
>
> cat: '/sys/class/net/e*/address': No such file or directory
>
> passed gvt-g optargs low_gm 64, high_gm 448, fence 8
>
> SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
>
> pm by vuart node-index = 0
>
> logger: name=console, level=4
>
> logger: name=kmsg, level=3
>
> logger: name=disk, level=5
>
> vm_create: vm1
>
> VHM api version 1.0
>
> vm_setup_memory: size=0x80000000
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
>
> level 0 free/need pages:0/1 page size:0x200000
>
> level 1 free/need pages:0/2 page size:0x40000000
>
> to reserve more free pages:
>
> to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
>
> to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> now enough free pages are reserved!
>
> try to setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> total_size 0x180000000
>
> mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
>
> mmap 0x80000000@0x7fdbc0000000
>
> touch 2 pages with pagesz 0x40000000
>
> mmap 0x200000@0x7fdcbfe00000
>
> touch 1 pages with pagesz 0x200000
>
> really setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> vm_init_vdevs
>
> polling 38...
>
> Listening 38...
>
> pci init hostbridge
>
> pci init lpc
>
> pci init pci-gvt
>
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
>
> GVT: init failed
>
> pci pci-gvt init failed
>
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
>
> Stop listening 38...
>
> Stop polling 38...
>
> Unable to init vdev (2)
>
> I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in
> configuring ACRN or UOS, I'll re-check what I have done.
>
> Best regards,
>
> Dubravko
>
> *Dubravko Moravski*
>
> /SW engineering/
>
> *Exor Embedded S.r.l.*
>
> p:
>
>
>
> +38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
>
> a:
>
>
>
> Slavonska avenija, 50, Zagreb, Croatia, 10000
>
> w:
>
>
>
> exorint.com <https://exorint.com/>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
>
> --------------------------------------------------------------------------------------------------------------------------
>
> *From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org
> <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
> *Sent:* Friday, February 28, 2020 9:41 AM
> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>>
> *Cc:* Wang, Hongbo <hongbo.wang@... <mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@...
> <mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... <mailto:fei1.li@...>>; Chen, Conghui
> <conghui.chen@... <mailto:conghui.chen@...>>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help
> check if your HV include it?
>
> commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
>
> Author: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> Date: Thu Dec 5 22:51:06 2019 +0800
>
> hv: vpci: trap PCIe ECAM access for SOS
>
> SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
>
> access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
>
> VM trap PCIe ECAM access in DM).
>
> Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
>
> platform hide PCI devices by BIOS to SOS.
>
> Tracked-On: #3475
>
> Signed-off-by: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> You can have a quick try to revert it firstly if it is included.
>
> Thanks
>
> shuo
>
> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
> *Sent:* Thursday, February 27, 2020 02:19
> *To:* acrn-users@... <mailto:acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Zide,
>
> Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
>
> I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the
> kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
> The new kernel mostly works, however igb driver doesn't (and we really need the network):
>
> clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
>
> [ 3.708554] calling igb_init_module+0x0/0x4d @ 1
>
> [ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
>
> [ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
>
> [ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
>
> [ 7.504456] igb 0000:04:00.0: Invalid MAC Address
>
> [ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
>
> [ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
>
> The "PCIe link lost" message originates in igb_main.c, approx line 740:
>
> u32 igb_rd32(struct e1000_hw *hw, u32 reg)
>
> {
>
> struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
>
> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>
> u32 value = 0;
>
> if (E1000_REMOVED(hw_addr))
>
> return ~value;
>
> value = readl(&hw_addr[reg]);
>
> /* reads should not return all F's */
>
> if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>
> struct net_device *netdev = igb->netdev;
>
> hw->hw_addr = NULL;
>
> netdev_err(netdev, "*PCIe link lost*\n");
>
> }
>
> return value;
>
> }
>
> I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible
> level, looking from the software side of things, so it doesn't look like I could do much debugging here.
>
> In regular Clear Linux, the driver consistently works and there are never any link issues:
>
> clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
>
> Password:
>
> [ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
>
> [ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
>
> [ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 4.519304] igb 0000:04:00.0: added PHC on eth0
>
> [ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
>
> [ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
>
> [ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
>
> [ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
>
> [ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
>
> [ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
>
> [ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>
> So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
>
> Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with
> the same file system and GUI and system settings, the mouse works and the cursor is visible.
>
> Best regards,
>
> Dubravko
>
>
|
|
|
Dubravko Moravski | Exor Embedded S.r.l. <dubravko.moravski@...>
Hi Geoffroy,
I've tried your recommendations, they partially work but not quite. I did already have acrn.conf, some tool must have generated it at some point.
I have three different situations, depending on how I start the system:
1) Regular Clear Linux - mouse works normally; and Terminal too.
2) ACRN without GVT settings - mouse is invisible; Terminal works normally.
3) ACRN with GVT settings - mouse is visible but it's flickering and duplicated several times (in all the assigned planes?); Terminal now doesn't work, the application window opens but it's unusable and it shows this message:
"There was an error creating the child process for this terminal.
Failed to open PTY: Permission denied"
Other applications, e.g. file browser and Firefox, do work.
I've also tried changing avail_planes_per_pipe to 0x00000F, it seemed more clean, if I've understood the documentation correctly. This changed nothing, except maybe reduced the flickering a bit.
All the tests were done without launching UOS. I'm using display port output. Our board also has eDP and HDMI outputs, which are all verified to work but aren't connected to a display at the moment.
Best regards,
Dubravko
 Prima di stampare pensa ai costi ambientali. Please consider the
environment before printing this email.
Privacy
From: acrn-users@... <acrn-users@...> on behalf of Geoffroy Van Cutsem via Lists.Projectacrn.Org <geoffroy.vancutsem=intel.com@...>
Sent: Wednesday, March 4, 2020 4:33 PM
To: acrn-users@... <acrn-users@...>
Subject: Re: [acrn-users] Getting ACRN to work
toggle quoted messageShow quoted text
From: acrn-users@... <acrn-users@...>
On Behalf Of Dubravko Moravski | Exor Embedded S.r.l.
Sent: Wednesday, March 4, 2020 2:14 PM
To: acrn-users@...
Subject: Re: [acrn-users] Getting ACRN to work
SOS command line parameters are:
initrd=\EFI\org.clearlinux\freestanding-00-intel-ucode.cpio initrd=\EFI\org.clearlinux\freestanding-i915-firmware.cpio.xz root=PARTUUID=ef5fabcd-7773-447d-ad51-0fb2b6522c90 quiet console=tty0 console=ttyS0,115200n8
cryptomgr.notests init=/usr/bin/initra-desktop initcall_debug intel_iommu=igfx_off kvm-intel.nested=1 no_timer_check noreplace-smp page_alloc.shuffle=1 rcu_nocbs=0-64 rcupdate.rcu_expedited=1 rootfstype=ext4,btrfs,xfs tsc=reliable rw ignore_loglevel earlyprintk=serial,ttyS0,115200n8,keep
I've searched for all config parameters containing 915_GVT substring, CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN are on, CONFIG_DRM_I915_GVT_KVMGT is off.
 Prima di
stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Can you send your kernel command-line parameters (in the SOS): $ cat /proc/cmdline?
We should also check in the kernel config (/proc/config.gz) in case GVT got inadvertently turned off (CONFIG_DRM_I915_GVT and CONFIG_DRM_I915_GVT_ACRN) when you reconfigured and recompiled your kernel.
Thanks,
Geoffroy
Yes, igb i211 driver now works on SOS, and UOS can access the network too, without changing any setting or modifying the kernel. Usually I would suspect some hardware issue, but we haven't seen any issues with PCIe
so far on the board, so this is puzzling.
Invisible mouse cursor, it works on regular Clear Linux with the same SOS kernel (that I've built, with igb driver), and the same rootfs (/dev/mmcblk1p3, and we have only one MMC device so it can't be confused with
anything else).
"sudo dmesg | grep -i mouse" produces identical output when it's visible and invisible, and nothing looks suspicious.
It looks like I don't need GVT, but just in case it's simple to fix, I've checked dmesg (no mentions of it) and /sys/kernel/ directory. Here, gvt/ subfolder is missing, and UOS launching script complains about that.
Do I just need to enable some driver or start some service to get it?
 Prima
di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy
Hi Dubravko,
I'm not familiar with GVT, and let other guys to comment on this topic.
You meant lgb i211 driver is working on SOS now? I'm sure it has nothing to do with acrn-dm which has impacts on UOS only.
The invisible mouse cursor issue, you meant it works on regular (native) Linux with same SOS kernel? Did you use the same
rootfs with SOS? I feel it more likely a rootfs issue and may not related Hypervisor. We need to look at Hypervisor only
when SOS Kernel + rootfs work on native Linux but not on top of ACRN.
Best Regards,
Zide
On 3/3/20 5:41 AM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
> Hi Zide,
>
> Your advice was helpful again. I have recompiled acrn-dm, edited the launch script, and now UOS works. I'm not exactly
> sure why, but Ethernet works too! I'm using the same board and the same kernel as few days ago. Does acrn-dm somehow
> affect it?
>
> I have changed the launch script like this:
>
> acrn-dm -A -m $mem_size -s 0:0,hostbridge \
> -s 5,virtio-console,@stdio:stdio_port \
> -s 6,virtio-hyper_dmabuf \
> -s 3,virtio-blk,/home/clear/work/uos/uos.img \
> -s 4,virtio-net,tap0 \
> -s 7,virtio-rnd \
> --ovmf /usr/share/acrn/bios/OVMF.fd \
> $pm_channel $pm_by_vuart $pm_vuart_node \
> $logger_setting \
> --mac_seed $mac_seed \
> $vm_name
> }
>
> Apparently it doesn't like this line, now commented out:
>
> # -s 2,pci-gvt -G "$2" \
>
> This happens if it's enabled:
>
> vm_init_vdevs
> polling 38...
> Listening 38...
> pci init hostbridge
> pci init lpc
> pci init pci-gvt
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
> GVT: init failed
> pci pci-gvt init failed
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
> Stop listening 38...
> Stop polling 38...
> Unable to init vdev (2)
>
> Reading the documentation, I'm not precisely sure what GVT does. Would it be beneficial to have it for using more display
> outputs simultaneously and drawing/compositing 2D graphics?
>
> About using the same kernel as acrn in regular Linux and testing Ethernet and the mouse, both work in regular Linux. In
> acrn, the mouse is still functional, but invisible (it's a plain Logitech USB mouse, nothing special about it).
>
> Best regards,
> Dubravko
>
>
> *Dubravko Moravski*
> /SW engineering/
> *Exor Embedded S.r.l.*
> p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
> a: Slavonska avenija, 50, Zagreb, Croatia, 10000
> w: exorint.com <https://exorint.com/>
>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
> --------------------------------------------------------------------------------------------------------------------------
> *From:* acrn-users@... <acrn-users@...> on behalf of Chen, Zide via
> Lists.Projectacrn.Org <zide.chen=intel.com@...>
> *Sent:* Saturday, February 29, 2020 9:04 AM
> *To:* acrn-users@... <acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> Since you are using the latest hypervisor code, I’d suggest build your own acrn-dm and use the launch script from the same
> acrn tree to avoid any version compatibility issue.
>
> $ cd acrn-hypervisor
>
> $ make devicemodel
>
> $ scp build/devicemodel/acrn-dm target:/usr/bin
>
> For the script, you may start from acrn-hypervisor/devicemodel/samples/nuc /launch_uos.sh. If it runs into problem, you
> may try customize the script, for example, remove some passthru device from acrn-dm command line if it complains issue of
> initializing that device.
>
> Regarding the Ethernet/mouse driver, did you boot the native Linux with the SOS kernel and the same SOS rootfs? It seems
> not very likely a kernel config issue, but probably it’s still worthy to rule out kernel issue.
>
> Best Regards,
>
> Zide
>
> *From:* acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor
> Embedded S.r.l.
> *Sent:* Friday, February 28, 2020 9:57 AM
> *To:* acrn-users@...
> *Cc:* Wang, Hongbo <hongbo.wang@...>; Wang, Yu1 <yu1.wang@...>; Li, Fei1 <fei1.li@...>; Chen, Conghui
> <conghui.chen@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi everyone,
>
> I confirm my ACRN sources did include 65ed6c3529de8b3f3d890e95a7d816afba7bf379 commit. I've tried reverting it - due to
> the subsequent modifications it wasn't trivial but in the end I've managed - the behavior however remained identical,
> "PCIe link lost". If you have any other ideas for me to try, please let me know.
>
> Regarding the invisible mouse pointer, I've found mouse acceleration settings in gnome-tweaks, but they relate only to the
> acceleration of the movement of the mouse pointer. I've checked all the other tweak pages, still I couldn't find anything
> related to hardware acceleration. If you have any other ideas...
>
> Unfortunately I'm also having issues launching User OS. I've followed instructions from here:
>
https://projectacrn.github.io/latest/tutorials/kbl-nuc-sdc.html#kbl-nuc-sdc, Set up Reference User VM chapter. Setting it
> up goes well, but in the end it fails to launch:
>
> clear@clr-sos-guest~/work/uos $ sudo ./launch_uos.sh
>
> cpu1 online=1
>
> cpu2 online=1
>
> cpu3 online=1
>
> cat: '/sys/class/net/e*/address': No such file or directory
>
> passed gvt-g optargs low_gm 64, high_gm 448, fence 8
>
> SW_LOAD: get ovmf path /usr/share/acrn/bios/OVMF.fd, size 0x200000
>
> pm by vuart node-index = 0
>
> logger: name=console, level=4
>
> logger: name=kmsg, level=3
>
> logger: name=disk, level=5
>
> vm_create: vm1
>
> VHM api version 1.0
>
> vm_setup_memory: size=0x80000000
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv1/vm1/D279543825D611E8864ECB7A18B34643
>
> open hugetlbfs file /run/hugepage/acrn/huge_lv2/vm1/D279543825D611E8864ECB7A18B34643
>
> level 0 free/need pages:0/1 page size:0x200000
>
> level 1 free/need pages:0/2 page size:0x40000000
>
> to reserve more free pages:
>
> to reserve pages (+orig 0): echo 2 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
>
> to reserve pages (+orig 0): echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> now enough free pages are reserved!
>
> try to setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> total_size 0x180000000
>
> mmap ptr 0x0x7fdbbc4e2000 -> baseaddr 0x0x7fdbc0000000
>
> mmap 0x80000000@0x7fdbc0000000
>
> touch 2 pages with pagesz 0x40000000
>
> mmap 0x200000@0x7fdcbfe00000
>
> touch 1 pages with pagesz 0x200000
>
> really setup hugepage with:
>
> level 0 - lowmem 0x0, biosmem 0x200000, highmem 0x0
>
> level 1 - lowmem 0x80000000, biosmem 0x0, highmem 0x0
>
> vm_init_vdevs
>
> polling 38...
>
> Listening 38...
>
> pci init hostbridge
>
> pci init lpc
>
> pci init pci-gvt
>
> GVT: open /sys/kernel/gvt/control/create_gvt_instance failed
>
> GVT: init failed
>
> pci pci-gvt init failed
>
> mngr_client_new: Failed to accept from fd 38, err: Invalid argument
>
> Stop listening 38...
>
> Stop polling 38...
>
> Unable to init vdev (2)
>
> I've used the regular ACRN to test it (without reverted PCIe-related commits). I might have missed some required step in
> configuring ACRN or UOS, I'll re-check what I have done.
>
> Best regards,
>
> Dubravko
>
> *Dubravko Moravski*
>
> /SW engineering/
>
> *Exor Embedded S.r.l.*
>
> p:
>
>
>
> +38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
>
> a:
>
>
>
> Slavonska avenija, 50, Zagreb, Croatia, 10000
>
> w:
>
>
>
> exorint.com <https://exorint.com/>
>
>
>
> Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
>
> Privacy <https://www.exorint.com/it/privacy>
>
> --------------------------------------------------------------------------------------------------------------------------
>
> *From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> on behalf of Shuo A Liu via Lists.Projectacrn.Org
> <shuo.a.liu=intel.com@... <mailto:shuo.a.liu=intel.com@...>>
> *Sent:* Friday, February 28, 2020 9:41 AM
> *To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>>
> *Cc:* Wang, Hongbo <hongbo.wang@... <mailto:hongbo.wang@...>>; Wang, Yu1 <yu1.wang@...
> <mailto:yu1.wang@...>>; Li, Fei1 <fei1.li@... <mailto:fei1.li@...>>; Chen, Conghui
> <conghui.chen@... <mailto:conghui.chen@...>>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Dubravko,
>
> I learned from Fei that one of his patch(merged, for security) might impact PCIe devices in SOS, could you please help
> check if your HV include it?
>
> commit 65ed6c3529de8b3f3d890e95a7d816afba7bf379
>
> Author: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> Date: Thu Dec 5 22:51:06 2019 +0800
>
> hv: vpci: trap PCIe ECAM access for SOS
>
> SOS will use PCIe ECAM access PCIe external configuration space. HV should trap this
>
> access for security(Now pre-launched VM doesn't want to support PCI ECAM; post-launched
>
> VM trap PCIe ECAM access in DM).
>
> Besides, update PCIe MMCONFIG region to be owned by hypervisor and expose and pass through
>
> platform hide PCI devices by BIOS to SOS.
>
> Tracked-On: #3475
>
> Signed-off-by: Li Fei1 <fei1.li@... <mailto:fei1.li@...>>
>
> You can have a quick try to revert it firstly if it is included.
>
> Thanks
>
> shuo
>
> *From:* acrn-users@... <mailto:acrn-users@...> <acrn-users@...
> <mailto:acrn-users@...>> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
> *Sent:* Thursday, February 27, 2020 02:19
> *To:* acrn-users@... <mailto:acrn-users@...>
> *Subject:* Re: [acrn-users] Getting ACRN to work
>
> Hi Zide,
>
> Thank you for explaining all the kernel and Ubuntu options. It looks like all the use cases we are interested in are covered.
>
> I've got a replacement board and I'm continuing with ACRN. I've followed your instructions regarding rebuilding the
> kernel, in which I've enabled the Intel igb network driver we need for the I211 chip.
> The new kernel mostly works, however igb driver doesn't (and we really need the network):
>
> clear@clr-sos-guest~/work/acrn-kernel $ cat ../../dmesg_bad.txt | grep igb
>
> [ 3.708554] calling igb_init_module+0x0/0x4d @ 1
>
> [ 3.709199] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
>
> [ 3.710159] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 3.802746] igb 0000:04:00.0 0000:04:00.0 (uninitialized): *PCIe link lost*
>
> [ 4.120516] igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
>
> [ 7.504456] igb 0000:04:00.0: Invalid MAC Address
>
> [ 7.505779] igb: probe of 0000:04:00.0 failed with error -5
>
> [ 7.507585] initcall igb_init_module+0x0/0x4d returned 0 after 3709354 usecs
>
> The "PCIe link lost" message originates in igb_main.c, approx line 740:
>
> u32 igb_rd32(struct e1000_hw *hw, u32 reg)
>
> {
>
> struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
>
> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr);
>
> u32 value = 0;
>
> if (E1000_REMOVED(hw_addr))
>
> return ~value;
>
> value = readl(&hw_addr[reg]);
>
> /* reads should not return all F's */
>
> if (!(~value) && (!reg || !(~readl(hw_addr)))) {
>
> struct net_device *netdev = igb->netdev;
>
> hw->hw_addr = NULL;
>
> netdev_err(netdev, "*PCIe link lost*\n");
>
> }
>
> return value;
>
> }
>
> I think readl() for PCIe devices is basically a single PCIe transfer; in other words it's already the lowest possible
> level, looking from the software side of things, so it doesn't look like I could do much debugging here.
>
> In regular Clear Linux, the driver consistently works and there are never any link issues:
>
> clear@clr-sos-guest~ $ sudo dmesg | grep -i igb
>
> Password:
>
> [ 4.473282] calling igb_init_module+0x0/0x1000 [igb] @ 317
>
> [ 4.473306] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
>
> [ 4.473307] igb: Copyright (c) 2007-2014 Intel Corporation.
>
> [ 4.519304] igb 0000:04:00.0: added PHC on eth0
>
> [ 4.519307] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
>
> [ 4.519310] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:30:d8:05:55:84
>
> [ 4.519314] igb 0000:04:00.0: eth0: PBA No: FFFFFF-0FF
>
> [ 4.519316] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
>
> [ 4.519418] initcall igb_init_module+0x0/0x1000 [igb] returned 0 after 33714 usecs
>
> [ 5.390826] igb 0000:04:00.0 enp4s0: renamed from eth0
>
> [ 8.766862] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>
> So is there some setting in ACRN that can affect PCIe communication, that I need to adjust for (external) PCIe devices?
>
> Also with my ACRN-kernel, the mouse works but the cursor is invisible. In regular Clear Linux with 'native' kernel, with
> the same file system and GUI and system settings, the mouse works and the cursor is visible.
>
> Best regards,
>
> Dubravko
>
>
|
|
|
