Yin, Fengwei <fengwei.yin@...>
Hi Dubravko,
I would like to check: What's the latest status for your new board
enabling?
Regards
Yin, Fengwei
toggle quoted message
Show quoted text
On 2019/9/29 上午11:23, Yin, Fengwei wrote: Hi Dubravko,
On 2019/9/28 上午12:58, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi everyone,
We've been busy bringing up some new hardware so it took me some time to respond. We are very grateful for all the assistance.
I have uploaded the requested new logs to https://www.amazon.com/clouddrive/share/fu8tJayVkkSkFZdfbVj8v7t32SZwsC7Cw9KNjLVMzdO
Our board has an Intel I211 network controller. lspci_hv log is with ACRN running, lspci_nohv is without ACRN.
Associated kernel and interrupt logs are included too.
We could download, configure and build our own version of the Clear Linux kernel to add the network controller, just I was unable to find instructions what specifically we need to set so it works with ACRN? It looks to me that it uses a different kernel than the plain Clear Linux. For SOS kernel, I suppose you could build it based on our acrn-kernel
with igb enabled.
Regarding serial port interrupts, I must admit I'm confused with all the port and interrupt settings; anyway I've followed the instructions and enabled console=ttyS2 (acrn log), and requested device 0.18.2/HSUART2 (efibootmgr log) because it's the only serial port on our board that "just works" (others need setting up some GPIOs first). With these settings, Clear Linux runs noticeably slow, something is obviously wrong.
If it simplifies things, we don't strictly need debug printouts from ACRN, we just want it to work.
It looks like the serial port is a little bit mess.
From dmesg_nohv.log, we could see following serial port:
[ 7.097307] printk: console [ttyS0] disabled
[ 7.097347] dw-apb-uart.5: ttyS0 at MMIO 0x91328000 (irq = 4, base_baud = 115200) is a 16550A
[ 7.097379] printk: console [ttyS0] enabled
[ 7.099613] probe of dw-apb-uart.5 returned 1 after 6946 usecs
[ 7.111867] dw-apb-uart.6: ttyS1 at MMIO 0x91326000 (irq = 5, base_baud = 115200) is a 16550A
[ 7.112334] probe of dw-apb-uart.6 returned 1 after 1479 usecs
[ 7.126345] dw-apb-uart.7: ttyS2 at MMIO 0x91324000 (irq = 6, base_baud = 115200) is a 16550A
We could see ttyS0 is used as kernel console. This could be confirmed
by interrupt_nohv.log as well:
4: 117 0 0 0 IR-IO-APIC 4-fasteoi idma64.5, ttyS0
But, you said the ttyS2 is only serial port on the board. Kind of
confuse here. So are you using ttyS2 or ttyS0 as kernel console without
HV?
With HV, from efibootmgr.txt, we could see you are 0.18.2 as HV console
which should be mapped to ttyS2. So I suppose you could get workable HV
console (let me know if I am wrong).
But you have wrong console defined for SOS. Let me step back and explain
more. With HV, the ttyS2 is hold in HV and HV emulates a virtual serial
port to SOS. From dmesg_hv.log, we could see:
[ 0.985360] serial8250: ttyS0 at I/O 0x3f8 (irq = 6, base_baud = 115200) is a 16550A
This is emulated serial port IO base = 0x3f8 and irq = 6. So I suppose
you should change the SOS kernel cmd line to use ttyS0 instead of ttyS2
for SOS. Again from dmesg_hv.log, we could see you still use ttyS2 as
sos kernel console (which is NOT correct):
[ 0.000000] Command line: console=tty0 console=ttyS2 root=PARTUUID=1bbb889b-e284-4d16-a1d7-79ef02ed69a3 rw rootwait ignore_loglevel no_timer_check consoleblank=0 i915.nuclear_pageflip=1 i915.avail_planes_per_pipe=0x01010F i915.domain_plane_owners=0x011111110000 i915.enable_gvt=1 i915.enable_guc=0
So could you please change the SOS console from ttyS2 to ttyS0? Suppose
you should change acrn.conf:
console=ttyS2 -> console=ttyS0
Another strange thing is:
From dmesg_hv.log, we could see following message:
[ 1.063019] dw-apb-uart.5: ttyS1 at MMIO 0x91328000 (irq = 4, base_baud = 115200) is a 16550A
[ 1.065287] idma64 idma64.6: Found Intel integrated DMA 64-bit
[ 1.066333] console [ttyS2] disabled
[ 1.066670] dw-apb-uart.6: ttyS2 at MMIO 0x91326000 (irq = 5, base_baud = 115200) is a 16550A
[ 1.314086] mmcblk1: p1 p2 p3
[ 1.666493] console [ttyS2] enabled
[ 1.770323] rpmb_mux:rpmb_key_get: Failed to get boot_params from the command line!
[ 2.036999] dw-apb-uart.7: ttyS3 at MMIO 0x91324000 (irq = 6, base_baud = 115200) is a 16550A
This ttyS3 should match the ttyS2 without HV (ttyS0 of SOS is emulated
serial port). Suppose if this device is hold by HV, we should hide it
from SOS. I will check how this happen.
In summary, could you please try:
1. enable igb driver in your sos kernel to see whether the network could
work or not.
2. Change the sos kernel console from ttyS2 to ttyS0
I will check why the ttyS3 is shown in SOS.
Regards
Yin, Fengwei
Other configuration files I've mentioned earlier are Clear-linux-iot...conf and Clear-linux-native...conf. They must have been provided by Clear Linux and ACRN installations. I don't have a problem with them, just I've wanted to be sure, since they are so different from acrn.conf, is acrn.conf missing something?
Regarding IPv6 stuff, I've included UP2 ifconfig log. This is with ACRN running; network and everything else works fine here.
All the tests/logs, except for the UP2, were made yesterday and today using newest Clear Linux and newest ACRN, rebuilt in release mode, with SOS_COM1_IRQ changed to 6 as instructed. All our issues are still present.
Best regards,
Dubravko
------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Yin, Fengwei via Lists.Projectacrn.Org <fengwei.yin=intel.com@...>
*Sent:* Monday, September 16, 2019 7:42 AM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi Dubravko,
The UP2 board uses realtek network controller. I believe the reason
that your network can't work is the network controller is not enabled
in SOS kernel. Could you please find out the network controller for
your board? Or share the output of "lspci -k" with/without hv. Thanks.
Regards
Yin, Fengwei
On 2019/9/16 上午10:48, Yin, Fengwei wrote:
Hi Dubravko,
On 2019/9/13 上午1:01, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi Geoffroy,
I have already tried with uart=disabled, and repeated the test now; everything remains the same, all the issues are present.
SSH works when not running in the hypervisor.
I'm using the default "clear" user from projectacrn.github.io tutorial.
When network doesn't work, I'm not even getting the IP address (we have a DHCP server in our LAN, it's otherwise working fine). I only see following message from your sos kernel log:
[ 1.200862] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 1.201695] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
And there is no interrupt for network chip in interrupt dump.
On my MRB (I don't have UP2 env setup. Will check UP2 info later):
I could see following interrupt info:
126: 40 0 0 0 PCI-MSI 245760-edge mei_me
127: 0 0 0 0 PCI-MSI 294912-edge ahci[0000:00:12.0]
128: 0 1 0 0 PCI-MSI 1048576-edge ens4
129: 0 0 38006 0 PCI-MSI 1048577-edge ens4-TxRx-0
130: 0 0 0 17499 PCI-MSI 1048578-edge ens4-TxRx-1
131: 15357 0 0 0 PCI-MSI 1048579-edge ens4-TxRx-2
132: 0 32620 0 0 PCI-MSI 1048580-edge ens4-TxRx-3
133: 0 0 0 0 PCI-MSI 344064-edge xhci_hcd
135: 0 0 0 4 PCI-MSI 49152-edge intel-ipu4
And following info in kernel log:
[ 0.789202] e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
[ 0.789203] e100: Copyright(c) 1999-2006 Intel Corporation
[ 0.789237] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 0.789238] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 0.818600] pps pps0: new PPS source ptp0
[ 0.818607] igb 0000:02:00.0: added PHC on eth0
[ 0.818609] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 0.818612] igb 0000:02:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 98:4f:ee:13:ce:43
[ 0.818653] igb 0000:02:00.0: eth0: PBA No: 000300-000
[ 0.818655] igb 0000:02:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 0.818711] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 0.818713] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
Can you also share the kernel/interrupt dump when there is no HV?
Thanks.
Regards
Yin, Fengwei
I have uploaded new dmesg and ifconfig logs to https://www.amazon.com/clouddrive/share/3wZ8oRCroPr9wpfcl60XAJov69Do8Hg5RiX1GRRpYRI,
76 kB RAR archive:
* log_hyper.log files are with hypervisor running (resulting in all
the problems, including no network)
* 'hyper kernel no hyper' files are with regular non-hypervisor boot,
plus the same kernel that hypervisor is using
(lts2018-sos-4.19.64-76) - less problems e.g. terminal works, but
still no network
* 'native kernel' files are with regular non-hypervisor boot, and
native-something kernel - no problems and network works; this is the
combination I'm using when SSH works
The date is always off by three days, it doesn't seem to affect anything: with hypervisor there are problems, without hypervisor there aren't. I'll fix it, just in case.
It looks like maybe the hypervisor is actually working fine all the time, but the kernel it is loading is somehow misconfigured so it doesn't initialize network interface(s) and user account(s)?
Best regards,
Dubravko
------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Geoffroy Van Cutsem via Lists.Projectacrn.Org <geoffroy.vancutsem=intel.com@...>
*Sent:* Thursday, September 12, 2019 5:46 PM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi Dubravko,
*From:*acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Thursday, September 12, 2019 5:15 PM
*To:* acrn-users@...
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi everyone,
SOS console/terminal can't be opened at all; the window actually opens but it says "There was an error creating a child process for this terminal. Failed to open PTY: Permission denied" and we can't do anything. It is the same if we use launcher on the left side on the screen, or ctrl+alt+t. Also ssh doesn't work (as well as networking in general) so we can't connect remotely either.
[Geoffroy] A few questions and one suggestion:
* Have you tried to start the ACRN hypervisor with “uart=disabled”? (see https://projectacrn.github.io/latest/getting-started/apl-nuc.html#add-the-acrn-hypervisor-to-the-efi-partition).
A quick test if you have an EFI shell on your platform is to run “acrn.efi uart=disabled” from it.
* Does SSH work when you are not running in the hypervisor?
* Which user are you using when trying to connect over SSH? (note that root is blocked by default)
* When you say “networking is not working in general”, does that mean you are not even getting an IP address?
200K JPG image uploaded to https://www.amazon.com/clouddrive/share/lWcIh67jPu7uhWrOzTBlOfcVwkfZ4ofYyEYTenbJrjo
(ignore the date, I don't know why it's three days in the past).
[Geoffroy] Is the date always off by three days (regardless of whether you’re running in the hypervisor or not)? I don’t know if that is the cause but dates in the past can cause problems with certificates (and hence something to look at as a potential issue related to SSH).
I don't see why the console would have anything to do with the hypervisor, but simply it doesn't work with it, and works without it, so it's a symptom of some problem.
These all work when not running with the hypervisor.
Regarding ACPI and LPC, unfortunately we are not that familiar with ACPI, I have uploaded our disassembled ACPI dump to https://www.amazon.com/clouddrive/share/cQik3C5S0ey40L9L2EtMumII322kV0rjmJ34dBpHjD3,
35 kB RAR file. String "LPC" occurs in dsdt, ssdt1, 3 and 4.
Best regards,
Dubravko
------------------------------------------------------------------------
*From:*acrn-users@... <mailto:acrn-users@...>
<acrn-users@... <mailto:acrn-users@...>> on behalf of Yin, Fengwei
via Lists.Projectacrn.Org <fengwei.yin=intel.com@... <mailto:fengwei.yin=intel.com@...>>
*Sent:* Thursday, September 12, 2019 3:21 PM
*To:* acrn-users@... <mailto:acrn-users@...>
<acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi Dubravko,
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:
+38 512455659 <tel:+38%20512455659> m: +38 5915402413 <tel:+38%205915402413>
a:
Slavonska avenija, 50, Zagreb, Croatia, 10000
w:
exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659> m:+38 5915402413 <tel:+38 5915402413>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
On 9/12/2019 7:32 PM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi,
We haven't observed any issues on the UP2 either, except one, screen updates been erratic and mouse pointer jumping around.
We've just followed instructions from https://projectacrn.github.io/latest/getting-started/apl-nuc.html,
automated setup with some manual fine tuning.
I guess since we are using the same software and UP2 works (mostly) fine, the causes of our problems must be either different hardware or BIOS. But the trouble is figuring out what exactly.
All mentioned issues are for the Service OS. We can't even install/launch Guest OS because the Terminal no longer works after we restart the board with ACRN. We checked the log you shared and didn't see anything unusual. We had
one issue which could make serial console use polling mode. I suspected
you hit that issue. But from the interrupt info, everything is fine (
there will be irq flooding if we hit that issue).We need more info.
When you say "Terminal no longer work", do you mean the SOS console
can't work any more? Or it could work for a while and then can't work?
Regards
Yin, Fengwei
Now that the memory log issue is fixed, could we use that to get additional debug information? If yes, how can I save that information for later? (We managed to create some scripts so we can save data like dmesg even when terminal can't be invoked.)
Best regards,
Dubravko
------------------------------------------------------------------------
*From:* acrn-users@... <mailto:acrn-users@...>
<acrn-users@...
<mailto:acrn-users@...>> on behalf of Liu, Fuzhong via
Lists.Projectacrn.Org <fuzhong.liu=intel.com@... <mailto:fuzhong.liu=intel.com@...>>
*Sent:* Thursday, September 12, 2019 11:47 AM
*To:* acrn-users@... <mailto:acrn-users@...>
<acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
No observed the issue on the UP2 board in our lab.
Need detail steps to reproduce.
BR.
Fuzhong
*From:*acrn-users@... [mailto:acrn-users@...] *On Behalf Of *Geoffroy Van
Cutsem
*Sent:* Thursday, September 12, 2019 4:42 PM
*To:* acrn-users@... <mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Have you observed the same problems on the UP2 board?
Thanks,
Geoffroy
*From:*acrn-users@... <mailto:acrn-users@...>
<acrn-users@... <mailto:acrn-users@...%20%0b>>
<mailto:acrn-users@...>> *On Behalf Of *Liu, Fuzhong
*Sent:* Thursday, September 12, 2019 7:15 AM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi Dubravko
All issues you mentioned are for Service OS, right?
Clear Linux image newer than 30900 fixed “memmap=2M$0x1FE00000” issue
BR.
Fuzhong
*From:*acrn-users@... <mailto:acrn-users@...>
[mailto:acrn-users@...] *On Behalf Of *Dubravko
Moravski | Exor Embedded S.r.l.
*Sent:* Wednesday, September 11, 2019 8:01 PM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi Yin Fenghwei,
Thank you for replying.
ACRN git commit hash is 555a03db99621c6a9d3bea0933c65db9dee44c46. I am getting identical behavior with ACRN that I've compiled, as well as with ACRN installed by Clear Linux (version id 30770).
I'm not sure if this mailing lists allows attachments, so I've uploaded the requested files to https://www.amazon.com/clouddrive/share/V7pJPEUKnBoDd3vkpM0ca4YLEKgNisSUkQdR03mr4sY(17
kB zip file, containing 4 text files). The zip archive contains:
* dmesg output from SOS
* /proc/interrupts from SOS
* loader.conf from EFI partition
* acrn.conf (here I've verified that root UUID is correct, and I've
had to delete settings related to hypervisor memory log because
otherwise it crashed)
Our board has 8 GB of memory and 64 GB MMC, and is similar to Up Squared and Leaf Hill CRB.
Best regards,
Dubravko
------------------------------------------------------------------------
*From:*acrn-users@... <mailto:acrn-users@...><acrn-users@...
<mailto:acrn-users@...%20%0b>>
<mailto:acrn-users@...>> on behalf of Yin, Fengwei via
Lists.Projectacrn.Org <fengwei.yin=intel.com@... <mailto:fengwei.yin=intel.com@...%20%0b>>
<mailto:fengwei.yin=intel.com@...>>
*Sent:* Wednesday, September 11, 2019 4:41 AM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...><acrn-users@...
<mailto:acrn-users@...%20%0b>>
<mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake
Hi,
On 2019/9/10 下午9:00, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hello everyone,
I work for Exor Embedded and we are developing a new board with an Apollo Lake E3940 CPU. We would like to run ACRN on it. Indeed, following the instructions on https://projectacrn.github.io/latest/index.htmlI've managed to get it
running, dmesg among other things says:
Hypervisor detected: ACRN
...
ACRNTrace: Initialized acrn trace module with 4 cpu
...
systemd[1]: Detected virtualization acrn.
However, it's not really usable. We have following issues:
* mouse cursor jumps around in a weird way, like there is some problem
with switching video buffers
* we can't open the Terminal in any way, there's always a message:
"There was an error creating a child process for this terminal.
Failed to open PTY: Permission denied"
* network doesn't work
* everything works much slower than usual
When not running with hypervisor, everything works.
We've noticed some differences between dmesgs when running the same kernel with and without hypervisor, but so far we were unable to figure out if any of these differences is significant.
We would appreciate if you have any hints for us, either what is exactly our problem, or at least how we should continue to debug the system. Could you share:
1. ACRN last git commit hash
2. dmesg output in your SOS
3. the output of cat /proc/interrupt in your SOS
Regards
Yin, Fengwei
Best regards,
Dubravko Moravski
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659 <tel:+38%20512455659
<tel:+38%20512455659%20%3ctel:+38%20512455659>>> m:+38 5915402413 <tel:+38
<tel:+38%20%0b>> 5915402413 <tel:+38%205915402413>>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:
+38 512455659 <tel:+38%20512455659> m: +38 5915402413
<tel:+38%205915402413>
a:
Slavonska avenija, 50, Zagreb, Croatia, 10000
w:
exorint.com <https://exorint.com/>
imap://fyin@...:993/fetch%3EUID%3E/INBOX%3E265992?header=quotebody&part=1.4&filename=x-sign_alone_89a12ff2-fa07-4acd-b9a0-8e9a8c2b3e4c.jpg
imap://fyin@...:993/fetch%3EUID%3E/INBOX%3E265992?header=quotebody&part=1.5&filename=environment-verde_4010c4a3-6fcf-4e6d-8a58-167a2248b283.jpg Prima
di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p: +38 512455659 <tel:+38 512455659 <tel:+38%20512455659>> m:+38 5915402413 <tel:+38
5915402413 <tel:+38%205915402413>>
a: Slavonska avenija, 50, Zagreb, Croatia, 10000
w: exorint.com <https://exorint.com/>
Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.
Privacy <https://www.exorint.com/it/privacy>
|
