Re: Setting up ACRN on a new board with Apollo Lake


Yin, Fengwei <fengwei.yin@...>
 

Hi Dubravko,
The UP2 board uses realtek network controller. I believe the reason
that your network can't work is the network controller is not enabled
in SOS kernel. Could you please find out the network controller for
your board? Or share the output of "lspci -k" with/without hv. Thanks.

Regards
Yin, Fengwei

On 2019/9/16 上午10:48, Yin, Fengwei wrote:
Hi Dubravko,
On 2019/9/13 上午1:01, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi Geoffroy,

I have already tried with uart=disabled, and repeated the test now; everything remains the same, all the issues are present.

SSH works when not running in the hypervisor.

I'm using the default "clear" user from projectacrn.github.io tutorial.

When network doesn't work, I'm not even getting the IP address (we have a DHCP server in our LAN, it's otherwise working fine).
I only see following message from your sos kernel log:
[    1.200862] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[    1.201695] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
And there is no interrupt for network chip in interrupt dump.
On my MRB (I don't have UP2 env setup. Will check UP2 info later):
I could see following interrupt info:
 126:         40          0          0          0   PCI-MSI 245760-edge      mei_me
 127:          0          0          0          0   PCI-MSI 294912-edge      ahci[0000:00:12.0]
 128:          0          1          0          0   PCI-MSI 1048576-edge      ens4
 129:          0          0      38006          0   PCI-MSI 1048577-edge      ens4-TxRx-0
 130:          0          0          0      17499   PCI-MSI 1048578-edge      ens4-TxRx-1
 131:      15357          0          0          0   PCI-MSI 1048579-edge      ens4-TxRx-2
 132:          0      32620          0          0   PCI-MSI 1048580-edge      ens4-TxRx-3
 133:          0          0          0          0   PCI-MSI 344064-edge      xhci_hcd
 135:          0          0          0          4   PCI-MSI 49152-edge      intel-ipu4
And following info in kernel log:
[    0.789202] e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
[    0.789203] e100: Copyright(c) 1999-2006 Intel Corporation
[    0.789237] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    0.789238] igb: Copyright (c) 2007-2014 Intel Corporation.
[    0.818600] pps pps0: new PPS source ptp0
[    0.818607] igb 0000:02:00.0: added PHC on eth0
[    0.818609] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[    0.818612] igb 0000:02:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 98:4f:ee:13:ce:43
[    0.818653] igb 0000:02:00.0: eth0: PBA No: 000300-000
[    0.818655] igb 0000:02:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    0.818711] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[    0.818713] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
Can you also share the kernel/interrupt dump when there is no HV?
Thanks.
Regards
Yin, Fengwei


I have uploaded new dmesg and ifconfig logs to https://www.amazon.com/clouddrive/share/3wZ8oRCroPr9wpfcl60XAJov69Do8Hg5RiX1GRRpYRI, 76 kB RAR archive:

  * log_hyper.log files are with hypervisor running (resulting in all
    the problems, including no network)
  * 'hyper kernel no hyper' files are with regular non-hypervisor boot,
    plus the same kernel that hypervisor is using
    (lts2018-sos-4.19.64-76) - less problems e.g. terminal works, but
    still no network
  * 'native kernel' files are with regular non-hypervisor boot, and
    native-something kernel - no problems and network works; this is the
    combination I'm using when SSH works

The date is always off by three days, it doesn't seem to affect anything: with hypervisor there are problems, without hypervisor there aren't. I'll fix it, just in case.

It looks like maybe the hypervisor is actually working fine all the time, but the kernel it is loading is somehow misconfigured so it doesn't initialize network interface(s) and user account(s)?

Best regards,
Dubravko


------------------------------------------------------------------------
*From:* acrn-users@... <acrn-users@...> on behalf of Geoffroy Van Cutsem via Lists.Projectacrn.Org <geoffroy.vancutsem=intel.com@...>
*Sent:* Thursday, September 12, 2019 5:46 PM
*To:* acrn-users@... <acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi Dubravko,

*From:*acrn-users@... <acrn-users@...> *On Behalf Of *Dubravko Moravski | Exor Embedded S.r.l.
*Sent:* Thursday, September 12, 2019 5:15 PM
*To:* acrn-users@...
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi everyone,

SOS console/terminal can't be opened at all; the window actually opens but it says "There was an error creating a child process for this terminal. Failed to open PTY: Permission denied" and we can't do anything. It is the same if we use launcher on the left side on the screen, or ctrl+alt+t. Also ssh doesn't work (as well as networking in general) so we can't connect remotely either.

[Geoffroy] A few questions and one suggestion:

* Have you tried to start the ACRN hypervisor with “uart=disabled”? (see https://projectacrn.github.io/latest/getting-started/apl-nuc.html#add-the-acrn-hypervisor-to-the-efi-partition). A quick test if you have an EFI shell on your platform is to run “acrn.efi uart=disabled” from it.

* Does SSH work when you are not running in the hypervisor?

* Which user are you using when trying to connect over SSH? (note that root is blocked by default)

* When you say “networking is not working in general”, does that mean you are not even getting an IP address?

200K JPG image uploaded to https://www.amazon.com/clouddrive/share/lWcIh67jPu7uhWrOzTBlOfcVwkfZ4ofYyEYTenbJrjo (ignore the date, I don't know why it's three days in the past).

[Geoffroy] Is the date always off by three days (regardless of whether you’re running in the hypervisor or not)? I don’t know if that is the cause but dates in the past can cause problems with certificates (and hence something to look at as a potential issue related to SSH).

I don't see why the console would have anything to do with the hypervisor, but simply it doesn't work with it, and works without it, so it's a symptom of some problem.

These all work when not running with the hypervisor.

Regarding ACPI and LPC, unfortunately we are not that familiar with ACPI, I have uploaded our disassembled ACPI dump to https://www.amazon.com/clouddrive/share/cQik3C5S0ey40L9L2EtMumII322kV0rjmJ34dBpHjD3, 35 kB RAR file. String "LPC" occurs in dsdt, ssdt1, 3 and 4.

Best regards,

Dubravko

------------------------------------------------------------------------

*From:*acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>> on behalf of Yin, Fengwei via Lists.Projectacrn.Org <fengwei.yin=intel.com@... <mailto:fengwei.yin=intel.com@...>>
*Sent:* Thursday, September 12, 2019 3:21 PM
*To:* acrn-users@... <mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi Dubravko,


*Dubravko Moravski*

/SW engineering/

*Exor Embedded S.r.l.*

p:



+38 512455659 <tel:+38%20512455659>  m: +38 5915402413 <tel:+38%205915402413>

a:



Slavonska avenija, 50, Zagreb, Croatia, 10000

w:



exorint.com <https://exorint.com/>



  Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.

Privacy <https://www.exorint.com/it/privacy>


*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:     +38 512455659 <tel:+38 512455659>  m:+38 5915402413 <tel:+38 5915402413>
a:     Slavonska avenija, 50, Zagreb, Croatia, 10000
w:     exorint.com <https://exorint.com/>




  Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.

Privacy <https://www.exorint.com/it/privacy>

On 9/12/2019 7:32 PM, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hi,

We haven't observed any issues on the UP2 either, except one, screen updates been erratic and mouse pointer jumping around.

We've just followed instructions from https://projectacrn.github.io/latest/getting-started/apl-nuc.html,
automated setup with some manual fine tuning.
I guess since we are using the same software and UP2 works (mostly) fine, the causes of our problems must be either different hardware or BIOS. But the trouble is figuring out what exactly.

All mentioned issues are for the Service OS. We can't even install/launch Guest OS because the Terminal no longer works after we restart the board with ACRN.
We checked the log you shared and didn't see anything unusual. We had
one issue which could make serial console use polling mode. I suspected
you hit that issue. But from the interrupt info, everything is fine (
there will be irq flooding if we hit that issue).We need more info.

When you say "Terminal no longer work", do you mean the SOS console
can't work any more? Or it could work for a while and then can't work?

Regards
Yin, Fengwei


Now that the memory log issue is fixed, could we use that to get additional debug information? If yes, how can I save that information for later? (We managed to create some scripts so we can save data like dmesg even when terminal can't be invoked.)

Best regards,
Dubravko


------------------------------------------------------------------------
*From:* acrn-users@... <mailto:acrn-users@...>
<acrn-users@...
<mailto:acrn-users@...>> on behalf of Liu, Fuzhong via
Lists.Projectacrn.Org <fuzhong.liu=intel.com@...
<mailto:fuzhong.liu=intel.com@...>>
*Sent:* Thursday, September 12, 2019 11:47 AM
*To:* acrn-users@...
<mailto:acrn-users@...> <acrn-users@... <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

No observed the issue on the UP2 board in our lab.

Need detail steps to reproduce.

BR.

Fuzhong

*From:*acrn-users@... [mailto:acrn-users@...] *On Behalf Of *Geoffroy Van Cutsem
*Sent:* Thursday, September 12, 2019 4:42 PM
*To:* acrn-users@... <mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Have you observed the same problems on the UP2 board?

Thanks,

Geoffroy

*From:*acrn-users@... <mailto:acrn-users@...>
<acrn-users@...
<mailto:acrn-users@...%20%0b>> <mailto:acrn-users@...>> *On Behalf Of *Liu, Fuzhong
*Sent:* Thursday, September 12, 2019 7:15 AM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi Dubravko

All issues you mentioned  are for Service OS, right?

Clear Linux image newer than 30900 fixed “memmap=2M$0x1FE00000” issue

BR.

Fuzhong

*From:*acrn-users@... <mailto:acrn-users@...>
[mailto:acrn-users@...] *On Behalf Of *Dubravko
Moravski | Exor Embedded S.r.l.
*Sent:* Wednesday, September 11, 2019 8:01 PM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi Yin Fenghwei,

Thank you for replying.

ACRN git commit hash is 555a03db99621c6a9d3bea0933c65db9dee44c46. I am getting identical behavior with ACRN that I've compiled, as well as with ACRN installed by Clear Linux (version id 30770).

I'm not sure if this mailing lists allows attachments, so I've uploaded the requested files to https://www.amazon.com/clouddrive/share/V7pJPEUKnBoDd3vkpM0ca4YLEKgNisSUkQdR03mr4sY(17

kB zip file, containing 4 text files). The zip archive contains:

   * dmesg output from SOS
   * /proc/interrupts from SOS
   * loader.conf from EFI partition
   * acrn.conf (here I've verified that root UUID is correct, and I've
     had to delete settings related to hypervisor memory log because
     otherwise it crashed)

Our board has 8 GB of memory and 64 GB MMC, and is similar to Up Squared and Leaf Hill CRB.

Best regards,

Dubravko

------------------------------------------------------------------------

*From:*acrn-users@... <mailto:acrn-users@...><acrn-users@...
<mailto:acrn-users@...%20%0b>> <mailto:acrn-users@...>> on behalf of Yin, Fengwei via
Lists.Projectacrn.Org <fengwei.yin=intel.com@...
<mailto:fengwei.yin=intel.com@...%20%0b>> <mailto:fengwei.yin=intel.com@...>>
*Sent:* Wednesday, September 11, 2019 4:41 AM
*To:* acrn-users@... <mailto:acrn-users@...>
<mailto:acrn-users@...><acrn-users@...
<mailto:acrn-users@...%20%0b>> <mailto:acrn-users@...>>
*Subject:* Re: [acrn-users] Setting up ACRN on a new board with Apollo Lake

Hi,

On 2019/9/10 下午9:00, Dubravko Moravski | Exor Embedded S.r.l. wrote:
Hello everyone,

I work for Exor Embedded and we are developing a new board with an Apollo Lake E3940 CPU. We would like to run ACRN on it. Indeed, following the instructions on https://projectacrn.github.io/latest/index.htmlI've managed to get it
running, dmesg among other things says:

Hypervisor detected: ACRN
...
ACRNTrace: Initialized acrn trace module with 4 cpu
...
systemd[1]: Detected virtualization acrn.

However, it's not really usable. We have following issues:

   * mouse cursor jumps around in a weird way, like there is some problem
     with switching video buffers
   * we can't open the Terminal in any way, there's always a message:
     "There was an error creating a child process for this terminal.
     Failed to open PTY: Permission denied"
   * network doesn't work
   * everything works much slower than usual

When not running with hypervisor, everything works.
We've noticed some differences between dmesgs when running the same kernel with and without hypervisor, but so far we were unable to figure out if any of these differences is significant.

We would appreciate if you have any hints for us, either what is exactly our problem, or at least how we should continue to debug the system.
Could you share:
     1. ACRN last git commit hash
     2. dmesg output in your SOS
     3. the output of cat /proc/interrupt in your SOS

Regards
Yin, Fengwei


Best regards,
Dubravko Moravski



*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:     +38 512455659 <tel:+38 512455659 <tel:+38%20512455659
<tel:+38%20512455659%20%3ctel:+38%20512455659>>>  m:+38 5915402413 <tel:+38
<tel:+38%20%0b>> 5915402413 <tel:+38%205915402413>>
a:     Slavonska avenija, 50, Zagreb, Croatia, 10000
w:     exorint.com <https://exorint.com/>



   Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.

Privacy <https://www.exorint.com/it/privacy>
*Dubravko Moravski*

/SW engineering/

*Exor Embedded S.r.l.*

p:


+38 512455659 <tel:+38%20512455659>  m: +38 5915402413
<tel:+38%205915402413>

a:


Slavonska avenija, 50, Zagreb, Croatia, 10000

w:


exorint.com <https://exorint.com/>


imap://fyin@...:993/fetch%3EUID%3E/INBOX%3E265992?header=quotebody&part=1.4&filename=x-sign_alone_89a12ff2-fa07-4acd-b9a0-8e9a8c2b3e4c.jpg

imap://fyin@...:993/fetch%3EUID%3E/INBOX%3E265992?header=quotebody&part=1.5&filename=environment-verde_4010c4a3-6fcf-4e6d-8a58-167a2248b283.jpg Prima
di stampare pensa ai costi ambientali. Please consider the environment before printing this email.

Privacy <https://www.exorint.com/it/privacy>


*Dubravko Moravski*
/SW engineering/
*Exor Embedded S.r.l.*
p:     +38 512455659 <tel:+38 512455659 <tel:+38%20512455659>>  m:+38 5915402413 <tel:+38
5915402413 <tel:+38%205915402413>>
a:     Slavonska avenija, 50, Zagreb, Croatia, 10000
w:     exorint.com <https://exorint.com/>



   Prima di stampare pensa ai costi ambientali. Please consider the environment before printing this email.

Privacy <https://www.exorint.com/it/privacy>


Join acrn-users@lists.projectacrn.org to automatically receive all group messages.