Hi Tom,

 

Great to know you’ve brought that up! We’ll take your suggestion into consideration the next time we refresh the guides about bringing up pre-launched VMs.

 

 

 

Hi Junjie,

 

thank you for sharing! I finally made it work. I don’t know exactly, but either the “-nounzip” parameter on the initrd or the kernel parameter “reboot=acpi” did the trick. I highly recommend to include such an example of a pre-launched VM into the documentation to avoid future questions.

 

Cheers! Tom

 

 

Hi Tom,

 

I verified that a pre-launched VM can enter the initrd shell successfully with the latest main branch (c7907f244 (HEAD -> master, origin/master, origin/HEAD) doc: add "Linux" and "ACRN" to grub display in GSG). The scenario XML I used is attached for your reference. The only notable change I made is adding “reboot=acpi” to the command line of the pre-launched VM kernel, which is required for ACRN config tools to generate ACPI_VM0.bin which is the vACPI table for that VM.

 

The grub entry looks like the following.

 

menuentry 'Ubuntu with ACRN hypervisor' --class ubuntu --class gnu-linux --class gnu --class os --class acrn $menuentry_id_option 'acrn-gnulinux-e9d0618b-ad63-45fb-b468-57ee33c65a6a' {

        insmod part_gpt

        insmod ext2

        if [ x$feature_platform_search_hint = xy ]; then

          search --no-floppy --fs-uuid --set=root  e9d0618b-ad63-45fb-b468-57ee33c65a6a

        else

          search --no-floppy --fs-uuid --set=root e9d0618b-ad63-45fb-b468-57ee33c65a6a

        fi

        echo    'Loading ACRN hypervisor 3.2 ...'

        multiboot2      /boot/acrn-3.2.bin cpu_perf_policy=Performance  root=UUID=e9d0618b-ad63-45fb-b468-57ee33c65a6a ro console=tty0 console=ttyS0,115200n8 quiet splash

        echo    'Loading ACRN Service VM Linux kernel 5.15.44-acrn-service-vm-dirty ...'

        module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty SOS_kernel

        echo    'Loading ACRN Service VM initial ramdisk ...'

        module2 --nounzip       /boot/initrd.img-5.15.44-acrn-service-vm-dirty

        echo    'Loading ACRN VM0 kernel for VM0'

        module2         /boot/vmlinuz-5.15.0-52-generic VM0_kernel

        echo    'Loading ACRN VM0 acpi for VM0'

        module2         /boot/ACPI_VM0.bin ACPI_VM0

        echo    'Loading ACRN VM0 ramdisk for VM0'

        module2 --nounzip       /boot/initrd.img-5.15.0-52-generic VM0_initrd

}

 

That is generated by the scripts installed by the grub-acrn package, after setting the following in /etc/default/grub.d/acrn.cfg:

 

export GRUB_ACRN_MOD_VM0_kernel="/boot/vmlinuz-5.15.0-52-generic"

export GRUB_ACRN_MOD_VM0_initrd="/boot/initrd.img-5.15.0-52-generic"

export GRUB_ACRN_MOD_SOS_kernel="/boot/vmlinuz-5.15.44-acrn-service-vm-dirty"

 

The pre-launched VM kernel is from Ubuntu 20.04 linux-image-5.15.0-52-generic package, but theoretically any recent kernel should work.

 

With that I can see the following dmesgs from the prelaunched VM:

 

[    0.186043] AppArmor: AppArmor sha1 policy hashing enabled

[    0.186116] ima: No TPM chip found, activating TPM-bypass!

[    0.186187] Loading compiled-in module X.509 certificates

[    0.186518] Loaded X.509 cert 'Build time autogenerated kernel key: 0f7b019bb693b03ef7a18ddcd1f94e1a1ee0f5d4'

[    0.186642] ima: Allocated hash algorithm: sha1

[    0.186704] ima: No architecture policies found

[    0.186767] evm: Initialising EVM extended attributes:

[    0.186833] evm: security.selinux

[    0.186876] evm: security.SMACK64

[    0.186919] evm: security.SMACK64EXEC

[    0.186966] evm: security.SMACK64TRANSMUTE

[    0.187018] evm: security.SMACK64MMAP

[    0.187066] evm: security.apparmor

[    0.187110] evm: security.ima

[    0.187148] evm: security.capability

[    0.187194] evm: HMAC attrs: 0x1

[    0.187341] PM:   Magic number: 7:930:718

[    0.187462] RAS: Correctable Errors collector initialized.

[    0.191840] Freeing unused decrypted memory: 2036K

[    0.192173] Freeing unused kernel image (initmem) memory: 3228K

[    0.198218] Write protecting the kernel read-only data: 30720k

[    0.198511] Freeing unused kernel image (text/rodata gap) memory: 2036K

[    0.198734] Freeing unused kernel image (rodata/data gap) memory: 1504K

[    0.203884] x86/mm: Checked W+X mappings: passed, no W+X pages found.

[    0.203967] Run /init as init process

[    0.204015]   with arguments:

[    0.204016]     /init

[    0.204016]   with environment:

[    0.204016]     HOME=/

[    0.204017]     TERM=linux

(initramfs) ls

dev      bin      init     lib64    sbin     var      tmp

root     conf     lib      libx32   scripts  sys

kernel   etc      lib32    run      usr      proc

(initramfs)

 

Hope that helps.

 

 

 

Hi Junjie,

 

thanks! Let me know how it goes please. If you get it to work I would appreciate if you can share your grub.cfg, scenario.xml etc. Thanks!

Tom

 

 

Hi Tom,

 

Let me try using INITRD shell in a pre-launched VM on my side first to check if that’s platform specific or not.

 

 

 

Hi Junjie,

 

thank you for helping me!

 

So I tried a few things:

 

1. Changed the VM settings from Realtime to Standard VM:
1. No change
2. Compiled the kernel w/ “CONFIG_SERIAL_8250_DETECT_IRQ=y” _AND_ removed BTRFs support (since I saw the hang shortly before the RAID6 code)
1. Now it hangs at a different stage:

[    0.247281] Key type asymmetric registered

[    0.247281] Asymmetric key parser 'x509' registered

[    0.247281] Key type pkcs7_test registered

[    0.247281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)

[    0.247281] io scheduler mq-deadline registered

[    0.247281] io scheduler kyber registered

[    0.247281] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled

 

Should I provide kernel & initrd for testing in your environment? Maybe its my ACRN build that’s problematic…?

 

Cheers, Tom

 

 

Hi Tom,

 

Would you please try changing the VM type of the pre-launched VM from “RTVM” to “Standard VM” and check if you can see the shell prompt from the initrd?

 

The hypervisor does not inject vUART interrupts to an RTVM, which could be why the shell prompt does not show up when the kernel is not built with CONFIG_SERIAL_8250_DETECT_IRQ=y.

 

 

 

Hi,

 

Quick update:

 

I have a kernel + initrd that boots into my NFSROOT properly when I am not using ACRN. All network drivers & nfs modules compiled in. As soon as I want to launch the kernel as a Preloaded VM, it hangs with the last line being:

 

[    0.315764] cryptd: max_cpu_qlen set to 1000

 

I can reproduce this reliable. Any way to debug this?

 

Kernel version is: 5.15.64 (ubuntu jammy latest source)

 

Thx, Tom

 

 

Hi Junjie,

 

thank you for the reply and the right pointers! Indeed the kernel couldn’t find my NVMe because I didn’t map it. However, since I have only one NVMe drive in my IPC I need some other boot media.

 

I tried using a kernel with built in nfsroot and IP autoconfig on kernel level but it hangs during boot inside ACRN:

 

[    0.304553] smpboot: Max logical packages: 1

[    0.304553] smpboot: Total of 1 processors activated (3763.20 BogoMIPS)

[    0.304553] devtmpfs: initialized

[    0.304553] ACPI: PM: Registering ACPI NVS region [mem 0x7ff00000-0x7fffffff] (1048576 bytes)

[    0.304553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns

[    0.304553] futex hash table entries: 256 (order: 2, 16384 bytes, linear)

[    0.304553] pinctrl core: initialized pinctrl subsystem

[    0.304553] PM: RTC time: 21:35:19, date: 2023-01-03

[    0.304553] NET: Registered PF_NETLINK/PF_ROUTE protocol family

[    0.304553] audit: initializing netlink subsys (disabled)

[    0.304553] thermal_sys: Registered thermal governor 'bang_bang'

[    0.304553] thermal_sys: Registered thermal governor 'step_wise'

[    0.304553] thermal_sys: Registered thermal governor 'user_space'

[    0.304553] cpuidle: using governor ladder

[    0.304553] cpuidle: using governor menu

[    0.304553] PCI: Using configuration type 1 for base access

[    0.304553] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'

[    0.304553] Kprobes globally optimized

[    0.304553] audit: type=2000 audit(1672781719.000:1): state=initialized audit_enabled=0 res=1

[    0.304553] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages

[    0.304553] cryptd: max_cpu_qlen set to 1000

 

 

I also have a hanging kernel when I try booting a stock Ubuntu 22.04 kernel with INITRD. In the scenario.xml I defined:

 

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

      <ramdisk_mod>RTVM_INITRD</ramdisk_mod>

    </os_config>

 

In grub.cfg:

 

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

    multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

    module2 /boot/vmlinuz-5.15.0-56-generic RTVM rw rootwait ip=dhcp root=/dev/nfs nfsrootdebug nfsroot=192.168.0.83:/netroot_rtvm console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072

    module2 /boot/initrd.img-5.15.0-56-generic RTVM_INITRD

    module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty Linux_bzImage

}

 

Once the kernel boots, I see on vm_console 0 the kernel being stuck:

[    2.579592] ima: Allocated hash algorithm: sha1

[    2.580065] ima: No architecture policies found

[    2.580566] evm: Initialising EVM extended attributes:

[    2.581092] evm: security.selinux

[    2.581438] evm: security.SMACK64

[    2.581797] evm: security.SMACK64EXEC

[    2.582177] evm: security.SMACK64TRANSMUTE

[    2.582597] evm: security.SMACK64MMAP

[    2.582985] evm: security.apparmor

[    2.583332] evm: security.ima

[    2.583637] evm: security.capability

[    2.584031] evm: HMAC attrs: 0x1

[    2.584950] PM:   Magic number: 7:898:549

[    2.585390] tty ttyS25: hash matches

[    2.585791] clockevents clockevent0: hash matches

[    2.586632] RAS: Correctable Errors collector initialized.

[    2.589445] Freeing unused decrypted memory: 2036K

[    2.590805] Freeing unused kernel image (initmem) memory: 3232K

 

No mention of a wrong /dev/nfs (because it hasn’t gotten the kernel module yet). Just hanging.

 

Any ideas?

 

Thanks! Tom

 

 

 

Hi Tom,

 

Some issues I noticed in your command line for the pre-launched VM kernel are:

 

1. root=/dev/nvme0n1p3, which is a path to your NVMe which is not accessible to the pre-launched VM. You may want to install your OS on the eMMC and passthrough the eMMC controller to the pre-launched VM.

 

2. quiet and loglevel=3, which prevents the kernel from printing early logs that greatly help you diagnose such boot issues. To enable the logs, remove “quiet” and replace “loglevel=3” with “loglevel=7”.

 

 

 

Hi,

 

I successfully installed ACRN and got the Service VM up and running. Now I want to create a pre-launched Linux RT VM that later runs some PLC logic.

 

ACRN 3.1

Platform: Intel Elkhart Atom x6425RE

Service OS: Ubuntu 22.04

RT VM OS: Ubuntu 22.04 (basically the Service OS installed again on a new partition)

 

Partition Layout is:

259        1    1100800 nvme0n1p1 -> UEFI Boot

259        2  104857600 nvme0n1p2 -> Service OS (100G)

259        3   10485760 nvme0n1p3 -> RTVM OS (10G)

 

I configured ACRN for a hybrid scenario (see attached board & scenario.xml). In an essence, when I boot my device I see the Service VM starting (and I can reach it via Network) however the RTVM isn’t starting. Using COM2 I can access the ACRN shell:

 

ACRN:\>vm_list

 

VM_ID VM_NAME                          VM_STATE

===== ================================ ========

  0   RTVM                             Running

  1   ACRN_Service_VM                  Running

ACRN:\>vm_console 0

 

----- Entering VM 0 Shell -----

 

Also the interrupts look pretty dead for VM0. What did I do wrong? Find below some more information my setup.

 

Thanks!

Tom

 

 

Grub.cfg is as follows:

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

module2 /boot/vmlinuz-5.15.0-56-generic RTVM

module2 /boot/vmlinuz-5.15.44-acrn-service-vm Linux_bzImage

}

 

Scenario.XML:

  <vm id="0">

    <vm_type>RTVM</vm_type>

    <console_vuart>COM Port 1</console_vuart>

    <memory>

      <size>2048</size>

    </memory>

    <cpu_affinity>

      <pcpu>

        <pcpu_id>3</pcpu_id>

        <real_time_vcpu>y</real_time_vcpu>

      </pcpu>

    </cpu_affinity>

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check quiet loglevel=3 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

    </os_config>

    <mmio_resources>

      <TPM2>n</TPM2>

      <p2sb>n</p2sb>

    </mmio_resources>

    <pci_devs>

      <pci_dev>02:00.0 Ethernet controller: Intel Corporation Ethernet Controller (2) I225-IT</pci_dev>

    </pci_devs>

    <lapic_passthrough>y</lapic_passthrough>

    <io_completion_polling>y</io_completion_polling>

    <virtual_cat_support>n</virtual_cat_support>

    <virtual_cat_number>0</virtual_cat_number>

    <epc_section>

      <base>0</base>

      <size>0</size>

    </epc_section>

    <load_order>PRE_LAUNCHED_VM</load_order>

    <name>RTVM</name>

    <priority>PRIO_LOW</priority>

    <companion_vmid>65535</companion_vmid>

  </vm>

 

 

CONFIDENTIALITY: The contents of this e-mail are confidential and intended only for the above addressee(s). If you are not the intended recipient, or the person responsible for delivering it to the intended recipient, copying or delivering it to anyone else or using it in any unauthorized manner is prohibited and may be unlawful. If you receive this e-mail by mistake, please notify the sender and the systems administrator at straymail@... immediately.

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

Hi Junjie,

 

thank you for sharing! I finally made it work. I don’t know exactly, but either the “-nounzip” parameter on the initrd or the kernel parameter “reboot=acpi” did the trick. I highly recommend to include such an example of a pre-launched VM into the documentation to avoid future questions.

 

Cheers! Tom

 

Hi Tom,

 

I verified that a pre-launched VM can enter the initrd shell successfully with the latest main branch (c7907f244 (HEAD -> master, origin/master, origin/HEAD) doc: add "Linux" and "ACRN" to grub display in GSG). The scenario XML I used is attached for your reference. The only notable change I made is adding “reboot=acpi” to the command line of the pre-launched VM kernel, which is required for ACRN config tools to generate ACPI_VM0.bin which is the vACPI table for that VM.

 

The grub entry looks like the following.

 

menuentry 'Ubuntu with ACRN hypervisor' --class ubuntu --class gnu-linux --class gnu --class os --class acrn $menuentry_id_option 'acrn-gnulinux-e9d0618b-ad63-45fb-b468-57ee33c65a6a' {

        insmod part_gpt

        insmod ext2

        if [ x$feature_platform_search_hint = xy ]; then

          search --no-floppy --fs-uuid --set=root  e9d0618b-ad63-45fb-b468-57ee33c65a6a

        else

          search --no-floppy --fs-uuid --set=root e9d0618b-ad63-45fb-b468-57ee33c65a6a

        fi

        echo    'Loading ACRN hypervisor 3.2 ...'

        multiboot2      /boot/acrn-3.2.bin cpu_perf_policy=Performance  root=UUID=e9d0618b-ad63-45fb-b468-57ee33c65a6a ro console=tty0 console=ttyS0,115200n8 quiet splash

        echo    'Loading ACRN Service VM Linux kernel 5.15.44-acrn-service-vm-dirty ...'

        module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty SOS_kernel

        echo    'Loading ACRN Service VM initial ramdisk ...'

        module2 --nounzip       /boot/initrd.img-5.15.44-acrn-service-vm-dirty

        echo    'Loading ACRN VM0 kernel for VM0'

        module2         /boot/vmlinuz-5.15.0-52-generic VM0_kernel

        echo    'Loading ACRN VM0 acpi for VM0'

        module2         /boot/ACPI_VM0.bin ACPI_VM0

        echo    'Loading ACRN VM0 ramdisk for VM0'

        module2 --nounzip       /boot/initrd.img-5.15.0-52-generic VM0_initrd

}

 

That is generated by the scripts installed by the grub-acrn package, after setting the following in /etc/default/grub.d/acrn.cfg:

 

export GRUB_ACRN_MOD_VM0_kernel="/boot/vmlinuz-5.15.0-52-generic"

export GRUB_ACRN_MOD_VM0_initrd="/boot/initrd.img-5.15.0-52-generic"

export GRUB_ACRN_MOD_SOS_kernel="/boot/vmlinuz-5.15.44-acrn-service-vm-dirty"

 

The pre-launched VM kernel is from Ubuntu 20.04 linux-image-5.15.0-52-generic package, but theoretically any recent kernel should work.

 

With that I can see the following dmesgs from the prelaunched VM:

 

[    0.186043] AppArmor: AppArmor sha1 policy hashing enabled

[    0.186116] ima: No TPM chip found, activating TPM-bypass!

[    0.186187] Loading compiled-in module X.509 certificates

[    0.186518] Loaded X.509 cert 'Build time autogenerated kernel key: 0f7b019bb693b03ef7a18ddcd1f94e1a1ee0f5d4'

[    0.186642] ima: Allocated hash algorithm: sha1

[    0.186704] ima: No architecture policies found

[    0.186767] evm: Initialising EVM extended attributes:

[    0.186833] evm: security.selinux

[    0.186876] evm: security.SMACK64

[    0.186919] evm: security.SMACK64EXEC

[    0.186966] evm: security.SMACK64TRANSMUTE

[    0.187018] evm: security.SMACK64MMAP

[    0.187066] evm: security.apparmor

[    0.187110] evm: security.ima

[    0.187148] evm: security.capability

[    0.187194] evm: HMAC attrs: 0x1

[    0.187341] PM:   Magic number: 7:930:718

[    0.187462] RAS: Correctable Errors collector initialized.

[    0.191840] Freeing unused decrypted memory: 2036K

[    0.192173] Freeing unused kernel image (initmem) memory: 3228K

[    0.198218] Write protecting the kernel read-only data: 30720k

[    0.198511] Freeing unused kernel image (text/rodata gap) memory: 2036K

[    0.198734] Freeing unused kernel image (rodata/data gap) memory: 1504K

[    0.203884] x86/mm: Checked W+X mappings: passed, no W+X pages found.

[    0.203967] Run /init as init process

[    0.204015]   with arguments:

[    0.204016]     /init

[    0.204016]   with environment:

[    0.204016]     HOME=/

[    0.204017]     TERM=linux

(initramfs) ls

dev      bin      init     lib64    sbin     var      tmp

root     conf     lib      libx32   scripts  sys

kernel   etc      lib32    run      usr      proc

(initramfs)

 

Hope that helps.

 

 

 

Hi Junjie,

 

thanks! Let me know how it goes please. If you get it to work I would appreciate if you can share your grub.cfg, scenario.xml etc. Thanks!

Tom

 

 

Hi Tom,

 

Let me try using INITRD shell in a pre-launched VM on my side first to check if that’s platform specific or not.

 

 

 

Hi Junjie,

 

thank you for helping me!

 

So I tried a few things:

 

1. Changed the VM settings from Realtime to Standard VM:
1. No change
2. Compiled the kernel w/ “CONFIG_SERIAL_8250_DETECT_IRQ=y” _AND_ removed BTRFs support (since I saw the hang shortly before the RAID6 code)
1. Now it hangs at a different stage:

[    0.247281] Key type asymmetric registered

[    0.247281] Asymmetric key parser 'x509' registered

[    0.247281] Key type pkcs7_test registered

[    0.247281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)

[    0.247281] io scheduler mq-deadline registered

[    0.247281] io scheduler kyber registered

[    0.247281] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled

 

Should I provide kernel & initrd for testing in your environment? Maybe its my ACRN build that’s problematic…?

 

Cheers, Tom

 

 

Hi Tom,

 

Would you please try changing the VM type of the pre-launched VM from “RTVM” to “Standard VM” and check if you can see the shell prompt from the initrd?

 

The hypervisor does not inject vUART interrupts to an RTVM, which could be why the shell prompt does not show up when the kernel is not built with CONFIG_SERIAL_8250_DETECT_IRQ=y.

 

 

 

Hi,

 

Quick update:

 

I have a kernel + initrd that boots into my NFSROOT properly when I am not using ACRN. All network drivers & nfs modules compiled in. As soon as I want to launch the kernel as a Preloaded VM, it hangs with the last line being:

 

[    0.315764] cryptd: max_cpu_qlen set to 1000

 

I can reproduce this reliable. Any way to debug this?

 

Kernel version is: 5.15.64 (ubuntu jammy latest source)

 

Thx, Tom

 

 

Hi Junjie,

 

thank you for the reply and the right pointers! Indeed the kernel couldn’t find my NVMe because I didn’t map it. However, since I have only one NVMe drive in my IPC I need some other boot media.

 

I tried using a kernel with built in nfsroot and IP autoconfig on kernel level but it hangs during boot inside ACRN:

 

[    0.304553] smpboot: Max logical packages: 1

[    0.304553] smpboot: Total of 1 processors activated (3763.20 BogoMIPS)

[    0.304553] devtmpfs: initialized

[    0.304553] ACPI: PM: Registering ACPI NVS region [mem 0x7ff00000-0x7fffffff] (1048576 bytes)

[    0.304553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns

[    0.304553] futex hash table entries: 256 (order: 2, 16384 bytes, linear)

[    0.304553] pinctrl core: initialized pinctrl subsystem

[    0.304553] PM: RTC time: 21:35:19, date: 2023-01-03

[    0.304553] NET: Registered PF_NETLINK/PF_ROUTE protocol family

[    0.304553] audit: initializing netlink subsys (disabled)

[    0.304553] thermal_sys: Registered thermal governor 'bang_bang'

[    0.304553] thermal_sys: Registered thermal governor 'step_wise'

[    0.304553] thermal_sys: Registered thermal governor 'user_space'

[    0.304553] cpuidle: using governor ladder

[    0.304553] cpuidle: using governor menu

[    0.304553] PCI: Using configuration type 1 for base access

[    0.304553] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'

[    0.304553] Kprobes globally optimized

[    0.304553] audit: type=2000 audit(1672781719.000:1): state=initialized audit_enabled=0 res=1

[    0.304553] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages

[    0.304553] cryptd: max_cpu_qlen set to 1000

 

 

I also have a hanging kernel when I try booting a stock Ubuntu 22.04 kernel with INITRD. In the scenario.xml I defined:

 

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

      <ramdisk_mod>RTVM_INITRD</ramdisk_mod>

    </os_config>

 

In grub.cfg:

 

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

    multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

    module2 /boot/vmlinuz-5.15.0-56-generic RTVM rw rootwait ip=dhcp root=/dev/nfs nfsrootdebug nfsroot=192.168.0.83:/netroot_rtvm console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072

    module2 /boot/initrd.img-5.15.0-56-generic RTVM_INITRD

    module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty Linux_bzImage

}

 

Once the kernel boots, I see on vm_console 0 the kernel being stuck:

[    2.579592] ima: Allocated hash algorithm: sha1

[    2.580065] ima: No architecture policies found

[    2.580566] evm: Initialising EVM extended attributes:

[    2.581092] evm: security.selinux

[    2.581438] evm: security.SMACK64

[    2.581797] evm: security.SMACK64EXEC

[    2.582177] evm: security.SMACK64TRANSMUTE

[    2.582597] evm: security.SMACK64MMAP

[    2.582985] evm: security.apparmor

[    2.583332] evm: security.ima

[    2.583637] evm: security.capability

[    2.584031] evm: HMAC attrs: 0x1

[    2.584950] PM:   Magic number: 7:898:549

[    2.585390] tty ttyS25: hash matches

[    2.585791] clockevents clockevent0: hash matches

[    2.586632] RAS: Correctable Errors collector initialized.

[    2.589445] Freeing unused decrypted memory: 2036K

[    2.590805] Freeing unused kernel image (initmem) memory: 3232K

 

No mention of a wrong /dev/nfs (because it hasn’t gotten the kernel module yet). Just hanging.

 

Any ideas?

 

Thanks! Tom

 

 

 

Hi Tom,

 

Some issues I noticed in your command line for the pre-launched VM kernel are:

 

1. root=/dev/nvme0n1p3, which is a path to your NVMe which is not accessible to the pre-launched VM. You may want to install your OS on the eMMC and passthrough the eMMC controller to the pre-launched VM.

 

2. quiet and loglevel=3, which prevents the kernel from printing early logs that greatly help you diagnose such boot issues. To enable the logs, remove “quiet” and replace “loglevel=3” with “loglevel=7”.

 

 

 

Hi,

 

I successfully installed ACRN and got the Service VM up and running. Now I want to create a pre-launched Linux RT VM that later runs some PLC logic.

 

ACRN 3.1

Platform: Intel Elkhart Atom x6425RE

Service OS: Ubuntu 22.04

RT VM OS: Ubuntu 22.04 (basically the Service OS installed again on a new partition)

 

Partition Layout is:

259        1    1100800 nvme0n1p1 -> UEFI Boot

259        2  104857600 nvme0n1p2 -> Service OS (100G)

259        3   10485760 nvme0n1p3 -> RTVM OS (10G)

 

I configured ACRN for a hybrid scenario (see attached board & scenario.xml). In an essence, when I boot my device I see the Service VM starting (and I can reach it via Network) however the RTVM isn’t starting. Using COM2 I can access the ACRN shell:

 

ACRN:\>vm_list

 

VM_ID VM_NAME                          VM_STATE

===== ================================ ========

  0   RTVM                             Running

  1   ACRN_Service_VM                  Running

ACRN:\>vm_console 0

 

----- Entering VM 0 Shell -----

 

Also the interrupts look pretty dead for VM0. What did I do wrong? Find below some more information my setup.

 

Thanks!

Tom

 

 

Grub.cfg is as follows:

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

module2 /boot/vmlinuz-5.15.0-56-generic RTVM

module2 /boot/vmlinuz-5.15.44-acrn-service-vm Linux_bzImage

}

 

Scenario.XML:

  <vm id="0">

    <vm_type>RTVM</vm_type>

    <console_vuart>COM Port 1</console_vuart>

    <memory>

      <size>2048</size>

    </memory>

    <cpu_affinity>

      <pcpu>

        <pcpu_id>3</pcpu_id>

        <real_time_vcpu>y</real_time_vcpu>

      </pcpu>

    </cpu_affinity>

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check quiet loglevel=3 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

    </os_config>

    <mmio_resources>

      <TPM2>n</TPM2>

      <p2sb>n</p2sb>

    </mmio_resources>

    <pci_devs>

      <pci_dev>02:00.0 Ethernet controller: Intel Corporation Ethernet Controller (2) I225-IT</pci_dev>

    </pci_devs>

    <lapic_passthrough>y</lapic_passthrough>

    <io_completion_polling>y</io_completion_polling>

    <virtual_cat_support>n</virtual_cat_support>

    <virtual_cat_number>0</virtual_cat_number>

    <epc_section>

      <base>0</base>

      <size>0</size>

    </epc_section>

    <load_order>PRE_LAUNCHED_VM</load_order>

    <name>RTVM</name>

    <priority>PRIO_LOW</priority>

    <companion_vmid>65535</companion_vmid>

  </vm>

 

 

CONFIDENTIALITY: The contents of this e-mail are confidential and intended only for the above addressee(s). If you are not the intended recipient, or the person responsible for delivering it to the intended recipient, copying or delivering it to anyone else or using it in any unauthorized manner is prohibited and may be unlawful. If you receive this e-mail by mistake, please notify the sender and the systems administrator at straymail@... immediately.

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

Hi Junjie,

 

thanks! Let me know how it goes please. If you get it to work I would appreciate if you can share your grub.cfg, scenario.xml etc. Thanks!

Tom

 

Hi Tom,

 

Let me try using INITRD shell in a pre-launched VM on my side first to check if that’s platform specific or not.

 

 

 

Hi Junjie,

 

thank you for helping me!

 

So I tried a few things:

 

1. Changed the VM settings from Realtime to Standard VM:
1. No change
2. Compiled the kernel w/ “CONFIG_SERIAL_8250_DETECT_IRQ=y” _AND_ removed BTRFs support (since I saw the hang shortly before the RAID6 code)
1. Now it hangs at a different stage:

[    0.247281] Key type asymmetric registered

[    0.247281] Asymmetric key parser 'x509' registered

[    0.247281] Key type pkcs7_test registered

[    0.247281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)

[    0.247281] io scheduler mq-deadline registered

[    0.247281] io scheduler kyber registered

[    0.247281] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled

 

Should I provide kernel & initrd for testing in your environment? Maybe its my ACRN build that’s problematic…?

 

Cheers, Tom

 

 

Hi Tom,

 

Would you please try changing the VM type of the pre-launched VM from “RTVM” to “Standard VM” and check if you can see the shell prompt from the initrd?

 

The hypervisor does not inject vUART interrupts to an RTVM, which could be why the shell prompt does not show up when the kernel is not built with CONFIG_SERIAL_8250_DETECT_IRQ=y.

 

 

 

Hi,

 

Quick update:

 

I have a kernel + initrd that boots into my NFSROOT properly when I am not using ACRN. All network drivers & nfs modules compiled in. As soon as I want to launch the kernel as a Preloaded VM, it hangs with the last line being:

 

[    0.315764] cryptd: max_cpu_qlen set to 1000

 

I can reproduce this reliable. Any way to debug this?

 

Kernel version is: 5.15.64 (ubuntu jammy latest source)

 

Thx, Tom

 

 

Hi Junjie,

 

thank you for the reply and the right pointers! Indeed the kernel couldn’t find my NVMe because I didn’t map it. However, since I have only one NVMe drive in my IPC I need some other boot media.

 

I tried using a kernel with built in nfsroot and IP autoconfig on kernel level but it hangs during boot inside ACRN:

 

[    0.304553] smpboot: Max logical packages: 1

[    0.304553] smpboot: Total of 1 processors activated (3763.20 BogoMIPS)

[    0.304553] devtmpfs: initialized

[    0.304553] ACPI: PM: Registering ACPI NVS region [mem 0x7ff00000-0x7fffffff] (1048576 bytes)

[    0.304553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns

[    0.304553] futex hash table entries: 256 (order: 2, 16384 bytes, linear)

[    0.304553] pinctrl core: initialized pinctrl subsystem

[    0.304553] PM: RTC time: 21:35:19, date: 2023-01-03

[    0.304553] NET: Registered PF_NETLINK/PF_ROUTE protocol family

[    0.304553] audit: initializing netlink subsys (disabled)

[    0.304553] thermal_sys: Registered thermal governor 'bang_bang'

[    0.304553] thermal_sys: Registered thermal governor 'step_wise'

[    0.304553] thermal_sys: Registered thermal governor 'user_space'

[    0.304553] cpuidle: using governor ladder

[    0.304553] cpuidle: using governor menu

[    0.304553] PCI: Using configuration type 1 for base access

[    0.304553] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'

[    0.304553] Kprobes globally optimized

[    0.304553] audit: type=2000 audit(1672781719.000:1): state=initialized audit_enabled=0 res=1

[    0.304553] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages

[    0.304553] cryptd: max_cpu_qlen set to 1000

 

 

I also have a hanging kernel when I try booting a stock Ubuntu 22.04 kernel with INITRD. In the scenario.xml I defined:

 

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

      <ramdisk_mod>RTVM_INITRD</ramdisk_mod>

    </os_config>

 

In grub.cfg:

 

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

    multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

    module2 /boot/vmlinuz-5.15.0-56-generic RTVM rw rootwait ip=dhcp root=/dev/nfs nfsrootdebug nfsroot=192.168.0.83:/netroot_rtvm console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072

    module2 /boot/initrd.img-5.15.0-56-generic RTVM_INITRD

    module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty Linux_bzImage

}

 

Once the kernel boots, I see on vm_console 0 the kernel being stuck:

[    2.579592] ima: Allocated hash algorithm: sha1

[    2.580065] ima: No architecture policies found

[    2.580566] evm: Initialising EVM extended attributes:

[    2.581092] evm: security.selinux

[    2.581438] evm: security.SMACK64

[    2.581797] evm: security.SMACK64EXEC

[    2.582177] evm: security.SMACK64TRANSMUTE

[    2.582597] evm: security.SMACK64MMAP

[    2.582985] evm: security.apparmor

[    2.583332] evm: security.ima

[    2.583637] evm: security.capability

[    2.584031] evm: HMAC attrs: 0x1

[    2.584950] PM:   Magic number: 7:898:549

[    2.585390] tty ttyS25: hash matches

[    2.585791] clockevents clockevent0: hash matches

[    2.586632] RAS: Correctable Errors collector initialized.

[    2.589445] Freeing unused decrypted memory: 2036K

[    2.590805] Freeing unused kernel image (initmem) memory: 3232K

 

No mention of a wrong /dev/nfs (because it hasn’t gotten the kernel module yet). Just hanging.

 

Any ideas?

 

Thanks! Tom

 

 

 

Hi Tom,

 

Some issues I noticed in your command line for the pre-launched VM kernel are:

 

1. root=/dev/nvme0n1p3, which is a path to your NVMe which is not accessible to the pre-launched VM. You may want to install your OS on the eMMC and passthrough the eMMC controller to the pre-launched VM.

 

2. quiet and loglevel=3, which prevents the kernel from printing early logs that greatly help you diagnose such boot issues. To enable the logs, remove “quiet” and replace “loglevel=3” with “loglevel=7”.

 

 

 

Hi,

 

I successfully installed ACRN and got the Service VM up and running. Now I want to create a pre-launched Linux RT VM that later runs some PLC logic.

 

ACRN 3.1

Platform: Intel Elkhart Atom x6425RE

Service OS: Ubuntu 22.04

RT VM OS: Ubuntu 22.04 (basically the Service OS installed again on a new partition)

 

Partition Layout is:

259        1    1100800 nvme0n1p1 -> UEFI Boot

259        2  104857600 nvme0n1p2 -> Service OS (100G)

259        3   10485760 nvme0n1p3 -> RTVM OS (10G)

 

I configured ACRN for a hybrid scenario (see attached board & scenario.xml). In an essence, when I boot my device I see the Service VM starting (and I can reach it via Network) however the RTVM isn’t starting. Using COM2 I can access the ACRN shell:

 

ACRN:\>vm_list

 

VM_ID VM_NAME                          VM_STATE

===== ================================ ========

  0   RTVM                             Running

  1   ACRN_Service_VM                  Running

ACRN:\>vm_console 0

 

----- Entering VM 0 Shell -----

 

Also the interrupts look pretty dead for VM0. What did I do wrong? Find below some more information my setup.

 

Thanks!

Tom

 

 

Grub.cfg is as follows:

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

module2 /boot/vmlinuz-5.15.0-56-generic RTVM

module2 /boot/vmlinuz-5.15.44-acrn-service-vm Linux_bzImage

}

 

Scenario.XML:

  <vm id="0">

    <vm_type>RTVM</vm_type>

    <console_vuart>COM Port 1</console_vuart>

    <memory>

      <size>2048</size>

    </memory>

    <cpu_affinity>

      <pcpu>

        <pcpu_id>3</pcpu_id>

        <real_time_vcpu>y</real_time_vcpu>

      </pcpu>

    </cpu_affinity>

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check quiet loglevel=3 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

    </os_config>

    <mmio_resources>

      <TPM2>n</TPM2>

      <p2sb>n</p2sb>

    </mmio_resources>

    <pci_devs>

      <pci_dev>02:00.0 Ethernet controller: Intel Corporation Ethernet Controller (2) I225-IT</pci_dev>

    </pci_devs>

    <lapic_passthrough>y</lapic_passthrough>

    <io_completion_polling>y</io_completion_polling>

    <virtual_cat_support>n</virtual_cat_support>

    <virtual_cat_number>0</virtual_cat_number>

    <epc_section>

      <base>0</base>

      <size>0</size>

    </epc_section>

    <load_order>PRE_LAUNCHED_VM</load_order>

    <name>RTVM</name>

    <priority>PRIO_LOW</priority>

    <companion_vmid>65535</companion_vmid>

  </vm>

 

 

CONFIDENTIALITY: The contents of this e-mail are confidential and intended only for the above addressee(s). If you are not the intended recipient, or the person responsible for delivering it to the intended recipient, copying or delivering it to anyone else or using it in any unauthorized manner is prohibited and may be unlawful. If you receive this e-mail by mistake, please notify the sender and the systems administrator at straymail@... immediately.

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

Hi Junjie,

 

thank you for helping me!

 

So I tried a few things:

 

1. Changed the VM settings from Realtime to Standard VM:
1. No change
2. Compiled the kernel w/ “CONFIG_SERIAL_8250_DETECT_IRQ=y” _AND_ removed BTRFs support (since I saw the hang shortly before the RAID6 code)
1. Now it hangs at a different stage:

[    0.247281] Key type asymmetric registered

[    0.247281] Asymmetric key parser 'x509' registered

[    0.247281] Key type pkcs7_test registered

[    0.247281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)

[    0.247281] io scheduler mq-deadline registered

[    0.247281] io scheduler kyber registered

[    0.247281] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled

 

Should I provide kernel & initrd for testing in your environment? Maybe its my ACRN build that’s problematic…?

 

Cheers, Tom

 

Hi Tom,

 

Would you please try changing the VM type of the pre-launched VM from “RTVM” to “Standard VM” and check if you can see the shell prompt from the initrd?

 

The hypervisor does not inject vUART interrupts to an RTVM, which could be why the shell prompt does not show up when the kernel is not built with CONFIG_SERIAL_8250_DETECT_IRQ=y.

 

 

 

Hi,

 

Quick update:

 

I have a kernel + initrd that boots into my NFSROOT properly when I am not using ACRN. All network drivers & nfs modules compiled in. As soon as I want to launch the kernel as a Preloaded VM, it hangs with the last line being:

 

[    0.315764] cryptd: max_cpu_qlen set to 1000

 

I can reproduce this reliable. Any way to debug this?

 

Kernel version is: 5.15.64 (ubuntu jammy latest source)

 

Thx, Tom

 

 

Hi Junjie,

 

thank you for the reply and the right pointers! Indeed the kernel couldn’t find my NVMe because I didn’t map it. However, since I have only one NVMe drive in my IPC I need some other boot media.

 

I tried using a kernel with built in nfsroot and IP autoconfig on kernel level but it hangs during boot inside ACRN:

 

[    0.304553] smpboot: Max logical packages: 1

[    0.304553] smpboot: Total of 1 processors activated (3763.20 BogoMIPS)

[    0.304553] devtmpfs: initialized

[    0.304553] ACPI: PM: Registering ACPI NVS region [mem 0x7ff00000-0x7fffffff] (1048576 bytes)

[    0.304553] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns

[    0.304553] futex hash table entries: 256 (order: 2, 16384 bytes, linear)

[    0.304553] pinctrl core: initialized pinctrl subsystem

[    0.304553] PM: RTC time: 21:35:19, date: 2023-01-03

[    0.304553] NET: Registered PF_NETLINK/PF_ROUTE protocol family

[    0.304553] audit: initializing netlink subsys (disabled)

[    0.304553] thermal_sys: Registered thermal governor 'bang_bang'

[    0.304553] thermal_sys: Registered thermal governor 'step_wise'

[    0.304553] thermal_sys: Registered thermal governor 'user_space'

[    0.304553] cpuidle: using governor ladder

[    0.304553] cpuidle: using governor menu

[    0.304553] PCI: Using configuration type 1 for base access

[    0.304553] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'

[    0.304553] Kprobes globally optimized

[    0.304553] audit: type=2000 audit(1672781719.000:1): state=initialized audit_enabled=0 res=1

[    0.304553] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages

[    0.304553] cryptd: max_cpu_qlen set to 1000

 

 

I also have a hanging kernel when I try booting a stock Ubuntu 22.04 kernel with INITRD. In the scenario.xml I defined:

 

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

      <ramdisk_mod>RTVM_INITRD</ramdisk_mod>

    </os_config>

 

In grub.cfg:

 

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

    multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

    module2 /boot/vmlinuz-5.15.0-56-generic RTVM rw rootwait ip=dhcp root=/dev/nfs nfsrootdebug nfsroot=192.168.0.83:/netroot_rtvm console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072

    module2 /boot/initrd.img-5.15.0-56-generic RTVM_INITRD

    module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty Linux_bzImage

}

 

Once the kernel boots, I see on vm_console 0 the kernel being stuck:

[    2.579592] ima: Allocated hash algorithm: sha1

[    2.580065] ima: No architecture policies found

[    2.580566] evm: Initialising EVM extended attributes:

[    2.581092] evm: security.selinux

[    2.581438] evm: security.SMACK64

[    2.581797] evm: security.SMACK64EXEC

[    2.582177] evm: security.SMACK64TRANSMUTE

[    2.582597] evm: security.SMACK64MMAP

[    2.582985] evm: security.apparmor

[    2.583332] evm: security.ima

[    2.583637] evm: security.capability

[    2.584031] evm: HMAC attrs: 0x1

[    2.584950] PM:   Magic number: 7:898:549

[    2.585390] tty ttyS25: hash matches

[    2.585791] clockevents clockevent0: hash matches

[    2.586632] RAS: Correctable Errors collector initialized.

[    2.589445] Freeing unused decrypted memory: 2036K

[    2.590805] Freeing unused kernel image (initmem) memory: 3232K

 

No mention of a wrong /dev/nfs (because it hasn’t gotten the kernel module yet). Just hanging.

 

Any ideas?

 

Thanks! Tom

 

 

 

Hi Tom,

 

Some issues I noticed in your command line for the pre-launched VM kernel are:

 

1. root=/dev/nvme0n1p3, which is a path to your NVMe which is not accessible to the pre-launched VM. You may want to install your OS on the eMMC and passthrough the eMMC controller to the pre-launched VM.

 

2. quiet and loglevel=3, which prevents the kernel from printing early logs that greatly help you diagnose such boot issues. To enable the logs, remove “quiet” and replace “loglevel=3” with “loglevel=7”.

 

 

 

Hi,

 

I successfully installed ACRN and got the Service VM up and running. Now I want to create a pre-launched Linux RT VM that later runs some PLC logic.

 

ACRN 3.1

Platform: Intel Elkhart Atom x6425RE

Service OS: Ubuntu 22.04

RT VM OS: Ubuntu 22.04 (basically the Service OS installed again on a new partition)

 

Partition Layout is:

259        1    1100800 nvme0n1p1 -> UEFI Boot

259        2  104857600 nvme0n1p2 -> Service OS (100G)

259        3   10485760 nvme0n1p3 -> RTVM OS (10G)

 

I configured ACRN for a hybrid scenario (see attached board & scenario.xml). In an essence, when I boot my device I see the Service VM starting (and I can reach it via Network) however the RTVM isn’t starting. Using COM2 I can access the ACRN shell:

 

ACRN:\>vm_list

 

VM_ID VM_NAME                          VM_STATE

===== ================================ ========

  0   RTVM                             Running

  1   ACRN_Service_VM                  Running

ACRN:\>vm_console 0

 

----- Entering VM 0 Shell -----

 

Also the interrupts look pretty dead for VM0. What did I do wrong? Find below some more information my setup.

 

Thanks!

Tom

 

 

Grub.cfg is as follows:

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

module2 /boot/vmlinuz-5.15.0-56-generic RTVM

module2 /boot/vmlinuz-5.15.44-acrn-service-vm Linux_bzImage

}

 

Scenario.XML:

  <vm id="0">

    <vm_type>RTVM</vm_type>

    <console_vuart>COM Port 1</console_vuart>

    <memory>

      <size>2048</size>

    </memory>

    <cpu_affinity>

      <pcpu>

        <pcpu_id>3</pcpu_id>

        <real_time_vcpu>y</real_time_vcpu>

      </pcpu>

    </cpu_affinity>

    <os_config>

      <kern_type>KERNEL_BZIMAGE</kern_type>

      <kern_mod>RTVM</kern_mod>

      <bootargs>rw rootwait root=/dev/nvme0n1p3 console=tty0 console=ttyS0 consoleblank=0 no_timer_check quiet loglevel=3 i915.nuclear_pageflip=1 swiotlb=131072</bootargs>

    </os_config>

    <mmio_resources>

      <TPM2>n</TPM2>

      <p2sb>n</p2sb>

    </mmio_resources>

    <pci_devs>

      <pci_dev>02:00.0 Ethernet controller: Intel Corporation Ethernet Controller (2) I225-IT</pci_dev>

    </pci_devs>

    <lapic_passthrough>y</lapic_passthrough>

    <io_completion_polling>y</io_completion_polling>

    <virtual_cat_support>n</virtual_cat_support>

    <virtual_cat_number>0</virtual_cat_number>

    <epc_section>

      <base>0</base>

      <size>0</size>

    </epc_section>

    <load_order>PRE_LAUNCHED_VM</load_order>

    <name>RTVM</name>

    <priority>PRIO_LOW</priority>

    <companion_vmid>65535</companion_vmid>

  </vm>

 

 

CONFIDENTIALITY: The contents of this e-mail are confidential and intended only for the above addressee(s). If you are not the intended recipient, or the person responsible for delivering it to the intended recipient, copying or delivering it to anyone else or using it in any unauthorized manner is prohibited and may be unlawful. If you receive this e-mail by mistake, please notify the sender and the systems administrator at straymail@... immediately.

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

 

TTTech - Internal

Hi,

 

Quick update:

 

I have a kernel + initrd that boots into my NFSROOT properly when I am not using ACRN. All network drivers & nfs modules compiled in. As soon as I want to launch the kernel as a Preloaded VM, it hangs with the last line being:

 

[    0.315764] cryptd: max_cpu_qlen set to 1000

 

I can reproduce this reliable. Any way to debug this?

 

Kernel version is: 5.15.64 (ubuntu jammy latest source)

 

Thx, Tom

 

Hi Junjie,

 

yes – I build a kernel that has the NIC, NFS Client & NFSROOT modules statically compiled in. If I boot the kernel w/o ACRN via Grub it gets an IP address and tries to reach the NFS server. If I boot via ACRN, the kernel hangs.

 

For testing, I have taken an alpine linux kernel + initramfs and try to boot as a pre-launched VM in ACRN but again, same hang.

 

This is my grub.cfg:

### BEGIN /etc/grub.d/40_custom ###

menuentry 'ACRN multiboot2 ' --id ACRN_deb_multiboot2 {

    load_video

    insmod gzio

    insmod part_gpt

    insmod ext2

    search --no-floppy --fs-uuid  --set f77802cb-6801-4f84-91da-e2d0cce3777f

multiboot2 /boot/acrn.spc-6000.my_board.bin root=PARTUUID=3623fd71-b19e-4129-96f0-a2e893898f11

module2 /boot/vmlinuz-lts RTVM console=tty0 console=ttyS0 consoleblank=0 no_timer_check loglevel=7 i915.nuclear_pageflip=1 swiotlb=131072

module2 /boot/initramfs-lts RTVM_INITRD

module2 /boot/vmlinuz-5.15.44-acrn-service-vm-dirty Linux_bzImage

}

 

Below pls find the kernel output.

 

Do you have an example of a working linux kernel + initrd pre-launched scenario & grub.cfg? Any ideas how to make this work?

 

Thx! Tom

 

[    0.329138] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)

[    0.329138] TCP bind hash table entries: 16384 (order: 6, 262144 bytes, linear)

[    0.329138] TCP: Hash tables configured (established 16384 bind 16384)

[    0.329138] UDP hash table entries: 1024 (order: 3, 32768 bytes, linear)

[    0.329138] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear)

[    0.329138] NET: Registered PF_UNIX/PF_LOCAL protocol family

[    0.329138] NET: Registered PF_XDP protocol family

[    0.329138] pci_bus 0000:00: resource 4 [io  0x0000-0xffff]

[    0.329138] pci_bus 0000:00: resource 5 [mem 0x00000000-0x7fffffffff]

[    0.329138] PCI: CLS 0 bytes, default 64

[    0.329138] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

[    0.329138] software IO TLB: mapped [mem 0x000000006a200000-0x000000007a200000] (256MB)

[    0.329138] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x363e8c91135, max_idle_ns: 881590568389 ns

[    0.329138] Trying to unpack rootfs image as initramfs...

[    0.329138] Freeing initrd memory: 51200K

[    0.329138] clocksource: Switched to clocksource tsc

[    0.329138] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.329138] Initialise system trusted keyrings

[    0.329138] workingset: timestamp_bits=40 max_order=19 bucket_order=0

[    0.329138] zbud: loaded

[    0.329138] Key type asymmetric registered

[    0.329138] Asymmetric key parser 'x509' registered

[    0.329138] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)

[    0.329138] io scheduler mq-deadline registered

[    0.329138] io scheduler kyber registered

[    0.329138] io scheduler bfq registered

[    0.329138] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4

[    0.329138] ERST DBG: ERST support is disabled.

[    0.329138] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    0.329138] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A

[    0.329138] serial8250: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A

[    0.329138] brd: module loaded

[    0.329138] i8042: PNP: No PS/2 controller found.

[    0.329138] i8042: Probing ports directly.

[    0.329138] i8042: No controller found

[    0.329138] rtc_cmos: probe of rtc_cmos failed with error -22

[    0.329138] intel_pstate: CPU model not supported

[    0.329138] ashmem: initialized

[    0.329138] gre: GRE over IPv4 demultiplexor driver

[    0.329138] can: controller area network core

[    0.329138] NET: Registered PF_CAN protocol family

[    0.329138] Key type dns_resolver registered

[    0.329138] IPI shorthand broadcast: enabled

[    0.329138] sched_clock: Marking stable (265830371, 59974376)->(325804747, 0)

[    0.686374] registered taskstats version 1

[    0.686776] Loading compiled-in X.509 certificates

[    0.687334] zswap: loaded using pool lzo/zbud

[    0.688325] Key type .fscrypt registered

[    0.688688] Key type fscrypt-provisioning registered

[    0.692368] Freeing unused kernel image (initmem) memory: 2216K