acrn-users@lists.projectacrn.org

Re: Using perf to access hardware events in pre-launched VM

Minggui Cao

#1152

I don’t meet that issue. If you have Pre-launched RTVM / Service VM, you can check in ACRN console: vm_list

And then vm_console N to switch the VM console.

Thanks

Minggui

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Thursday, March 31, 2022 1:07 AM
To: acrn-users@...
Cc: 204572@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

IT WORKS! I set the scenario as you said and now I can read the cache events.

Thank you so much for the help.

Only one small note: when I use the GUEST_FLAG_LAPIC_PASSTHROUGH flag, the terminal of the VM (minicom) becomes unresponsive. When I start the VM, I see some of the kernel messages: it displays only the ones with the time (like [ 1.299022] Run /init as init process) and none with the "ok" (like [ OK ] Started WPA supplicant.). I cannot write or interact in any way with the shell.

It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.

It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

Anyway, thanks again for your help.

Show quoted text

On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Re: CAT definition in the scenario xmls

Minggui Cao

#1151

Yes, it is for L3 CAT. You can check tgl-rvp.xml the board file, there are some changes for L3 info.

Current, ACRN just supports one of L2/L3 CAT, if you need support both, you can do it as hard code.

Thanks

Minggui

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Wednesday, March 30, 2022 4:00 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

Hi Minggui,

Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion?
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?

Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti

#1150

IT WORKS! I set the scenario as you said and now I can read the cache events.

Thank you so much for the help.

Only one small note: when I use the GUEST_FLAG_LAPIC_PASSTHROUGH flag, the terminal of the VM (minicom) becomes unresponsive. When I start the VM, I see some of the kernel messages: it displays only the ones with the time (like [ 1.299022] Run /init as init process) and none with the "ok" (like [ OK ] Started WPA supplicant.). I cannot write or interact in any way with the shell.

It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.

It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

Anyway, thanks again for your help.

Show quoted text

On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Re: CAT definition in the scenario xmls

jordan.nowak22@...

#1149

Hi Minggui,

Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion?
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?

Re: CAT definition in the scenario xmls

Minggui Cao

#1148

Hi, for CAT support (L2/L3), you can check it by cpuid;

For L2, if on core platform it is private for a physical core, L3/LLC is shared between cores, if hyper-threading is not enabled, L2 CAT not helpful.

L2/L3 CAT use different MSR to set, so no matter the cache ways the same or not.

You can refer ACRN: release_2.5 branch, the tgl-rvp board industry scenario has L3 CAT setting.

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Tuesday, March 29, 2022 11:02 PM
To: acrn-users@...
Subject: [acrn-users] CAT definition in the scenario xmls

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls?

Re: Using perf to access hardware events in pre-launched VM

Minggui Cao

#1147

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm_type>PRE_RT_VM</vm_type>

<guest_flags>

<guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

<guest_flag>GUEST_FLAG_RT</guest_flag>

</guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23: .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

CAT definition in the scenario xmls

jordan.nowak22@...

#1146

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls?

Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti

#1145

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Re: Using perf to access hardware events in pre-launched VM

Minggui Cao

#1144

Hi, Paolo,

Could you check again to rdmsr 0x186 and 0x38f? If the ACRN has not pass-through PMU to VM, it shall like following:

root@10239146109sos-dom0:~# dmesg | grep -i pmu

[ 0.143455] Performance Events: unsupported p6 CPU model 140 no PMU driver, software events only.

root@10239146109sos-dom0:~# rdmsr 0x186

rdmsr: CPU 0 cannot read MSR 0x00000186

root@10239146109sos-dom0:~# rdmsr 0x38f

rdmsr: CPU 0 cannot read MSR 0x0000038f

But from your feedback, you can rdmsr 0x186 and no PMU driver. It is very strange.

Thanks!

Minggui

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Monday, March 28, 2022 6:15 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:

[ 0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".

Re: Extracting scenario xml information

jordan.nowak22@...

#1143

noted, would this be implemented in the future?

i don't have a use case for this, just want to double confirm my changes on scenario xml in the target board itself

Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti

#1142

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:

[ 0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".

Re: Using perf to access hardware events in pre-launched VM

Minggui Cao

#1141

Following is an example from my side:

./perf stat -C 1 -e mem_load_retired.l2_hit,mem_load_retired.l2_miss,mem_load_retired.l3_hit,mem_load_retired.l3_miss ./pmu_mem -a 1 -p 80 -l 1 -m 12

Performance counter stats for 'CPU(s) 1':

30260 mem_load_retired.l2_hit

7718 mem_load_retired.l2_miss

2587 mem_load_retired.l3_hit

4240 mem_load_retired.l3_miss

From: acrn-users@... <acrn-users@...> On Behalf Of Minggui Cao
Sent: Saturday, March 26, 2022 8:57 AM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi, Paolo,

If you can read 0x186, it means, you can read PMU MSRs, it shall work for perf to sample some events, like LLC misses.

You can also check by: dmesg | grep -i pmu

To check PMU driver is probed.

You don’t need modify the scenario file.

Minggui

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Friday, March 25, 2022 7:08 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

Thanks for the quick answer. With my original scenario, I tried rdmsr 0x186 and it returned rdmsr(0x186):0x0. Is it ok?

Then I modified the scenario and i get a lot of errors:

First I changed the <vm_type> PRE_STD_VM to RTVM, then run make hypervisor and I got this error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Traceback (most recent call last):

File "/home/paoloc/acrn-work/acrn-hypervisor/hypervisor/../misc/config_tools/board_config/board_cfg_gen.py", line 137, in <module>

err_dic = main(ARGS)

File "/home/paoloc/acrn-work/acrn-hypervisor/hypervisor/../misc/config_tools/board_config/board_cfg_gen.py", line 95, in main

err_dic = board_info_h.generate_file(config)

File "/home/paoloc/acrn-work/acrn-hypervisor/misc/config_tools/board_config/board_info_h.py", line 96, in generate_file

scenario_cfg_lib.VM_DB[common.VM_TYPES[0]]['load_type'] == "PRE_LAUNCHED_VM"

KeyError: 'RTVM'

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make -C /home/paoloc/acrn-work/acrn-hypervisor/hypervisor BOARD=/home/paoloc/acrn-work/my_board.xml SCENARIO=/home/paoloc/acrn-work/notPartitioned-EnablePMU.xml HV_OBJDIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor RELEASE=

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: *** Nessuna regola per generare l'obiettivo «/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/scenarios/partitioned_CAT/vm_configurations.c», necessario per «default». Arresto.

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

Then if I add this line <load_order>PRE_LAUNCHED_VM</load_order>, I get a different error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Board configurations for my_board is generated successfully.

Error: Scenario xml file validation failed:

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make -C /home/paoloc/acrn-work/acrn-hypervisor/hypervisor BOARD=/home/paoloc/acrn-work/my_board.xml SCENARIO=/home/paoloc/acrn-work/notPartitioned-EnablePMU.xml HV_OBJDIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor RELEASE=

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: *** Nessuna regola per generare l'obiettivo «/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/scenarios/partitioned_CAT/vm_configurations.c», necessario per «default». Arresto.

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

And if I add <lapic_passthrough>y</lapic_passthrough>, I get another different error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Board configurations for my_board is generated successfully.

Error: Scenario xml file validation failed:

Error: vm:id=0: 'vm:id=0' Unexpected child with tag 'lapic_passthrough' at position 4.: last call: <Element 'vm' at 0x7fc2ef3a05e0>

Error: vm:id=1: 'vm:id=1' Unexpected child with tag 'lapic_passthrough' at position 4.: last call: <Element 'vm' at 0x7fc2ef3a4770>

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make -C /home/paoloc/acrn-work/acrn-hypervisor/hypervisor BOARD=/home/paoloc/acrn-work/my_board.xml SCENARIO=/home/paoloc/acrn-work/notPartitioned-EnablePMU.xml HV_OBJDIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor RELEASE=

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: *** Nessuna regola per generare l'obiettivo «/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/scenarios/partitioned_CAT/vm_configurations.c», necessario per «default». Arresto.

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

Do you have any idea what the problem could be?

P.S. Attached my original scenario and the modified one.

Re: Using perf to access hardware events in pre-launched VM

Minggui Cao

#1140