acrn-users@lists.projectacrn.org | Using perf to access hardware events in pre-launched VM

Home Messages Hashtags Subgroups

Date Date 1 - 16 of 16

Using perf to access hardware events in pre-launched VM

Paolo Crotti

I managed to compile the kernel with the option you told me. The console is now working correctly.

Thanks again for your help!

Paolo Crotti

#1160

Hi Minggui,

Sorry if i haven't answered anymore but i'm having problems with kernel recompilation. As soon as I have solved it I'll let you know.

Minggui Cao

#1157

Just synced with my colleague, the issue could be caused in your kernel configure of pre-launched VM:

CONFIG_SERIAL_8250_DETECT_IRQ=y

Please confirm.

Thanks

Minggui

Show quoted text

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Thursday, March 31, 2022 4:57 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Yes, with the vm terminal I meant ACRN console. After the boot (vm_console 0), the console becomes unusable but the kernel starts correctly.

Paolo Crotti

#1154

Yes, with the vm terminal I meant ACRN console. After the boot (vm_console 0), the console becomes unusable but the kernel starts correctly.

Minggui Cao

#1152

I don’t meet that issue. If you have Pre-launched RTVM / Service VM, you can check in ACRN console: vm_list

And then vm_console N to switch the VM console.

Thanks

Minggui

Show quoted text

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Thursday, March 31, 2022 1:07 AM
To: acrn-users@...
Cc: 204572@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

IT WORKS! I set the scenario as you said and now I can read the cache events.

Thank you so much for the help.

Only one small note: when I use the GUEST_FLAG_LAPIC_PASSTHROUGH flag, the terminal of the VM (minicom) becomes unresponsive. When I start the VM, I see some of the kernel messages: it displays only the ones with the time (like [ 1.299022] Run /init as init process) and none with the "ok" (like [ OK ] Started WPA supplicant.). I cannot write or interact in any way with the shell.

It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.

It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

Anyway, thanks again for your help.

On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Paolo Crotti

#1150

IT WORKS! I set the scenario as you said and now I can read the cache events.

Thank you so much for the help.

It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.

It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

Anyway, thanks again for your help.

Show quoted text

On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Minggui Cao

#1147

That is reasonable. It means APIC pass-through not set. Let me check,

You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

<vm_type>PRE_RT_VM</vm_type>

<guest_flags>

<guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

<guest_flag>GUEST_FLAG_RT</guest_flag>

</guest_flags>

After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23: .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

good luck.

Show quoted text

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Paolo Crotti

#1145

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

Minggui Cao

#1144

Hi, Paolo,

Could you check again to rdmsr 0x186 and 0x38f? If the ACRN has not pass-through PMU to VM, it shall like following:

root@10239146109sos-dom0:~# dmesg | grep -i pmu

[ 0.143455] Performance Events: unsupported p6 CPU model 140 no PMU driver, software events only.

root@10239146109sos-dom0:~# rdmsr 0x186

rdmsr: CPU 0 cannot read MSR 0x00000186

root@10239146109sos-dom0:~# rdmsr 0x38f

rdmsr: CPU 0 cannot read MSR 0x0000038f

But from your feedback, you can rdmsr 0x186 and no PMU driver. It is very strange.

Thanks!

Minggui

Show quoted text

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Monday, March 28, 2022 6:15 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:

[ 0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".

Paolo Crotti

#1142

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:

[ 0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".

Minggui Cao

#1141

Following is an example from my side:

./perf stat -C 1 -e mem_load_retired.l2_hit,mem_load_retired.l2_miss,mem_load_retired.l3_hit,mem_load_retired.l3_miss ./pmu_mem -a 1 -p 80 -l 1 -m 12

Performance counter stats for 'CPU(s) 1':

30260 mem_load_retired.l2_hit

7718 mem_load_retired.l2_miss

2587 mem_load_retired.l3_hit

4240 mem_load_retired.l3_miss

Show quoted text

From: acrn-users@... <acrn-users@...> On Behalf Of Minggui Cao
Sent: Saturday, March 26, 2022 8:57 AM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi, Paolo,

If you can read 0x186, it means, you can read PMU MSRs, it shall work for perf to sample some events, like LLC misses.

You can also check by: dmesg | grep -i pmu

To check PMU driver is probed.

You don’t need modify the scenario file.

Minggui

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Friday, March 25, 2022 7:08 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

Hi Minggui,

Thanks for the quick answer. With my original scenario, I tried rdmsr 0x186 and it returned rdmsr(0x186):0x0. Is it ok?

Then I modified the scenario and i get a lot of errors:

First I changed the <vm_type> PRE_STD_VM to RTVM, then run make hypervisor and I got this error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Traceback (most recent call last):

File "/home/paoloc/acrn-work/acrn-hypervisor/hypervisor/../misc/config_tools/board_config/board_cfg_gen.py", line 137, in <module>

err_dic = main(ARGS)

File "/home/paoloc/acrn-work/acrn-hypervisor/hypervisor/../misc/config_tools/board_config/board_cfg_gen.py", line 95, in main

err_dic = board_info_h.generate_file(config)

File "/home/paoloc/acrn-work/acrn-hypervisor/misc/config_tools/board_config/board_info_h.py", line 96, in generate_file

scenario_cfg_lib.VM_DB[common.VM_TYPES[0]]['load_type'] == "PRE_LAUNCHED_VM"

KeyError: 'RTVM'

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make -C /home/paoloc/acrn-work/acrn-hypervisor/hypervisor BOARD=/home/paoloc/acrn-work/my_board.xml SCENARIO=/home/paoloc/acrn-work/notPartitioned-EnablePMU.xml HV_OBJDIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor RELEASE=

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: *** Nessuna regola per generare l'obiettivo «/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/scenarios/partitioned_CAT/vm_configurations.c», necessario per «default». Arresto.

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

Then if I add this line <load_order>PRE_LAUNCHED_VM</load_order>, I get a different error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Board configurations for my_board is generated successfully.

Error: Scenario xml file validation failed:

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

And if I add <lapic_passthrough>y</lapic_passthrough>, I get another different error:

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

Board XML is fetched from /home/paoloc/acrn-work/my_board.xml

Scneario XML is configuration fetched from /home/paoloc/acrn-work/notPartitioned-EnablePMU.xml

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/unified.xml generated

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs/config.mk generated

Board configurations for my_board is generated successfully.

Error: Scenario xml file validation failed:

Error: vm:id=0: 'vm:id=0' Unexpected child with tag 'lapic_passthrough' at position 4.: last call: <Element 'vm' at 0x7fc2ef3a05e0>

Error: vm:id=1: 'vm:id=1' Unexpected child with tag 'lapic_passthrough' at position 4.: last call: <Element 'vm' at 0x7fc2ef3a4770>

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make[1]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/include/config.h generated

Start pre-build static check ...

make -C ../misc/hv_prebuild BOARD=my_board SCENARIO=partitioned_CAT TARGET_DIR=/home/paoloc/acrn-work/acrn-hypervisor/build/hypervisor/configs

make[2]: ingresso nella directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[2]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/misc/hv_prebuild»

make[1]: *** [Makefile:413: pre_build] Errore 2

make[1]: uscita dalla directory «/home/paoloc/acrn-work/acrn-hypervisor/hypervisor»

make: *** [Makefile:109: hypervisor] Errore 2

Do you have any idea what the problem could be?

P.S. Attached my original scenario and the modified one.

Minggui Cao

#1140