Date   

Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

I managed to compile the kernel with the option you told me. The console is now working correctly.

Thanks again for your help!


Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

Hi Minggui,

Sorry if i haven't answered anymore but i'm having problems with kernel recompilation. As soon as I have solved it I'll let you know.


Re: CAT definition in the scenario xmls

Geoffroy Van Cutsem
 

I’d love to see that being added to our code, would you care to develop and patch and submit it upstream? 😉

 

Thanks!
Geoffroy

 

From: acrn-users@... <acrn-users@...> On Behalf Of Minggui Cao
Sent: Saturday, April 2, 2022 2:04 am
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes, correct.

 

You can add your code in function: setup_clos

 

There is a data structure: res_cap_info, if possible, you also can re-use it to configure your L2 CAT.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Friday, April 1, 2022 2:54 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c


Re: CAT definition in the scenario xmls

Minggui Cao
 

Yes, correct.

 

You can add your code in function: setup_clos

 

There is a data structure: res_cap_info, if possible, you also can re-use it to configure your L2 CAT.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Friday, April 1, 2022 2:54 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c


Re: Using perf to access hardware events in pre-launched VM

Minggui Cao
 

Just synced with my colleague, the issue could be caused in your kernel configure of pre-launched VM:

CONFIG_SERIAL_8250_DETECT_IRQ=y

 

 

Please confirm.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Thursday, March 31, 2022 4:57 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

Yes, with the vm terminal I meant ACRN console. After the boot (vm_console 0), the console becomes unusable but the kernel starts correctly.


Re: CAT definition in the scenario xmls

jordan.nowak22@...
 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c


Re: CAT definition in the scenario xmls

Minggui Cao
 

Hi, Jordan,

   You can use “wrmsr “ in ACRN console first to configure L2 CAT. If it works for your expectation. You can it into rdt code.

 

Thanks!

Minggui

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Thursday, March 31, 2022 9:35 AM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

[Edited Message Follows]

Hi Minggui,

For both L2 and L3 CAT support, do you have any example when you say hard code? 
or is there any documentation that states this?


Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

Yes, with the vm terminal I meant ACRN console. After the boot (vm_console 0), the console becomes unusable but the kernel starts correctly.


Re: CAT definition in the scenario xmls

jordan.nowak22@...
 
Edited

Hi Minggui,

For both L2 and L3 CAT support, do you have any example when you say hard code? 
or is there any documentation that states this?


Re: Using perf to access hardware events in pre-launched VM

Minggui Cao
 

I don’t meet that issue.   If you have Pre-launched RTVM / Service VM, you can check in ACRN console:   vm_list

And then vm_console  N to switch the VM console.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Thursday, March 31, 2022 1:07 AM
To: acrn-users@...
Cc: 204572@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

IT WORKS! I set the scenario as you said and now I can read the cache events.

 

Thank you so much for the help.

 

Only one small note: when I use the GUEST_FLAG_LAPIC_PASSTHROUGH flag, the terminal of the VM (minicom) becomes unresponsive. When I start the VM, I see some of the kernel messages: it displays only the ones with the time (like [    1.299022] Run /init as init process) and none with the "ok" (like [  OK  ] Started WPA supplicant.). I cannot write or interact in any way with the shell.

It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.

It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

 

Anyway, thanks again for your help.

 

On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

 

 

  1. You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

 

 

  <vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

 

 

  1. After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

 

 

good luck.

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f

 


Re: CAT definition in the scenario xmls

Minggui Cao
 

Yes, it is for L3 CAT. You can check tgl-rvp.xml the board file, there are some changes for L3 info.

 

Current, ACRN just supports one of L2/L3 CAT, if you need support both, you can do it as hard code.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Wednesday, March 30, 2022 4:00 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Hi Minggui,



Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion? 
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?


Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

IT WORKS! I set the scenario as you said and now I can read the cache events.

Thank you so much for the help.

Only one small note: when I use the GUEST_FLAG_LAPIC_PASSTHROUGH flag, the terminal of the VM (minicom) becomes unresponsive. When I start the VM, I see some of the kernel messages: it displays only the ones with the time (like [    1.299022] Run /init as init process) and none with the "ok" (like [  OK  ] Started WPA supplicant.). I cannot write or interact in any way with the shell.
It seems that, for some reason, it no longer shows the stdout and doesn't get the stdin.
It's not a real problem because the kernel starts correctly and I connect to the machine via SSH, still it's strange because it happens only with that flag.

Anyway, thanks again for your help.


On Wed, 30 Mar 2022 at 02:40, Minggui Cao <minggui.cao@...> wrote:

That is reasonable. It means APIC pass-through not set. Let me check,

 

 

  1. You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

 

 

  <vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

 

 

  1. After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

 

 

good luck.

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f


 


Re: CAT definition in the scenario xmls

jordan.nowak22@...
 

Hi Minggui,



Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion? 
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?


Re: CAT definition in the scenario xmls

Minggui Cao
 

Hi, for CAT support (L2/L3), you can check it by cpuid;

 

For L2, if on core platform it is private for a physical core,  L3/LLC is shared between cores, if hyper-threading is not enabled, L2 CAT not helpful.

 

L2/L3 CAT use different MSR to set, so no matter the cache ways the same or not.

 

You can refer ACRN: release_2.5 branch, the tgl-rvp board industry scenario has L3 CAT setting.

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Tuesday, March 29, 2022 11:02 PM
To: acrn-users@...
Subject: [acrn-users] CAT definition in the scenario xmls

 

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls? 


Re: Using perf to access hardware events in pre-launched VM

Minggui Cao
 

That is reasonable. It means APIC pass-through not set. Let me check,

 

 

  1. You can refer this file to modify yours:

misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml

 

 

  <vm id="0">

    <vm_type>PRE_RT_VM</vm_type>

    <name>PRE_RT_VM0</name>

    <guest_flags>

      <guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>

      <guest_flag>GUEST_FLAG_RT</guest_flag>

    </guest_flags>

 

 

  1. After you build hypervisor, you can check:

build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c:23:            .guest_flags = (GUEST_FLAG_STATIC_VM | GUEST_FLAG_LAPIC_PASSTHROUGH | GUEST_FLAG_RT),

 

 

good luck.

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Tuesday, March 29, 2022 6:16 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x00000186

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!

rdmsr: CPU 0 cannot read MSR 0x0000038f


 


CAT definition in the scenario xmls

jordan.nowak22@...
 

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls? 


Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

Hi Minggui,

I'm sorry, I didn't realize I had to read the 0x186 register from inside the VM. I had read it from the ACRN shell (ACRN: \> rdmsr 0x186).

Actually, I cannot read these registers from the virtual machine. In addition, when I try to read them I got two strange error messages:

usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x186[1131822537us][cpu=0][vm0:vcpu0][sev=1][seq=44]:dispatch VM exit handler failed for reason 31, ret = -13!
rdmsr: CPU 0 cannot read MSR 0x00000186
usb-ubuntu@usbubuntu-C93:~$ sudo rdmsr 0x38f[1149902298us][cpu=0][vm0:vcpu0][sev=1][seq=46]:dispatch VM exit handler failed for reason 31, ret = -13!
rdmsr: CPU 0 cannot read MSR 0x0000038f





Re: Using perf to access hardware events in pre-launched VM

Minggui Cao
 

Hi, Paolo,

 

Could you check again to rdmsr 0x186 and 0x38f?  If the ACRN has not pass-through PMU to VM, it shall  like following:

 

root@10239146109sos-dom0:~# dmesg | grep -i pmu

[    0.143455] Performance Events: unsupported p6 CPU model 140 no PMU driver, software events only.

root@10239146109sos-dom0:~# rdmsr 0x186

rdmsr: CPU 0 cannot read MSR 0x00000186

root@10239146109sos-dom0:~# rdmsr 0x38f

rdmsr: CPU 0 cannot read MSR 0x0000038f

 

 

But from your feedback, you can rdmsr 0x186 and no PMU driver. It is very strange.

 

Thanks!

Minggui

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of Paolo Crotti
Sent: Monday, March 28, 2022 6:15 PM
To: acrn-users@...
Subject: Re: [acrn-users] Using perf to access hardware events in pre-launched VM

 

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:

[    0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".







Re: Extracting scenario xml information

jordan.nowak22@...
 

noted, would this be implemented in the future? 

i don't have a use case for this, just want to double confirm my changes on scenario xml in the target board itself 


Re: Using perf to access hardware events in pre-launched VM

Paolo Crotti
 

Hi Minggui,

I checked "dmesg | grep -i pmu" and it returned me:
[    0.075539] Performance Events: unsupported p6 CPU model 150 no PMU driver, software events only.

And all the events in your perf command are not in my "perf list".







41 - 60 of 1201