1. Acrn-Users
  2. Topics

CAT definition in the scenario xmls

jordan.nowak22@...
 

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls? 
Minggui Cao
 

Hi, for CAT support (L2/L3), you can check it by cpuid;

 

For L2, if on core platform it is private for a physical core,  L3/LLC is shared between cores, if hyper-threading is not enabled, L2 CAT not helpful.

 

L2/L3 CAT use different MSR to set, so no matter the cache ways the same or not.

 

You can refer ACRN: release_2.5 branch, the tgl-rvp board industry scenario has L3 CAT setting.

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Tuesday, March 29, 2022 11:02 PM
To: acrn-users@...
Subject: [acrn-users] CAT definition in the scenario xmls

 

Hi ACRN experts,

based on this guide https://projectacrn.github.io/latest/tutorials/rdt_configuration.html#configure-rdt-for-vm-using-vm-configuration
I see that L2 cache is enabled for CAT but i'm not sure on the L3/LLC, how about for platforms that have different number of cacheways for L2 and L3, what's the right way to enable CAT for L3/LLC in the scenario xmls? 
jordan.nowak22@...
 

Hi Minggui,



Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion? 
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?

Minggui Cao
 

Yes, it is for L3 CAT. You can check tgl-rvp.xml the board file, there are some changes for L3 info.

 

Current, ACRN just supports one of L2/L3 CAT, if you need support both, you can do it as hard code.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Wednesday, March 30, 2022 4:00 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Hi Minggui,



Do you mean that for L3 CAT setting, i also need specify it in the board xmls in the CLOS info portion? 
if i want to use both L2 and L3 CAT, how would i definte that in the board xmls?

jordan.nowak22@...
 
Edited

Hi Minggui,

For both L2 and L3 CAT support, do you have any example when you say hard code? 
or is there any documentation that states this?
Minggui Cao
 

Hi, Jordan,

   You can use “wrmsr “ in ACRN console first to configure L2 CAT. If it works for your expectation. You can it into rdt code.

 

Thanks!

Minggui

 

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Thursday, March 31, 2022 9:35 AM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

[Edited Message Follows]

Hi Minggui,

For both L2 and L3 CAT support, do you have any example when you say hard code? 
or is there any documentation that states this?

jordan.nowak22@...
 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c
Minggui Cao
 

Yes, correct.

 

You can add your code in function: setup_clos

 

There is a data structure: res_cap_info, if possible, you also can re-use it to configure your L2 CAT.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Friday, April 1, 2022 2:54 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c
Geoffroy Van Cutsem
 

I’d love to see that being added to our code, would you care to develop and patch and submit it upstream? 😉

 

Thanks!
Geoffroy

 

From: acrn-users@... <acrn-users@...> On Behalf Of Minggui Cao
Sent: Saturday, April 2, 2022 2:04 am
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes, correct.

 

You can add your code in function: setup_clos

 

There is a data structure: res_cap_info, if possible, you also can re-use it to configure your L2 CAT.

 

Thanks

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Friday, April 1, 2022 2:54 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Yes i already tested on ACRN console, what i'm trying to achieve is a permanent configuration

When you say RDT code. do you mean this below path to code?
hypervisor/arch/x86/rdt.c
jordan.nowak22@...
 

Hi Minggui,

i've tried  https://github.com/projectacrn/acrn-hypervisor/commit/61074ce1ee1e3925699613c92a49db0264ebf4c3
Similar parameter that i define in my board and scenario xmls.

but i run into a trouble in booting up
has this been tested?
see this below logs from ACRN Shell.

================================================

[131610410us][cpu=2][(null)][sev=2][seq=11160]:================================

=

 

[131618981us][cpu=2][(null)][sev=2][seq=11161]:= Unhandled exception: 6 (Invalid Opcode)

 

[131628141us][cpu=2][(null)][sev=2][seq=11162]:

Host Registers:

 

[131635138us][cpu=2][(null)][sev=2][seq=11163]:=  Vector=0x0000000000000006  RIP=0x000000001100E629

 

[131645374us][cpu=2][(null)][sev=2][seq=11164]:=     RAX=0x0000000000000000  RBX=0x0000000000000002  RCX=0x0000000000000002

 

[131657979us][cpu=2][(null)][sev=2][seq=11165]:=     RDX=0x0000000011218000  RDI=0x0000000011219A58  RSI=0x000000000000000A

 

[131670586us][cpu=2][(null)][sev=2][seq=11166]:=     RSP=0x000000001121E798  RBP=0x000000001121E7A8  RBX=0x0000000000000002

 

[131683194us][cpu=2][(null)][sev=2][seq=11167]:=      R8=0x0000000000000002   R9=0x0000000000000000  R10=0x00000000000003F8

 

[131695800us][cpu=2][(null)][sev=2][seq=11168]:=     R11=0x000000001104917A  R12=0x0000000000000002  R13=0x0000000000000002

 

[131708405us][cpu=2][(null)][sev=2][seq=11169]:=  RFLAGS=0x0000000000010046  R14=0x0000000000000002  R15=0x0000000011054140

 

[131721014us][cpu=2][(null)][sev=2][seq=11170]:= ERRCODE=0x0000000000000000   CS=0x0000000000000008   SS=0x0000000000000010

 

[131733623us][cpu=2][(null)][sev=2][seq=11171]:= CR2=0x0000000000000000

[131740911us][cpu=2][(null)][sev=2][seq=11172]:

 

[131746134us][cpu=2][(null)][sev=2][seq=11173]:=====================================================

[131756280us][cpu=2][(null)][sev=2][seq=11174]:===========================

 

[131764065us][cpu=2][(null)][sev=2][seq=11175]:

delta = (actual_load_address - CONFIG_HV_RAM_START) = 0x0

 

[131775292us][cpu=2][(null)][sev=2][seq=11176]:

Host Stack: CPU_ID = 2

 

[131782979us][cpu=2][(null)][sev=2][seq=11177]:addr(0x1121e700) 0x000000001121e7a8  0x000000000000000a  0x0000000011219a58  0x0000000000000002

 

[131797557us][cpu=2][(null)][sev=2][seq=11178]:addr(0x1121e720) 0x0000000000000000  0x00000000000003f8  0x000000001104917a  0x0000000000000002

 

[131812137us][cpu=2][(null)][sev=2][seq=11179]:addr(0x1121e740) 0x0000000000000002  0x0000000000000002  0x0000000011054140  0x0000000000000006

 

[131826717us][cpu=2][(null)][sev=2][seq=11180]:addr(0x1121e760) 0x0000000000000000  0x000000001100e629  0x0000000000000008  0x0000000000010046

 

[131841296us][cpu=2][(null)][sev=2][seq=11181]:addr(0x1121e780) 0x000000001121e798  0x0000000000000010  0x000000001121e7e8  0x0000000000000002

 

[131855876us][cpu=2][(null)][sev=2][seq=11182]:addr(0x1121e7a0) 0xc2af95c22d966d8f  0x000000001121e7d8  0x000000001100919e  0x0000000000000002

 

[131870447us][cpu=2][(null)][sev=2][seq=11183]:addr(0x1121e7c0) 0x0000000000000002  0x0000000000000002  0x0000000000000002  0x000000001121e8b8

 

[131885022us][cpu=2][(null)][sev=2][seq=11184]:addr(0x1121e7e0) 0x0000000011013588  0x0000000000000000  0x0000000000000002  0x0000000011218000

 

[131899593us][cpu=2][(null)][sev=2][seq=11185]:addr(0x1121e800) 0x0000000000000002  0x000000001121e810  0x000000001121e8b8  0x000000000000000a

 

[131914169us][cpu=2][(null)][sev=2][seq=11186]:addr(0x1121e820) 0x0000000011219a58  0x0000000000000002  0x0000000000000000  0x00000000000003f8

 

[131928747us][cpu=2][(null)][sev=2][seq=11187]:addr(0x1121e840) 0x000000001104917a  0x0000000000000002  0x0000000000000002  0x0000000000000002

 

[131943325us][cpu=2][(null)][sev=2][seq=11188]:addr(0x1121e860) 0x0000000011054140  0x0000000000000006  0x0000000000000000  0x000000001100e629

 

[131957900us][cpu=2][(null)][sev=2][seq=11189]:addr(0x1121e880) 0x0000000000000008  0x0000000000010046  0x000000001121e8a8  0x0000000000000010

 

[131972481us][cpu=2][(null)][sev=2][seq=11190]:addr(0x1121e8a0) 0x000000001121e8f8  0x0000000000000002  0xc2af95c22d966d8f  0x000000001121e8e8

 

[131987052us][cpu=2][(null)][sev=2][seq=11191]:addr(0x1121e8c0) 0x000000001100919e  0x0000000000000002  0x0000000000000002  0x0000000000000002

 

[132001627us][cpu=2][(null)][sev=2][seq=11192]:addr(0x1121e8e0) 0x0000000000000002  0x000000001121e9c8  0x0000000011013588  0x0000000000000000

 

[132016204us][cpu=2][(null)][sev=2][seq=11193]:

 

[132021423us][cpu=2][(null)][sev=2][seq=11194]:Host Call Trace:

 

[132028223us][cpu=2][(null)][sev=2][seq=11195]:----> 0x000000001100919e

 

[132035802us][cpu=2][(null)][sev=2][seq=11196]:----> 0x0000000011013588

 

[132043390us][cpu=2][(null)][sev=2][seq=11197]:----> 0x000000001100919e

 

[132050980us][cpu=2][(null)][sev=2][seq=11198]:----> 0x0000000011013588

 

[132058567us][cpu=2][(null)][sev=2][seq=11199]:----> 0x000000001100919e

 

[132066151us][cpu=2][(null)][sev=2][seq=11200]:----> 0x0000000011013588

 

[132073735us][cpu=2][(null)][sev=2][seq=11201]:----> 0x000000001100919e

 

[132081319us][cpu=2][(null)][sev=2][seq=11202]:----> 0x0000000011013588

 

[132088906us][cpu=2][(null)][sev=2][seq=11203]:----> 0x000000001100919e

 

[132096494us][cpu=2][(null)][sev=2][seq=11204]:----> 0x0000000011013588

 

[132104082us][cpu=2][(null)][sev=2][seq=11205]:----> 0x000000001100919e

 

[132111671us][cpu=2][(null)][sev=2][seq=11206]:----> 0x0000000011013588

 

[132119260us][cpu=2][(null)][sev=2][seq=11207]:----> 0x000000001100919e

 

[132126847us][cpu=2][(null)][sev=2][seq=11208]:----> 0x0000000011013588

 

[132134436us][cpu=2][(null)][sev=2][seq=11209]:----> 0x000000001100919e

 

[132142020us][cpu=2][(null)][sev=2][seq=11210]:----> 0x0000000011013588

 

[132149602us][cpu=2][(null)][sev=2][seq=11211]:----> 0x000000001100919e

 

[132157189us][cpu=2][(null)][sev=2][seq=11212]:----> 0x0000000011013588

 

[132164777us][cpu=2][(null)][sev=2][seq=11213]:----> 0x000000001100919e

 

[132172364us][cpu=2][(null)][sev=2][seq=11214]:----> 0x0000000011013588

 

[132179952us][cpu=2][(null)][sev=2][seq=11215]:

 

[132185781us][cpu=3][(null)][sev=2][seq=11216]:
Minggui Cao
 

It depends on your board, you can use wrmsr in ACRN console to try your setting first.

 

Thanks!

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Monday, April 11, 2022 11:32 AM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Hi Minggui,

i've tried  https://github.com/projectacrn/acrn-hypervisor/commit/61074ce1ee1e3925699613c92a49db0264ebf4c3
Similar parameter that i define in my board and scenario xmls.

but i run into a trouble in booting up
has this been tested?
see this below logs from ACRN Shell.

================================================

[131610410us][cpu=2][(null)][sev=2][seq=11160]:================================

=

 

[131618981us][cpu=2][(null)][sev=2][seq=11161]:= Unhandled exception: 6 (Invalid Opcode)

 

[131628141us][cpu=2][(null)][sev=2][seq=11162]:

Host Registers:

 

[131635138us][cpu=2][(null)][sev=2][seq=11163]:=  Vector=0x0000000000000006  RIP=0x000000001100E629

 

[131645374us][cpu=2][(null)][sev=2][seq=11164]:=     RAX=0x0000000000000000  RBX=0x0000000000000002  RCX=0x0000000000000002

 

[131657979us][cpu=2][(null)][sev=2][seq=11165]:=     RDX=0x0000000011218000  RDI=0x0000000011219A58  RSI=0x000000000000000A

 

[131670586us][cpu=2][(null)][sev=2][seq=11166]:=     RSP=0x000000001121E798  RBP=0x000000001121E7A8  RBX=0x0000000000000002

 

[131683194us][cpu=2][(null)][sev=2][seq=11167]:=      R8=0x0000000000000002   R9=0x0000000000000000  R10=0x00000000000003F8

 

[131695800us][cpu=2][(null)][sev=2][seq=11168]:=     R11=0x000000001104917A  R12=0x0000000000000002  R13=0x0000000000000002

 

[131708405us][cpu=2][(null)][sev=2][seq=11169]:=  RFLAGS=0x0000000000010046  R14=0x0000000000000002  R15=0x0000000011054140

 

[131721014us][cpu=2][(null)][sev=2][seq=11170]:= ERRCODE=0x0000000000000000   CS=0x0000000000000008   SS=0x0000000000000010

 

[131733623us][cpu=2][(null)][sev=2][seq=11171]:= CR2=0x0000000000000000

[131740911us][cpu=2][(null)][sev=2][seq=11172]:

 

[131746134us][cpu=2][(null)][sev=2][seq=11173]:=====================================================

[131756280us][cpu=2][(null)][sev=2][seq=11174]:===========================

 

[131764065us][cpu=2][(null)][sev=2][seq=11175]:

delta = (actual_load_address - CONFIG_HV_RAM_START) = 0x0

 

[131775292us][cpu=2][(null)][sev=2][seq=11176]:

Host Stack: CPU_ID = 2

 

[131782979us][cpu=2][(null)][sev=2][seq=11177]:addr(0x1121e700) 0x000000001121e7a8  0x000000000000000a  0x0000000011219a58  0x0000000000000002

 

[131797557us][cpu=2][(null)][sev=2][seq=11178]:addr(0x1121e720) 0x0000000000000000  0x00000000000003f8  0x000000001104917a  0x0000000000000002

 

[131812137us][cpu=2][(null)][sev=2][seq=11179]:addr(0x1121e740) 0x0000000000000002  0x0000000000000002  0x0000000011054140  0x0000000000000006

 

[131826717us][cpu=2][(null)][sev=2][seq=11180]:addr(0x1121e760) 0x0000000000000000  0x000000001100e629  0x0000000000000008  0x0000000000010046

 

[131841296us][cpu=2][(null)][sev=2][seq=11181]:addr(0x1121e780) 0x000000001121e798  0x0000000000000010  0x000000001121e7e8  0x0000000000000002

 

[131855876us][cpu=2][(null)][sev=2][seq=11182]:addr(0x1121e7a0) 0xc2af95c22d966d8f  0x000000001121e7d8  0x000000001100919e  0x0000000000000002

 

[131870447us][cpu=2][(null)][sev=2][seq=11183]:addr(0x1121e7c0) 0x0000000000000002  0x0000000000000002  0x0000000000000002  0x000000001121e8b8

 

[131885022us][cpu=2][(null)][sev=2][seq=11184]:addr(0x1121e7e0) 0x0000000011013588  0x0000000000000000  0x0000000000000002  0x0000000011218000

 

[131899593us][cpu=2][(null)][sev=2][seq=11185]:addr(0x1121e800) 0x0000000000000002  0x000000001121e810  0x000000001121e8b8  0x000000000000000a

 

[131914169us][cpu=2][(null)][sev=2][seq=11186]:addr(0x1121e820) 0x0000000011219a58  0x0000000000000002  0x0000000000000000  0x00000000000003f8

 

[131928747us][cpu=2][(null)][sev=2][seq=11187]:addr(0x1121e840) 0x000000001104917a  0x0000000000000002  0x0000000000000002  0x0000000000000002

 

[131943325us][cpu=2][(null)][sev=2][seq=11188]:addr(0x1121e860) 0x0000000011054140  0x0000000000000006  0x0000000000000000  0x000000001100e629

 

[131957900us][cpu=2][(null)][sev=2][seq=11189]:addr(0x1121e880) 0x0000000000000008  0x0000000000010046  0x000000001121e8a8  0x0000000000000010

 

[131972481us][cpu=2][(null)][sev=2][seq=11190]:addr(0x1121e8a0) 0x000000001121e8f8  0x0000000000000002  0xc2af95c22d966d8f  0x000000001121e8e8

 

[131987052us][cpu=2][(null)][sev=2][seq=11191]:addr(0x1121e8c0) 0x000000001100919e  0x0000000000000002  0x0000000000000002  0x0000000000000002

 

[132001627us][cpu=2][(null)][sev=2][seq=11192]:addr(0x1121e8e0) 0x0000000000000002  0x000000001121e9c8  0x0000000011013588  0x0000000000000000

 

[132016204us][cpu=2][(null)][sev=2][seq=11193]:

 

[132021423us][cpu=2][(null)][sev=2][seq=11194]:Host Call Trace:

 

[132028223us][cpu=2][(null)][sev=2][seq=11195]:----> 0x000000001100919e

 

[132035802us][cpu=2][(null)][sev=2][seq=11196]:----> 0x0000000011013588

 

[132043390us][cpu=2][(null)][sev=2][seq=11197]:----> 0x000000001100919e

 

[132050980us][cpu=2][(null)][sev=2][seq=11198]:----> 0x0000000011013588

 

[132058567us][cpu=2][(null)][sev=2][seq=11199]:----> 0x000000001100919e

 

[132066151us][cpu=2][(null)][sev=2][seq=11200]:----> 0x0000000011013588

 

[132073735us][cpu=2][(null)][sev=2][seq=11201]:----> 0x000000001100919e

 

[132081319us][cpu=2][(null)][sev=2][seq=11202]:----> 0x0000000011013588

 

[132088906us][cpu=2][(null)][sev=2][seq=11203]:----> 0x000000001100919e

 

[132096494us][cpu=2][(null)][sev=2][seq=11204]:----> 0x0000000011013588

 

[132104082us][cpu=2][(null)][sev=2][seq=11205]:----> 0x000000001100919e

 

[132111671us][cpu=2][(null)][sev=2][seq=11206]:----> 0x0000000011013588

 

[132119260us][cpu=2][(null)][sev=2][seq=11207]:----> 0x000000001100919e

 

[132126847us][cpu=2][(null)][sev=2][seq=11208]:----> 0x0000000011013588

 

[132134436us][cpu=2][(null)][sev=2][seq=11209]:----> 0x000000001100919e

 

[132142020us][cpu=2][(null)][sev=2][seq=11210]:----> 0x0000000011013588

 

[132149602us][cpu=2][(null)][sev=2][seq=11211]:----> 0x000000001100919e

 

[132157189us][cpu=2][(null)][sev=2][seq=11212]:----> 0x0000000011013588

 

[132164777us][cpu=2][(null)][sev=2][seq=11213]:----> 0x000000001100919e

 

[132172364us][cpu=2][(null)][sev=2][seq=11214]:----> 0x0000000011013588

 

[132179952us][cpu=2][(null)][sev=2][seq=11215]:

 

[132185781us][cpu=3][(null)][sev=2][seq=11216]:
jordan.nowak22@...
 

Hi Minggui,

The settings that i set in wrmsr through ACRN console is working. 
i'm trying to reproduce this through VM Configuration. 

Below is the xmls that i trying to set, but its weird that ACRN is not booting up after i change the value in xmls.

 

  <CLOS_INFO>

        rdt resources supported: L3

        rdt resource clos max: 4

        rdt resource mask max: '0xfff'

        </CLOS_INFO>

 

 

      <RDT>

        <RDT_ENABLED>y</RDT_ENABLED>

        <CDP_ENABLED>n</CDP_ENABLED>

        <CLOS_MASK>0x0ff</CLOS_MASK>

        <CLOS_MASK>0xf00</CLOS_MASK>

        <CLOS_MASK>0x0ff</CLOS_MASK>

        <CLOS_MASK>0x0ff</CLOS_MASK>

      </RDT>
Minggui Cao
 

I mean you can get L3 caches info from CPUID first.

 

And then you can try cmd in ACRN console:

  rdmsr 0xc90

 

or wrmsr 0xc90 0x0ff

 

thanks!

Minggui

 

From: acrn-users@... <acrn-users@...> On Behalf Of jordan.nowak22@...
Sent: Monday, April 11, 2022 2:14 PM
To: acrn-users@...
Subject: Re: [acrn-users] CAT definition in the scenario xmls

 

Hi Minggui,

The settings that i set in wrmsr through ACRN console is working. 
i'm trying to reproduce this through VM Configuration. 

Below is the xmls that i trying to set, but its weird that ACRN is not booting up after i change the value in xmls.

 

  <CLOS_INFO>

        rdt resources supported: L3

        rdt resource clos max: 4

        rdt resource mask max: '0xfff'

        </CLOS_INFO>

 

 

      <RDT>

        <RDT_ENABLED>y</RDT_ENABLED>

        <CDP_ENABLED>n</CDP_ENABLED>

        <CLOS_MASK>0x0ff</CLOS_MASK>

        <CLOS_MASK>0xf00</CLOS_MASK>

        <CLOS_MASK>0x0ff</CLOS_MASK>

        <CLOS_MASK>0x0ff</CLOS_MASK>

      </RDT>

1 - 13 of 13