Add section describing CPU and memory placement (#1917)

jfehlig · sounix000 · web-flow · commit 45ea4fff5c70 · 2026-04-15T16:29:59.000+05:30
* vt-doc: Move 'Changing boot options' section

The 'Changing boot options' section is sandwiched between 'Configuring CPU'
and 'Configuring memory allocation'. Move it before the section describing
CPU configuration, which more closely matches the flow of upstream domXML
documentation.

Signed-off-by: Jim Fehlig &lt;jfehlig@suse.com&gt;

* vt-doc: Add section describing CPU and memory placement

Add a section describing the considerations of VM CPU and memory placement
on large physical NUMA systems. Include examples of how to configure a VM
to use memory backed by hugepages and get NUMA placement advice from the
numa-preplace binary.

References: jsc#PED-15886

Signed-off-by: Jim Fehlig &lt;jfehlig@suse.com&gt;

* Incorporated Daria's editorial suggestions

---------

Signed-off-by: Jim Fehlig &lt;jfehlig@suse.com&gt;
Co-authored-by: Souvik Sarkar &lt;souvik.sarkar@suse.com&gt;
diff --git a/xml/libvirt_configuration_virsh.xml b/xml/libvirt_configuration_virsh.xml
@@ -205,6 +205,115 @@ Product Name: Standard PC (Q35 + ICH9, 2009)</screen>
       for more information.
     </para>
   </sect1>
+  <sect1 xml:id="sec-libvirt-config-boot-menu-virsh">
+    <title>Changing boot options</title>
+
+    <para>
+      The boot menu of the &vmguest; can be found in the <tag>os</tag> element
+      and looks similar to this example:
+    </para>
+
+<screen>&lt;os&gt;
+  &lt;type&gt;hvm&lt;/type&gt;
+  &lt;loader&gt;readonly='yes' secure='no' type='rom'/&gt;/usr/lib/xen/boot/hvmloader&lt;/loader&gt;
+  &lt;nvram template='/usr/share/OVMF/OVMF_VARS.fd'/&gt;/var/lib/libvirt/nvram/guest_VARS.fd&lt;/nvram&gt;
+  &lt;boot dev='hd'/&gt;
+  &lt;boot dev='cdrom'/&gt;
+  &lt;bootmenu enable='yes' timeout='3000'/&gt;
+  &lt;smbios mode='sysinfo'/&gt;
+  &lt;bios useserial='yes' rebootTimeout='0'/&gt;
+  &lt;/os&gt;</screen>
+
+    <para>
+      In this example, two devices are available: <tag class="attvalue">hd</tag> and <tag
+      class="attvalue">cdrom</tag>. The configuration also reflects the actual boot order, so the
+      <tag class="attvalue">hd</tag> comes before the <tag class="attvalue">cdrom</tag>.
+    </para>
+
+    <sect2 xml:id="sec-libvirt-config-bootorder-virsh">
+      <title>Changing boot order</title>
+      <para>
+        The &vmguest;'s boot order is represented through the order of devices
+        in the XML configuration file. As the devices are interchangeable, it
+        is possible to change the boot order of the &vmguest;.
+      </para>
+      <procedure>
+        <step>
+          <para>
+            Open the &vmguest;'s XML configuration.
+          </para>
+<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
+        </step>
+        <step>
+          <para>
+            Change the sequence of the bootable devices.
+          </para>
+<screen>
+...
+&lt;boot dev='cdrom'/&gt;
+&lt;boot dev='hd'/&gt;
+...
+      </screen>
+        </step>
+        <step>
+          <para>
+            Check if the boot order was changed successfully by looking at the
+            boot menu in the BIOS of the &vmguest;.
+          </para>
+        </step>
+      </procedure>
+    </sect2>
+
+    <sect2 xml:id="sec-libvirt-config-directkernel-virsh">
+      <title>Using direct kernel boot</title>
+      <para>
+        Direct kernel boot allows you to boot from a kernel and initrd stored
+        on the host. Set the path to both files in the <tag>kernel</tag> and
+        <tag>initrd</tag> elements:
+      </para>
+<screen>&lt;os&gt;
+    ...
+  &lt;kernel&gt;/root/f8-i386-vmlinuz&lt;/kernel&gt;
+  &lt;initrd&gt;/root/f8-i386-initrd&lt;/initrd&gt;
+    ...
+&lt;os&gt;</screen>
+      <para>
+        To enable direct kernel boot:
+      </para>
+      <procedure>
+        <step>
+          <para>
+            Open the &vmguest;'s XML configuration:
+          </para>
+<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
+        </step>
+        <step>
+          <para>
+            Inside the <tag>os</tag> element, add a <tag>kernel</tag> element
+            and the path to the kernel file on the host:
+          </para>
+<screen>...
+&lt;kernel&gt;/root/f8-i386-vmlinuz&lt;/kernel&gt;
+...</screen>
+        </step>
+        <step>
+          <para>
+            Add an <tag>initrd</tag> element and the path to the initrd file on
+            the host:
+          </para>
+<screen>...
+&lt;initrd&gt;/root/f8-i386-initrd&lt;/initrd&gt;
+...</screen>
+        </step>
+        <step>
+          <para>
+            Start your VM to boot from the new kernel:
+          </para>
+<screen>&prompt.sudo;<command>virsh start sles15</command></screen>
+        </step>
+      </procedure>
+    </sect2>
+  </sect1>
   <sect1 xml:id="libvirt-cpu-virsh">
     <title>Configuring CPU</title>
 
@@ -392,117 +501,6 @@ current      live           4
       </para>
     </sect2>
   </sect1>
-  <sect1 xml:id="sec-libvirt-config-boot-menu-virsh">
-    <title>Changing boot options</title>
-
-    <para>
-      The boot menu of the &vmguest; can be found in the <tag>os</tag> element
-      and looks similar to this example:
-    </para>
-
-<screen>&lt;os&gt;
-  &lt;type&gt;hvm&lt;/type&gt;
-  &lt;loader&gt;readonly='yes' secure='no' type='rom'/&gt;/usr/lib/xen/boot/hvmloader&lt;/loader&gt;
-  &lt;nvram template='/usr/share/OVMF/OVMF_VARS.fd'/&gt;/var/lib/libvirt/nvram/guest_VARS.fd&lt;/nvram&gt;
-  &lt;boot dev='hd'/&gt;
-  &lt;boot dev='cdrom'/&gt;
-  &lt;bootmenu enable='yes' timeout='3000'/&gt;
-  &lt;smbios mode='sysinfo'/&gt;
-  &lt;bios useserial='yes' rebootTimeout='0'/&gt;
-  &lt;/os&gt;</screen>
-
-    <para>
-      In this example, two devices are available,
-      <tag class="attvalue">hd</tag> and <tag class="attvalue">cdrom</tag> .
-      The configuration also reflects the actual boot order, so the
-      <tag class="attvalue">hd</tag> comes before the
-      <tag class="attvalue">cdrom</tag> .
-    </para>
-
-    <sect2 xml:id="sec-libvirt-config-bootorder-virsh">
-      <title>Changing boot order</title>
-      <para>
-        The &vmguest;'s boot order is represented through the order of devices
-        in the XML configuration file. As the devices are interchangeable, it
-        is possible to change the boot order of the &vmguest;.
-      </para>
-      <procedure>
-        <step>
-          <para>
-            Open the &vmguest;'s XML configuration.
-          </para>
-<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
-        </step>
-        <step>
-          <para>
-            Change the sequence of the bootable devices.
-          </para>
-<screen>
-...
-&lt;boot dev='cdrom'/&gt;
-&lt;boot dev='hd'/&gt;
-...
-      </screen>
-        </step>
-        <step>
-          <para>
-            Check if the boot order was changed successfully by looking at the
-            boot menu in the BIOS of the &vmguest;.
-          </para>
-        </step>
-      </procedure>
-    </sect2>
-
-    <sect2 xml:id="sec-libvirt-config-directkernel-virsh">
-      <title>Using direct kernel boot</title>
-      <para>
-        Direct Kernel Boot allows you to boot from a kernel and initrd stored
-        on the host. Set the path to both files in the <tag>kernel</tag> and
-        <tag>initrd</tag> elements:
-      </para>
-<screen>&lt;os&gt;
-    ...
-  &lt;kernel&gt;/root/f8-i386-vmlinuz&lt;/kernel&gt;
-  &lt;initrd&gt;/root/f8-i386-initrd&lt;/initrd&gt;
-    ...
-&lt;os&gt;</screen>
-      <para>
-        To enable Direct Kernel Boot:
-      </para>
-      <procedure>
-        <step>
-          <para>
-            Open the &vmguest;'s XML configuration:
-          </para>
-<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
-        </step>
-        <step>
-          <para>
-            Inside the <tag>os</tag> element, add a <tag>kernel</tag> element
-            and the path to the kernel file on the host:
-          </para>
-<screen>...
-&lt;kernel&gt;/root/f8-i386-vmlinuz&lt;/kernel&gt;
-...</screen>
-        </step>
-        <step>
-          <para>
-            Add an <tag>initrd</tag> element and the path to the initrd file on
-            the host:
-          </para>
-<screen>...
-&lt;initrd&gt;/root/f8-i386-initrd&lt;/initrd&gt;
-...</screen>
-        </step>
-        <step>
-          <para>
-            Start your VM to boot from the new kernel:
-          </para>
-<screen>&prompt.sudo;<command>virsh start sles15</command></screen>
-        </step>
-      </procedure>
-    </sect2>
-  </sect1>
   <sect1 xml:id="sec-libvirt-config-memory-virsh">
     <title>Configuring memory allocation</title>
 
@@ -612,6 +610,111 @@ Used memory:    8388608 KiB
       </para>
     </important>
   </sect1>
+  <sect1 xml:id="sec-libvirt-config-cpu-memory-virsh">
+    <title>CPU and Memory configuration and placement on large NUMA systems</title>
+
+    <para>
+      When deploying &vmguest;s with large CPU and memory allocations on large
+      &vmhost;s with multiple NUMA nodes, it may be necessary for both
+      performance and resource planning to tune the &vmguest;'s allocations.
+    </para>
+
+    <sect2 xml:id="sec-libvirt-config-hugepages-virsh">
+      <title>Huge pages</title>
+      <para>
+        Using huge pages from the &vmhost; to back memory allocations for the &vmguest; can improve
+        performance for many workloads. The following procedure shows how to back &vmguest; memory
+        with huge pages. The &vmguest; should be shut off or restarted after changing the configuration.
+      </para>
+      <procedure>
+        <step>
+          <para>
+            Edit the &vmguest; configuration:
+          </para>
+<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
+        </step>
+        <step>
+          <para>
+            Add the <tag>memoryBacking</tag> element under the top level
+            <tag>domain</tag> element:
+          </para>
+          <screen>&lt;memoryBacking&gt;
+            &lt;hugePages/&gt;
+            &lt;memoryBacking/&gt;
+          </screen>
+        </step>
+        <step>
+          <para>
+            If not running, start the &vmguest;, otherwise restart.
+          </para>
+        </step>
+      </procedure>
+      <para>
+        For more information on advanced <tag>memoryBacking</tag> settings, see
+        the <citetitle>Memory Backing</citetitle> documentation at
+        <link xlink:href="https://libvirt.org/formatdomain.html#memory-backing"/>.
+      </para>
+    </sect2>
+    <sect2>
+      <title>Automatic NUMA placement of &vmguest;</title>
+      <para>
+        When deploying large VMs on large physical NUMA systems, one challenge
+        is ensuring the &vmguest; CPU and memory resources are not randomly
+        spread across the &vmhost;'s NUMA nodes; otherwise, performance may
+        suffer. All resources should be close to each other with respect to
+        NUMA distance. To ensure best performance, the &vmguest;'s CPU and
+        memory allocations should fit on a single &vmhost; NUMA node.
+      </para>
+
+      <para>
+        &libvirt; can automatically place &vmguest; CPU and memory resources
+        using the external <command>numa-preplace</command> program. To activate
+        automatic placement, ensure the <tag class="attvalue">placement</tag>
+        attribute on the <tag>cpu</tag> element is set to <literal>auto</literal>.
+        Memory placement should also be set to <literal>auto</literal> using
+        the <tag>numatune</tag> element. The &vmguest; should be shutoff or
+        restarted after changing the configuration.
+      </para>
+      <procedure>
+        <step>
+          <para>
+            Edit the &vmguest; configuration:
+          </para>
+<screen>&prompt.sudo;<command>virsh edit sles15</command></screen>
+        </step>
+        <step>
+          <para>
+            Add the <tag>numatune</tag> element under the top level
+            <tag>domain</tag> element, and set the memory mode to
+            <literal>auto</literal>:
+          </para>
+          <screen>&lt;numatune&gt;
+            &lt;memory mode="auto"/&gt;
+            &lt;numatune/&gt;
+          </screen>
+        </step>
+        <step>
+          <para>
+            If not running, start the &vmguest;, otherwise restart.
+          </para>
+        </step>
+      </procedure>
+      <important>
+        <title>Automatic NUMA placement</title>
+        <para>
+          Automatic NUMA placement is only available if the
+          <package>numa-preplace</package> package is installed. Automatic
+          NUMA placement with <command>numa-preplace</command> works with
+          both normal memory and hugepages.
+        </para>
+      </important>
+      <para>
+        For more information on advanced <tag>numatune</tag> settings, see the
+        <citetitle>NUMA Node Tuning</citetitle> documentation at
+        <link xlink:href="https://libvirt.org/formatdomain.html#numa-node-tuning"/>.
+      </para>
+    </sect2>
+  </sect1>
   <sect1 xml:id="sec-libvirt-config-pci-virsh">
     <title>Adding a PCI device</title>
 
