Ensure RMT conforms with sles 16 immutability requirements

[ngetahun]

Description

This PR aims to migrate RMT directory and symlink management from RPM %post scripts to systemd-tmpfiles for SLES 16 Immutability compatibility, read more here.

• Immutability: Ensures /var provisioning works when /usr is read-only.

• Fresh Installs: Enables "stateless" image building; /var/lib/rmt is created at runtime, not during RPM install.

• Upgrades: Seamlessly handles existing RMT 2 data in /var/lib/rmt while ensuring correct _rmt:nginx permissions.

• Related Issue / Ticket / Trello card:

How to test

1. Build the RPM.

# before starting, make sure to clone rmt-server-3 in your obs workspace 
> cd <obs workspace> && osc co https://build.opensuse.org/package/show/systemsmanagement:SCC:RMT/rmt-server-3
# In rmt repo, build the package
> make dist && cp package/obs/* <obs workspace>/systemsmanagement\:SCC\:RMT/rmt-server-3/
> osc build 16.0 x86_64 -k ..
# built rpm package are in ../

2. Install on a fresh system; verify /var/lib/rmt is created via systemd-tmpfiles.
3. Verify symlinks in /usr/share/rmt/public/ point correctly to /var/lib/rmt/public/.
4. Perform an upgrade from an older RMT version and verify existing mirrored data remains intact.

Change Type

• Bug Fix (a non-breaking change which fixes an issue)
• New Feature (a non-breaking change which adds new functionality)
• Documentation Update (a change which only updates documentation)

Checklist

• I have reviewed my own code and believe that it's ready for an external review.
• I have provided comments for any hard-to-understand code.
• I have documented the MANUAL.md file with any changes to the user experience.
• If my changes are non-trivial, I have added a changelog entry to notify users at package/obs/rmt-server.changes.

Review

Please check out our review guidelines
and get in touch with the author to get a shared understanding of the change.

[arharovets]

isn't /usr read-only on an immutable system? if yes then writes (system_uuid, logs, cache config) to it would fail, right?

[ngetahun]

isn't /usr read-only on an immutable system? if yes then writes (system_uuid, logs, cache config) to it would fail, right?

That is exactly the point, installation should be idempotent and that is attained through the use of immutable file systems. Read: https://confluence.suse.com/pages/viewpage.action?pageId=1989378136&spaceKey=~RBrownSUSE&title=Packaging%2Bfor%2BImmutable%2BMode%2B-%2BBest%2BPractices for more information

[arharovets]

That is exactly the point, installation should be idempotent and that is attained through the use of immutable file systems.

This does not address my concern, as idempotent installation and runtime writability are two different things, and my concern is about runtime:
rmt tries to write system_uuid, zypper_auth.log, rmt-cache-trim.sh and regsharing state and those writes are now pointing to /usr/share/rmt/

rmt-tmpfiles.conf provisions /var/lib/rmt correctly for exactly this purpose but then we change the code to not use it.

The doc you shared only confirms this:

If content must be created outside of /usr or /etc the best solution is to use systemd-tmpfiles to create these files and directories:

• Create a /usr/lib/tmpfiles.d/.conf file which copies or links the file to /var

Basically:

• /usr/share/rmt - immutable, shipped in an RPM, for read-only app files;
• /var/lib/rmt - created by systemd-tmpfiles at boot, holds writable runtime data.

[ngetahun]

That is exactly the point, installation should be idempotent and that is attained through the use of immutable file systems.

This does not address my concern, as idempotent installation and runtime writability are two different things, and my concern is about runtime: rmt tries to write system_uuid, zypper_auth.log, rmt-cache-trim.sh and regsharing state and those writes are now pointing to /usr/share/rmt/

rmt-tmpfiles.conf provisions /var/lib/rmt correctly for exactly this purpose but then we change the code to not use it.

The doc you shared only confirms this:

If content must be created outside of /usr or /etc the best solution is to use systemd-tmpfiles to create these files and directories:

• Create a /usr/lib/tmpfiles.d/.conf file which copies or links the file to /var

Basically:

• /usr/share/rmt - immutable, shipped in an RPM, for read-only app files;
• /var/lib/rmt - created by systemd-tmpfiles at boot, holds writable runtime data.

You are right except the purpose of tmpfile is also for migrating from rmt 2 to rmt 3. During installs, we run post-installation actions which reference /var/lib/rmt which need be linked. Post upgrade, rmt should use /usr/share/rmt for all actions that use /var/lib/rmt in the subsequent actions, we will completely migrate data from /var to /usr

[digitaltom]

Basically:

/usr/share/rmt - immutable, shipped in an RPM, for read-only app files;
/var/lib/rmt - created by systemd-tmpfiles at boot, holds writable runtime data. 

That's how I understand the concept, too.
So I don't understand why this pr moves the locations where (at RMT runtime) files are getting created from /var/lib/ to /usr/share, because as I understand it, /usr/share is not writable at runtime. Examples: RMT_REGSHARING_DEFAULT_DATA_DIR, data_dir, '/usr/share/rmt/zypper_auth.log', UUID_FILE_LOCATION...

[arharovets]

That is exactly the point, installation should be idempotent and that is attained through the use of immutable file systems.

This does not address my concern, as idempotent installation and runtime writability are two different things, and my concern is about runtime: rmt tries to write system_uuid, zypper_auth.log, rmt-cache-trim.sh and regsharing state and those writes are now pointing to /usr/share/rmt/

rmt-tmpfiles.conf provisions /var/lib/rmt correctly for exactly this purpose but then we change the code to not use it.

The doc you shared only confirms this:

If content must be created outside of /usr or /etc the best solution is to use systemd-tmpfiles to create these files and directories:

• Create a /usr/lib/tmpfiles.d/.conf file which copies or links the file to /var

Basically:

• /usr/share/rmt - immutable, shipped in an RPM, for read-only app files;
• /var/lib/rmt - created by systemd-tmpfiles at boot, holds writable runtime data.

You are right except the purpose of tmpfile is also for migrating from rmt 2 to rmt 3. During installs, we run post-installation actions which reference /var/lib/rmt which need be linked. Post upgrade, rmt should use /usr/share/rmt for all actions that use /var/lib/rmt in the subsequent actions, we will completely migrate data from /var to /usr

currently, the code is trying to write directly into /usr/share/rmt/* which is read-only on an immutable system. If the assumption is that it will follow a symlink to writeable /var/lib/rmt/* then we're missing symlinks for:

• /usr/share/rmt/system_uuid -> /var/lib/rmt/system_uuid
• /usr/share/rmt/zypper_auth.log -> /var/lib/rmt/zypper_auth.log
• /usr/share/rmt/rmt-cache-trim.sh -> .../rmt-cache-trim.sh
• /usr/share/rmt/regsharing -> .../regsharing

If we're not planning to introduce these then we have to keep writing directly to /var/lib/rmt

[felixsch]

I agree with @digitaltom and @arharovets here. Runtime data generally lives in /var and not in /usr. At install time /var is not available and needs to create those directories via the tempfile later.

What made the switch to /usr neccessary, since /usr is the immutable part and /var is not? The spec file which tries to create these directories/files?