Feature/multiserver plugin

[Muddyblack]

Reference: #3420

Summary

This PR adds the multiserver plugin to distribute a single Netlab topology across multiple physical servers.
Sadly for now containerlab-provider only.

Key Details

• Self-contained: The implementation is entirely within netsim/extra/multiserver/ and doesn't modify any core Netlab engine logic.
• Consistent Allocations: IP, interface, and VNI allocations are computed once on the workstation. The plugin then generates a per-server directory with a filtered clab.yml and netlab.snapshot.pickle.
• Native Remote Deployments: Remote servers launch using standard sudo netlab up --snapshot -vv without needing custom CLI options.

For the test-files I am not sure if they make any sense. But they show at least it does not interfere with the normal netlab workflow.

Explanations on how it works can be found in docs/plugins/multiserver.md.

[ipspace]

Super-awesome-job!!! Thanks a million.

Tons of comments (as you expected ;). Some of them are just suggestions or pointers to existing helper functions, in other cases I think we can make the whole thing a lot more streamlined with significant rewrites.

[Muddyblack]

I guess about 99% of them are valid 😁
I will have a look at it next week 👍

[Muddyblack]

Refactor & review-response changelog

clab.yml generation

• Dropped ~200 lines of hand-rolled Python clab generation (_build_server_clab & co).
• output() now filters the topology and renders via the standard clab.j2 template.
• Cross-server P2P links carry a link.clab.vxlan attribute; clab.j2 got one extra
  branch under node_count == 2. Regular topologies are unaffected.

Data types

• Added must_be_group_id and must_be_node_or_group (mirroring must_be_node_id).
• Schema now validates groups (group_id), members (node_id), replicate (node_or_group).
• Removed the manual existence-check loops in _resolve_assignments /
  _resolve_replicated; they only expand references now.

Servers as a dictionary

• multiserver.servers is a name-keyed dict — duplicate names impossible by construction.
• Dropped the seen_ids loop; _validate_servers uses _dataplane IDs (like VLANs/VRFs).

CLI hooks for VXLAN setup/teardown

• Hooks (post_start_clab / pre_stop_clab) are baked into each server's snapshot with a
  relative vxlan-setup.sh path, so they fire locally where netlab up --snapshot runs.

Single-server netlab up --snapshot fixes

• _classify_links marked a link cross-server if endpoints spanned >1 server,
  ignoring clab.uplink — uplink NIC + VXLAN on the same bridge, no STP →
  broadcast storm → 100 % CPU. Fix: clab.uplink links are now forced cross: False.
• set -e + bare ip link add vxlanN aborted on "File exists" after a crash.
  Fix: ip link del vxlanN 2>/dev/null || true before each add in
  vxlan-setup.j2 / generated script.
• New multiserver.vxlan.auto_start bool (default true). When false, scripts
  are generated but CLI hooks are not registered — tunnels stay down until run
  manually (staged convergence or single-server local testing).

Remote-server snapshots (superseded by _resolve_snapshot_paths above)

• make_paths_absolute() no longer touches per-server snapshots.

Bug fixes

• Fixed two Box crashes in _build_server_topo (default_box=True missing → BoxKeyError).
• Box cleanup per AGENTS.md: to_dict() and dotted-path .get().
• Module config render failed in per-server dirs ('frr.j2' not found, OSPF/etc.):
  snapshot load skips make_paths_absolute(), so a bare relative templates dir
  reached render_template(), whose "not .//" branch re-based it onto the netlab
  install dir → in-template {% import %} lookups missed. Fix: _resolve_snapshot_paths
  now anchors relative entries with ./ via _cwd_relative() (stays portable across hosts).

Tests & docs

• Merged srlinux/frr coverage into multiserver-explicit.yml; dropped the separate test.
• docs/plugins/multiserver.md updated for the dict format, CLI hooks, and clab.j2 flow.

Open / outlook

• Waiting on a core post-output callback (offered by maintainer) to replace the atexit
  file-copy handler.
• Future: optional remote orchestration — control node pushes dirs and runs netlab
  remotely (SSH), so the whole lab comes up from one command. Out of scope here.

---

I am not sure on the package: (netlab installation path) is it baked into the snapshot file?

This would require the user to have on all their machines the same path where netlab is installed onto.
Although I would say usually you do similar setups of your machines anyways (just cloning them) .

if "package:" in entry:
  resolved.append(str(moddir / entry.replace("package:", "")))

Also unhappy with _cwd_relative and _resolve_snapshot_paths

[ipspace]

A few quick thoughts -- more details after I implement the hooks this needs and we rebase it.

Added must_be_group_id and must_be_node_or_group (mirroring must_be_node_id).

Will cherry-pick this change into a new PR. Don't want to have this hidden in a large blob of unrelated code.

• Hooks (post_start_clab / pre_stop_clab) are baked into each server's snapshot with a
  relative vxlan-setup.sh path, so they fire locally where netlab up --snapshot runs.

👍

• New multiserver.vxlan.auto_start bool (default true). When false, scripts
  are generated but CLI hooks are not registered — tunnels stay down until run
  manually (staged convergence or single-server local testing).

👍

• make_paths_absolute() no longer touches per-server snapshots.

That's a huge can of worms. We need the absolute paths in the snapshot and in the Ansible inventory. I think it would be best to have a plugin hook executed very early in "netlab up" so it can adjust the topology data and recreate the snapshot and Ansible inventory before "netlab up" does some real work.

I am not sure on the package: (netlab installation path) is it baked into the snapshot file?

It's resolved from the current installation path in make_paths_absolute

This would require the user to have on all their machines the same path where netlab is installed onto. Although I would say usually you do similar setups of your machines anyways (just cloning them) .

Unless you install netlab as root on lab VMs and in virtual environment on your local machine.

Also unhappy with _cwd_relative and _resolve_snapshot_paths

Agreed. That whole thing has to be solved in a different way.

[ipspace]

A mixed bag of nits, things that should be fixed (more info on replicated nodes, default server uplink interface name...), and Another Grand Idea (😜). I'm perfectly fine if you tell me to defer the Grand Idea to a later time and just merge this thing.

[ipspace]

The "post-output" callback is there. Please use that.

[ipspace]

@Muddyblack -- I'm sorry for the long delay. I wanted to push the June release out before Autocon5 (and it turns out one has a limited daily amount of mental energy as one gets older), and then Autocon5 struck (just joking, it was a great conference).

Anyway, I think we should implement multi-access links with bridge nodes in the long run to make your life easier. I could even make that functionality part of netlab core (in the "bridge" role) to have it available regardless of whether the user uses this plugin. Hey, maybe I could even do that by default just to simplify containerlab provisioning (no need for "manual" creation of Linux bridges).

Whether we do that as part of this PR, or merge this and then work on that, is completely up to you. Just let me know ;)

[ipspace]

Update: the "implement clab multi-access networks with bridge nodes" idea is much harder to implement than I thought if we want to support VLAN trunks or routed VLANs across multi-access networks -- we would have to split the multi-access link into P2P links very late in the transformation process.

For the moment, let's fix the other stuff and move forward with this. When I implement that other idea, the multi-access part of the code in this PR will become irrelevant, at which point we can do a cleanup.

[Muddyblack]

No Problem some time off the computer and charge energy is also good for me ;)
Autocon would had been nice especially as it was in Munich this year I forgot about it after some time 😅
Maybe I find online material about it tho.

[Muddyblack]

Just a note:
I committed for example only one doc md file but the entire checks did run though.

Is that intended?
Check integration tests or ruff linting does not have to run if no python file was touched.
Nothing big just a remark as it might be a waste of water xD

[Muddyblack]

Just a note: I committed for example only one doc md file but the entire checks did run though.

Is that intended? Check integration tests or ruff linting does not have to run if no python file was touched. Nothing big just a remark as it might be a waste of water xD

Hm I see it uses the entire pr not just latest commits or so and would require a own job or so.