Skip to content

Advanced submitter settings#3141

Open
Alxiice wants to merge 7 commits into
developfrom
feat/advanced_submitter_settings
Open

Advanced submitter settings#3141
Alxiice wants to merge 7 commits into
developfrom
feat/advanced_submitter_settings

Conversation

@Alxiice

@Alxiice Alxiice commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Description

Add a system to setup advanced settings on submitter

Before this PR there was 2 issues :

  • cpu/ram/gpu are the same for preprocess/process/postprocess
  • it's quite difficult to add new parameters in a proper way

Also for the mrSubmitters submitter it was really challenging to setup a proper job for Windows.

Details

In this PR we are adding 2 key elements :

  • SubmitionSettings & StageSettings classes to describe settings for a specific processing stage (preprocess, process/processChunk and postprocess)
  • A new method on desc.BaseNode : getSubmitionSettings. By default it is built using the nodeDesc : cpu/ram/gpu are settings used for the process stage only, pre/post process are using default settings.

Features

  • by default use the same behaviour as currently, except for preprocess/postprocess stages that use default parameters which is more logical.
  • by stage we can set :
    • cpu, ram, gpu
    • a setup/teardown commands that the submitter can use
    • a command wrapper
  • the StageSettings is built using SimpleNamespace so we can technically set anything on a stage settings and use it later on the submitter.

Example

class TestWindowsNode(desc.Node):
    def getSubmitSettings(self, node) -> desc.SubmitionSettings:
        nodeSubmitSettings = desc.SubmitionSettings(node)
        nodeSubmitSettings.process.env_key = {
            "PROD_ROOT": self.get_win_path("P:\\my_prod"),
            "PROD_TMP": self.get_win_path("P:\\my_prod\\tmp"),
            "PYTHONPATH": convert_to_windows(os.getenv("PYTHONPATH")),
            "USER": "me",
        }
        nodeSubmitSettings.process.target_os = "windows"
        nodeSubmitSettings.process.wrapCommand = windowsCommandWrapper
        nodeSubmitSettings.process.setup_command = get_mount_volumes_command()
        return nodeSubmitSettings

    def processChunk(self, chunk):
        ...

@Alxiice Alxiice self-assigned this Jun 18, 2026
@Alxiice Alxiice added the feature new feature (proposed as PR or issue planned by dev) label Jun 18, 2026
@Alxiice Alxiice added this to the Meshroom 2026.1.0 milestone Jun 18, 2026
Comment thread meshroom/core/desc/node.py Fixed
Comment thread meshroom/core/desc/node.py Fixed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces submission and stage settings classes (MeshroomCommandWrapper, StageSettings, and SubmitionSettings) to manage remote computing configurations for nodes, alongside type annotation and naming refactors in OrderedTask. Feedback focuses on correcting spelling typos (e.g., renaming SubmitionSettings to SubmissionSettings), fixing potential AttributeErrors when accessing .value on level parameters, correcting typos in deprecation warnings, and removing redundant attribute assignments in StageSettings due to its inheritance from SimpleNamespace.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread meshroom/core/desc/node.py Outdated
Comment thread meshroom/core/desc/node.py Outdated
Comment thread meshroom/core/desc/node.py
Comment thread meshroom/core/desc/node.py Outdated
Comment thread meshroom/core/desc/node.py Outdated
Comment thread meshroom/core/desc/__init__.py Outdated
Comment thread meshroom/core/desc/node.py Outdated
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.52174% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.96%. Comparing base (6d610e0) to head (ae77fb7).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
meshroom/core/desc/node.py 28.57% 20 Missing ⚠️

❌ Your patch check has failed because the patch coverage (56.52%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3141      +/-   ##
===========================================
- Coverage    86.09%   85.96%   -0.14%     
===========================================
  Files           80       80              
  Lines        12127    12154      +27     
===========================================
+ Hits         10441    10448       +7     
- Misses        1686     1706      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread meshroom/core/desc/node.py Outdated
Comment thread meshroom/core/submitter.py Outdated
@Alxiice
Alxiice force-pushed the feat/advanced_submitter_settings branch 2 times, most recently from aff904a to 4b56ddb Compare June 22, 2026 14:41
@Alxiice
Alxiice force-pushed the feat/advanced_submitter_settings branch 4 times, most recently from a170be1 to f563ef2 Compare July 1, 2026 08:17
@Alxiice
Alxiice force-pushed the feat/advanced_submitter_settings branch 3 times, most recently from 2b94bcc to 7264272 Compare July 7, 2026 21:33
@Alxiice
Alxiice force-pushed the feat/advanced_submitter_settings branch from 7264272 to 63fce55 Compare July 15, 2026 16:12
@cbentejac
cbentejac force-pushed the feat/advanced_submitter_settings branch from 63fce55 to ae77fb7 Compare July 23, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature new feature (proposed as PR or issue planned by dev)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants