Bugfix malloc upload

[folke-nordunet]

Summary

This PR improves large file upload handling and makes the upload UI more reliable while files are being stored and processed by the backend.

The main change is that uploaded files are no longer read fully into memory before being sent to the backend. Instead, uploads are saved temporarily to disk and streamed from there. This reduces memory pressure and avoids malloc/RAM issues when uploading large files.

Main changes

• Stream uploaded files from temporary disk files instead of loading them fully into memory.
• Clean up temporary upload files after each upload attempt.
• Make the multipart spool size configurable via settings.
• Keep the upload dialog open until the backend upload/storage step has completed.
• Improve upload status messages so users can see when files are being saved locally and when they are being stored/encrypted by the backend.
• Avoid clearing the jobs table during temporary empty responses from the backend, which can happen while large uploads are being processed.
• Restore initial table loading and polling behavior after the table refresh changes.

Why

Large uploads could previously cause excessive memory usage because files were handled as in-memory byte objects. This PR changes the upload flow to be disk-backed, making uploads more robust for large files and improving the user experience during longer upload/storage operations.

[Copilot]

Pull request overview

This PR improves robustness of large-file uploads and job table polling by switching uploads to a disk-backed workflow, making multipart spooling configurable, and adjusting UI behavior to remain responsive while backend processing completes.

Changes:

• Stream uploads to the backend from temporary disk files (instead of reading full file bytes into memory) and clean up temp files after upload attempts.
• Make Starlette multipart spooling size configurable via settings and refine upload dialog status messaging/flow.
• Adjust home page polling to avoid clearing the jobs table during temporary empty responses and tweak initial loading/polling behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
utils/settings.py	Adds a configurable setting for multipart spool max size.
utils/common.py	Reworks upload handling to be disk-backed/streamed and updates upload UI flow/status updates.
pages/home.py	Updates jobs table polling/initial load to reduce UI churn during backend processing.

Comments suppressed due to low confidence (1)

utils/common.py:892

• post_file catches HTTPStatusError and calls ui.notify, but handle_upload_with_feedback already shows success/failure notifications. This can lead to duplicate error notifications and (since post_file is called from a background task) the ui.notify here may run outside a with client: context. Consider removing UI side effects from post_file (return an error/False) and let the caller notify within the client context.

            if response.status_code != 200:
                raise httpx.HTTPStatusError(
                    f"Upload failed, status code: {response.status_code}",
                    request=response.request,
                    response=response,
                )
    except httpx.HTTPStatusError as e:
        ui.notify(
            f"Error when uploading file: {str(e)}", type="negative", position="top"
        )
        return False

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[krihal]

Thanks. I've been trying to keep everything in memory in order to not risk leaving any files on disk with potential sensitive data. Can you try the feature_upload_memory_bug branch and see if that one solves the problem for you?