diff --git a/docs/network-upgrades/berkeley/index.mdx b/docs/network-upgrades/berkeley/index.mdx new file mode 100644 index 000000000..a2f82c3fe --- /dev/null +++ b/docs/network-upgrades/berkeley/index.mdx @@ -0,0 +1,41 @@ +--- +title: Berkeley Upgrade +sidebar_label: Berkeley Upgrade +hide_title: true +description: Overview of the Berkeley network upgrade for Mina Protocol — the transition to the Berkeley proof system with zkApp support. +keywords: + - Berkeley + - upgrade + - hard fork + - zkApps +--- + +# Berkeley Upgrade + +The Berkeley upgrade was the most significant network upgrade in Mina's history, transitioning mainnet from the legacy proof system to the Berkeley proof system. It was completed in June 2024. + +## What Berkeley Introduced + +### zkApp Programmability + +Berkeley brought full zkApp (zero-knowledge application) support to mainnet. Developers can deploy smart contracts that execute off-chain computation and generate zero-knowledge proofs verified on-chain, enabling privacy-preserving applications with minimal on-chain footprint. + +### Recursive Proofs + +The upgrade enabled recursive proof composition, allowing proofs to verify other proofs. This is foundational to Mina's constant-size blockchain — the entire chain state can be verified with a single proof regardless of history length. + +### New Transaction Model + +Berkeley introduced a new transaction format supporting zkApp commands alongside traditional payment and delegation transactions. This included on-chain state storage for smart contracts, events, and actions. + +### Archive Database Migration + +The upgrade required a full archive database migration from the legacy schema to the Berkeley schema. This was the most operationally intensive part of the upgrade for archive node operators, taking up to 48 hours for the trustless migration path. + +## Upgrade Details + +- **Release**: [3.0.0](https://github.com/MinaProtocol/mina/releases/tag/3.0.0) (Berkeley mainnet release) +- **Upgrade mode**: Manual only (automode was not available for Berkeley) +- **Archive migration**: Trustless (48h) or trustful (o1Labs database export) + +For the operational details of the Berkeley upgrade, see the sub-pages below. These are preserved for historical reference. diff --git a/docs/network-upgrades/index.mdx b/docs/network-upgrades/index.mdx index 6ca6768bb..e533ec5a6 100644 --- a/docs/network-upgrades/index.mdx +++ b/docs/network-upgrades/index.mdx @@ -1,18 +1,19 @@ --- title: Network Upgrades sidebar_label: Network Upgrades -description: History of Mina protocol network upgrades on mainnet. +hide_title: true +description: History of Mina Protocol mainnet network upgrades (hard forks), including Berkeley and Mesa. keywords: - - mina network upgrades - - berkeley upgrade - - mesa upgrade + - Mina + - network upgrade - hard fork - - protocol upgrade + - Berkeley + - Mesa --- # Network Upgrades -Mina protocol evolves through network upgrades (hard forks) that introduce new features and improvements. Each upgrade requires node operators to update their software to remain compatible with the network. +Mina Protocol evolves through network upgrades (hard forks). Each upgrade introduces new protocol features, performance improvements, or governance changes. Hard forks are not backward compatible — all node operators must upgrade before the fork activates. :::tip Mesa preflight hard fork completed diff --git a/docs/network-upgrades/mesa/appendix/archive-node-schema-changes.mdx b/docs/network-upgrades/mesa/appendix/archive-node-schema-changes.mdx new file mode 100644 index 000000000..26bed9243 --- /dev/null +++ b/docs/network-upgrades/mesa/appendix/archive-node-schema-changes.mdx @@ -0,0 +1,71 @@ +--- +title: Archive Node Schema Changes +sidebar_label: Archive Node Schema Changes +hide_title: true +description: Reference of database schema changes applied to the Mina archive node during the Mesa hard fork — new columns, modified tables, and the SQL applied by the upgrade script. +keywords: + - Mesa + - upgrade + - appendix + - archive + - schema + - database +--- + +# Archive Node Schema Changes + +## Upgrading archive nodes from Berkeley to Mesa + +Below we present details of what changed in the archive node database schema between Berkeley and Mesa versions. + +### Extended zkApp State Fields + +Both zkApp state tables have been modified to support additional state elements: + +**zkapp_states_nullable table** + - Added columns `element8` through `element31` (nullable integer fields) + - Each new column references `zkapp_field(id)` + - These fields allow zkApps to store additional state information beyond the original 8 elements + +```sql +ALTER TABLE zkapp_states_nullable ADD COLUMN IF NOT EXISTS element8 INT REFERENCES zkapp_field(id); +... +ALTER TABLE zkapp_states_nullable ADD COLUMN IF NOT EXISTS element31 INT REFERENCES zkapp_field(id); +``` + +**zkapp_states table** + - Added columns `element8` through `element31` (non-nullable integer fields) + - Each new column references `zkapp_field(id)` with a default value pointing to the zero field + - Unlike the nullable version, these fields are required and default to the zero field ID + +```sql +ALTER TABLE zkapp_states ADD COLUMN IF NOT EXISTS element8 INT DEFAULT NOT NULL REFERENCES zkapp_field(id); +... +ALTER TABLE zkapp_states ADD COLUMN IF NOT EXISTS element31 INT DEFAULT NOT NULL REFERENCES zkapp_field(id); +``` + +This expansion allows zkApps to store up to 32 state elements instead of the previous 8, significantly increasing the state storage capacity for complex smart contracts. + +### Version Tracking + +The upgrade introduces a new `migration_history` table to keep track of the database schema version. The purpose of this table is to help with future database migrations. The table tracks which migration scripts were applied and when. + +**migration_history table** +```sql +CREATE TABLE IF NOT EXISTS migration_history ( + commit_start_at timestamptz NOT NULL DEFAULT now() PRIMARY KEY, + protocol_version text NOT NULL, + migration_version text NOT NULL, + description text NOT NULL, + status migration_status NOT NULL +); + +``` + +The migration_history table provides: +- **Migration tracking**: Records which migrations have been applied +- **Timestamp tracking**: Shows when each migration was executed +- **Idempotency**: Prevents duplicate migration runs +- **Version identification**: Easily identify the current database schema version + +The table is created if it does not exist already. Rollback and upgrade scripts will insert a new row with the version number and timestamp when the script was applied. \ No newline at end of file diff --git a/docs/network-upgrades/mesa/appendix/automode-docker-compose-quickstart.mdx b/docs/network-upgrades/mesa/appendix/automode-docker-compose-quickstart.mdx new file mode 100644 index 000000000..c49b5ba75 --- /dev/null +++ b/docs/network-upgrades/mesa/appendix/automode-docker-compose-quickstart.mdx @@ -0,0 +1,69 @@ +--- +title: Automode Docker Compose Quickstart +sidebar_label: Automode Docker Compose Quickstart +hide_title: true +description: Run a Mina node with automode enabled for the Mesa upgrade using Docker Compose. +keywords: + - Mesa + - Docker Compose + - automode + - quickstart +--- + +# Automode Docker Compose Quickstart + +```yaml +services: + mina_node: + image: 'minaprotocol/mina-daemon-auto-hardfork:-noble-mainnet' + restart: always + environment: + MINA_CLIENT_TRUSTLIST: "0.0.0.0/0" + entrypoint: [] + command: > + bash -c ' + mina daemon \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --insecure-rest-server \ + --rest-port 3085 \ + --hardfork-handling migrate-exit + ' + volumes: + - './mina-config:/root/.mina-config' + ports: + - '3085:3085' + - '8302:8302' +``` + +Replace `` with the published stop-slot release from the [Mina releases page](https://github.com/MinaProtocol/mina/releases). + +:::tip About `--hardfork-handling` +The automode image still expects `--hardfork-handling migrate-exit`, so you always pass it here. The automode dispatcher consumes the flag and routes to the correct binary, so you do not need any conditional logic around it. Remove it only when you switch off the automode image (see [Switching back to normal Docker](#switching-back-to-normal-docker)). +::: + +:::tip Docker image tag +Use the stop-slot release image for your target network. For devnet, replace the image tag suffix `mainnet` with `devnet` and use the corresponding devnet release tag. +::: + +:::tip Peer list URL +The `--peer-list-url` above points to the mainnet bootnodes. For devnet, replace it with `https://bootnodes.minaprotocol.com/networks/devnet.txt`. +::: + +:::tip Additional environment variables +Add any extra configuration (e.g. `MINA_PRIVKEY_PASS`) under the `environment` section. +::: + +## Switching back to normal Docker + +After your node has completed the Mesa transition and you want to move off `mina-daemon-auto-hardfork`, update your Compose config as follows: + +- Remove `--hardfork-handling migrate-exit`. +- Set the Docker image to the post-fork artifact, for example `minaprotocol/mina-daemon:-noble-mainnet`. + +```bash +docker compose up -d +``` + +:::caution +`restart: always` and the persistent `mina-config` volume are required for automode. If the volume is wiped between restarts, the node cannot transition to Mesa. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for details. +::: diff --git a/docs/network-upgrades/mesa/appendix/upgrade-modes-details.mdx b/docs/network-upgrades/mesa/appendix/upgrade-modes-details.mdx new file mode 100644 index 000000000..2e89360eb --- /dev/null +++ b/docs/network-upgrades/mesa/appendix/upgrade-modes-details.mdx @@ -0,0 +1,210 @@ +--- +title: Upgrade Modes - Details +sidebar_label: Upgrade Modes - Details +hide_title: true +description: In-depth technical details of the Mesa upgrade mechanism, including automode internals, the dispatcher, stop-slot system, and manual upgrade flow. +keywords: + - Mesa + - upgrade + - automode + - technical details + - stop-slot + - dispatcher + - manual + - block producer +--- + +# Upgrade Modes - Details + +This page explains **how the two upgrade modes work under the hood**. If you just want to know which mode to pick and what to do, see [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). This page is for operators who want to understand the mechanism before trusting it with their nodes. + +## The Big Picture + +During the Mesa hard fork, the network transitions from the current chain (referred to internally as "berkeley") to the new Mesa chain. Both upgrade modes use the same **stop-slot mechanism** to halt the old chain at a predetermined slot. The difference is what happens next: + +- **Automode**: the node automatically switches to the Mesa binary and resumes. No operator action needed. +- **Manual mode**: the operator stops the old node, installs the Mesa release, and starts it themselves. + +Both modes reach the same end state — a node running on the Mesa network, producing blocks after the Mesa genesis timestamp. + +## Stop-Slot Mechanism + +Both modes rely on two critical slot numbers baked into the stop-slot release (3.x.x): + +| Slot | What happens | +|---|---| +| **stop-transaction-slot** | The network stops accepting transactions. Blocks produced after this slot are empty — no user commands, no coinbase rewards, no fee transfers. This begins the State Finalization period. | +| **stop-network-slot** | The network stops producing and accepting blocks entirely. The chain halts. This is the point where the fork happens. | + +The gap between these two slots is exactly 100 slots (5 hours) — the **State Finalization** period. It ensures all nodes converge on the same final state before the fork. + +:::tip For block producers +You **will not earn block rewards** during State Finalization (no coinbase), but you **must keep your node running** to maintain network stability and block density. If you are in the Delegation Program, uptime tracking continues during this phase. +::: + +## Automode — How It Works Internally + +### Dual-Binary Architecture + +The automode release ships **two complete sets of Mina binaries plus the dispatcher** in a single `mina-{network}-automode` Debian package: + +| Component | Path | Purpose | +|---|---|---| +| **Pre-fork binary** | `{RUNTIMES_BASE_PATH}/berkeley/mina` | Runs the current chain up to the stop-network-slot | +| **Post-fork binary** | `{RUNTIMES_BASE_PATH}/mesa/mina` | Runs the Mesa chain after the fork | +| **Dispatcher** (`mina-dispatch`) | `/usr/local/bin/mina-dispatch` | Routes your commands to the correct binary | + +When you install the automode package (or use the [automode Docker image](/network-upgrades/mesa/glossary#automode-image)), all three components are installed together. + +### The Dispatcher + +The dispatcher (`mina-dispatch`) is a shell script that acts as a transparent wrapper around the real `mina` binary. When you run a `mina` command, the dispatcher decides which binary to execute: + +- **`daemon` subcommand** — routes based on the activation state file: + ``` + Does the activation state file exist? + → NO: route to the pre-fork (berkeley) binary + → YES: route to the post-fork (mesa) binary + ``` +- **`client` subcommand** — always routes to the post-fork (mesa) binary, regardless of activation state. This works because the GraphQL schema did not change between Berkeley and Mesa. +- **`--version`** — passed through without processing. + +The **activation state file** is located at `{MINA_HARDFORK_STATE_DIR}/auto-fork-mesa-{network_id}/activated` (e.g., `~/.mina-config/auto-fork-mesa-mainnet/activated` on the host, or `/root/.mina-config/auto-fork-mesa-mainnet/activated` in Docker). This file is created by the daemon itself when it detects that the network has reached the stop-network-slot. + +### Dispatcher Limitations + +:::info Current implementation — may change in future releases +This limitation exists because, for most subcommands, the dispatcher does not receive the config directory location as an argument. Without access to the config directory, it cannot check for the `activated` state file and therefore cannot determine whether the node is running Berkeley or Mesa. +::: + +The dispatcher supports the following subcommands: + +| Subcommand | Routing behavior | +|---|---| +| `daemon` | Routes to pre-fork or post-fork binary based on activation state | +| `client` | Always routes to the post-fork (mesa) binary | +| `--version` | Passed through without processing | + +Any other subcommand (e.g., `accounts list`, `ledger export`) will fail with an error: + +``` +mina-dispatch ERROR: unsupported subcommand 'accounts' for automatic hardfork handling +``` + +For unsupported subcommands, invoke the correct version-specific binary directly: + +| Binary | When to use | Path | +|---|---|---| +| `mina-berkeley` | Before the fork (or to query pre-fork state) | `/usr/lib/mina/berkeley/mina` | +| `mina-mesa` | After the fork | `/usr/lib/mina/mesa/mina` | + +```bash +# These work at any time — they bypass the dispatcher +mina-berkeley client status +mina-mesa accounts list +mina-mesa ledger export + +# Full paths also work for any binary in either runtime +/usr/lib/mina/mesa/mina client status +``` + +:::note About `client` routing +The `client` subcommand (e.g., `mina client status`) is always routed to the mesa binary because it communicates with the running daemon over GraphQL, and the GraphQL schema did not change between Berkeley and Mesa. This means `mina client status` works through the dispatcher at any point — before or after the fork — as long as a daemon is running. +::: + +### What the Dispatcher Does for `daemon` Commands + +When the dispatcher routes a `daemon` command to the Mesa binary, it automatically adjusts your command-line arguments: + +1. **Config files**: Your existing `-config-file` arguments are kept, and the Mesa-specific configuration is **appended as the last** `-config-file` entry. This ensures Mesa settings take precedence. +2. **Genesis ledger directory**: Any `--genesis-ledger-dir` argument is rewritten to point to the Mesa ledger directory. +3. **Hardfork handling flag**: The `--hardfork-handling` argument is **removed** (it is not supported on the Mesa chain). + +### Automode Timeline + +Here is what happens from a block producer's perspective when using automode: + +Automode upgrade flow diagram showing four phases: pre-upgrade, state finalization, automatic upgrade with process restart, and post-upgrade + +### Restart and Filesystem Requirements + +The automode transition involves a **process restart**. When the daemon reaches the stop-network-slot, it: + +1. Generates the Mesa configuration (`daemon.json`) and genesis ledger tarballs in the config directory +2. Writes an `activated` sentinel file to mark the fork as complete +3. **Exits with code 0** (clean shutdown) + +The daemon does **not** restart itself. Your process manager must detect the exit and restart the process. On restart, the dispatcher sees the `activated` file and launches the Mesa binary with the auto-generated config. + +**This means two things are critical:** + +**Persistent config directory** — The config directory (typically `~/.mina-config` or `/root/.mina-config` in Docker) **must survive across restarts**. It contains the `activated` file and the generated Mesa configuration. If this directory is ephemeral or gets wiped on restart, the node will restart into the Berkeley binary and fail to join the Mesa network. + +**Automatic restart on clean exit** — Your process manager must be configured to restart the daemon after exit code 0: + +| Platform | Configuration | Notes | +|---|---|---| +| **systemd** | `Restart=always` | Default in the Mina systemd unit (`mina.service`). Restarts after 30 seconds. | +| **Docker** | `--restart=always` or `--restart=unless-stopped` | Set when creating the container. The default (`no`) will **not** restart. | +| **Kubernetes** | `restartPolicy: Always` | The k8s default for pods. Ensure your liveness probes and Helm chart configuration do not treat exit code 0 as a failure that triggers a volume wipe or full pod replacement. The config directory volume **must** be a `PersistentVolumeClaim`, not `emptyDir`. | + +:::danger For Kubernetes / Helm users +If your Helm chart or pod spec uses `emptyDir` for the config directory, the `activated` file and generated Mesa config will be lost when the pod restarts. Use a `PersistentVolumeClaim` instead. Also verify that your restart logic does not re-initialize the config directory from scratch — the auto-generated Mesa config must be preserved. +::: + +### How automode is packaged + +Operator install instructions (Debian packages, Docker image) live on [Upgrade Modes — Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). The detail worth keeping here is what each artifact contributes to the mechanism described above: + +- The `mina-{network}-automode` Debian package is an umbrella package whose dependencies pull in the pre-fork binary, post-fork binary, dispatcher, and dispatcher configuration. Internally it depends on `mina-{network}-prefork-mesa` (supplies `/usr/lib/mina/berkeley/mina`) and `mina-{network}-postfork-mesa` (supplies `/usr/lib/mina/mesa/mina`, plus `/usr/local/bin/mina-dispatch` and `/etc/default/mina-dispatch`). Operators only need to install `mina-{network}-automode`; apt resolves the rest. +- The `minaprotocol/mina-daemon-auto-hardfork` Docker image bundles the same artifacts and sets `MINA_APP=/usr/local/bin/mina-dispatch` and `MINA_HARDFORK_STATE_DIR=/root/.mina-config` in the entrypoint. + +:::caution +The dispatcher reads its configuration from `/etc/default/mina-dispatch`. This file must exist and be owned by root. Do not modify it unless you know what you are doing. +::: + +## Manual Mode — How It Works + +Manual mode is the traditional upgrade approach, similar to the Berkeley upgrade. You are in full control of every step. + +### Manual Mode Timeline + +Manual mode upgrade flow diagram showing four phases: pre-upgrade, state finalization, manual upgrade with 5 operator steps, and post-upgrade + +### Manual Mode — What Gets Installed + +When you install the Mesa release manually, the package includes: +- The Mesa `mina` binary +- A new runtime configuration JSON for the Mesa network +- New genesis and epoch ledger tarballs + +These replace the pre-fork components. There is no dispatcher involved — you are running the Mesa binary directly. + +### Manual Mode — Updating Your Flags + +When switching from the pre-fork to the Mesa binary, you need to update your startup flags: + +1. **Remove** `--hardfork-handling` if you were using it +2. **Update** your `--genesis-ledger-dir` to point to the Mesa ledger directory (included in the package) +3. **Update** your `-config-file` to use the Mesa configuration +4. **Keep** your existing `--block-producer-key`, `--libp2p-keypair`, and other operator-specific flags + +See [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) for exact flag values. + +### Docker (Manual Mode) + +For manual mode with Docker, use the **hardfork** image (not auto-hardfork): + +``` +minaprotocol/mina-daemon-hardfork:{version}-{codename}-{network} +``` + +This image includes both pre-fork and post-fork packages but uses a dedicated hardfork entrypoint that requires manual intervention to complete the transition. + +## Which mode should I pick? + +The "who should use" criteria for each mode and the side-by-side comparison live on [Upgrade Modes](/network-upgrades/mesa/upgrade-modes#comparison) — this page intentionally stays focused on the underlying mechanism so the two documents do not drift. + +## Operator troubleshooting + +Operator-facing troubleshooting (how to tell which binary is active, dispatcher debug mode, when to use `mina-berkeley` vs `mina-mesa` directly) has moved to its own page: [Troubleshooting](/network-upgrades/mesa/troubleshooting). diff --git a/docs/network-upgrades/mesa/archive-upgrade.mdx b/docs/network-upgrades/mesa/archive-upgrade.mdx index f1b070220..15977c2d0 100644 --- a/docs/network-upgrades/mesa/archive-upgrade.mdx +++ b/docs/network-upgrades/mesa/archive-upgrade.mdx @@ -34,7 +34,7 @@ The commands below use `` as a placeholder for the archive build y To successfully upgrade the archive database to Mesa, you must ensure that your environment meets the foundational requirements. -## Migration host +## Upgrade host - PostgreSQL database for database server - If you use Docker, then any of the supported OS by Mina (bullseye, focal, noble, bookworm or jammy) with at least 32 GB of RAM @@ -51,9 +51,9 @@ you can always download it from the O1Labs Google Cloud bucket. ### Upgrade script Assuming that you have a PostgreSQL database with Mainnet archive data, in order to upgrade it to Mesa version, you need to run SQL upgrade script. -We put all efforts to make the upgrade process as smooth as possible. Script can be run on archive node which is online or offline. +The script can be run on an archive node that is either online or offline. Script can be run multiple times, it will skip steps that were already completed. It also performs sanity checks before each step to ensure that the upgrade process is successful. -Finally it creates new table (version) in the database to keep track of the upgrade process. +Finally it creates a new table (`migration_history`) in the database to keep track of the upgrade process. #### Getting the script @@ -106,8 +106,8 @@ Make sure to replace `` and `` with your actual PostgreSQL u #### Rollback You can rollback the upgrade process by restoring the database from a backup taken before running the upgrade script. -Another is to run rollback script which is part of the upgrade script. It will drop all tables and other database objects created by the upgrade script. -It will also update the version table to reflect the rollback. +Another alternative is to run the rollback script, which is part of the upgrade script. It will drop all tables and other database objects created by the upgrade script. +It will also update the `migration_history` table to reflect the rollback. ##### Running the rollback script @@ -125,54 +125,112 @@ Make sure to replace `` and `` with your actual PostgreSQL u After successfully running the upgrade script, you DO NOT need to restart your archive node or Rosetta API. Changes in upgrade script are backward compatible and will be picked up by the archive node and Rosetta API automatically. -### Verification -To verify that the upgrade was successful, you can check the version table in the PostgreSQL database. +### Verification with the Archive Hardfork Toolbox -You can do this by running the following command: +The `mina-archive-hardfork-toolbox` is a dedicated CLI tool for verifying the integrity of archive database upgrades and fork transitions. It is shipped with the Mina archive package and available as a standalone binary. + +For full details, see the [toolbox README](https://github.com/MinaProtocol/mina/blob/compatible/src/app/archive_hardfork_toolbox/README.md). + +All commands below require a `--postgres-uri` flag in the format: +``` +postgresql://:@:/ +``` + +#### Step 1: Pre-fork validation (before the fork) + +Verify the fork block candidate is valid before the network halts. + +**Check that the fork block is in the best chain:** ```bash -psql -U -d -c "SELECT * FROM version;" +mina-archive-hardfork-toolbox fork-candidate is-in-best-chain \ + --postgres-uri \ + --fork-state-hash \ + --fork-height \ + --fork-slot ``` -Make sure to replace `` and `` with your actual PostgreSQL username and database name. -If the upgrade was successful, you should see the new version number in the output. +**Verify the fork block has enough confirmations:** +```bash +mina-archive-hardfork-toolbox fork-candidate confirmations \ + --postgres-uri \ + --latest-state-hash \ + --fork-slot \ + --required-confirmations +``` -We put a lot of effort into making the upgrade process as smooth as possible. -However, if you encounter any issues or need assistance, please reach out to the Mina community on [Discord](https://discord.gg/minaprotocol) or [GitHub Discussions](https://github.com/MinaProtocol/mina/discussions). +**Verify no commands were executed after the fork block** (ensures a clean fork point): +```bash +mina-archive-hardfork-toolbox fork-candidate no-commands-after \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` -## Appendix: Database Schema Changes +**Find the last block with transactions** (useful for identifying the fork point): +```bash +mina-archive-hardfork-toolbox fork-candidate last-filled-block \ + --postgres-uri +``` -Below we present details of what was changed in the archive node database schema between Berkeley and Mesa versions. +#### Step 2: Verify the schema upgrade -### Zkapp_state_nullable Additional Columns +After running `upgrade-to-mesa.sql`, verify the database schema was upgraded correctly: -The `zkapp_state_nullable` table has been modified to include new columns `element8` through `element31` which are nullable and can store additional state information for zkApps. +```bash +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri \ + --protocol-version \ + --migration-version +``` -```sql -, element8 int REFERENCES zkapp_field(id) -... -, element31 int REFERENCES zkapp_field(id) -); +You can also verify manually by checking the `migration_history` table: +```bash +psql -U -d -c "SELECT * FROM migration_history;" ``` -This expansion allows zkApps to store up to 31 state elements instead of the previous 8, significantly increasing the state storage capacity for complex smart contracts. +If the upgrade was successful, the `migration_history` table will contain an entry with the expected migration version and a recent timestamp. + +#### Step 3: Post-fork validation -### Version Table +After the fork activates and the Mesa chain is running, validate the fork block and its ancestry: + +```bash +mina-archive-hardfork-toolbox validate-fork \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` -We also introduced a new table `version` to keep track of the database schema version. -The purpose of this table is to help with future database migrations. The table tracks which migration scripts were applied and when. -Ultimately it helps to determine the current version of the database schema and helps to avoid applying the same migration script multiple times. +#### Step 4: Mark the canonical chain (if needed) -This table is created if it does not exist already. Rollback and upgrade scripts will insert a new row with the version number and timestamp when the script was applied. +If you need to mark the chain leading to a specific block as canonical for the new protocol version: -```sql -CREATE TABLE IF NOT EXISTS version ( - version_num INT PRIMARY KEY, - applied_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP -); +```bash +mina-archive-hardfork-toolbox convert-chain-to-canonical \ + --postgres-uri \ + --target-block-hash \ + --protocol-version .. ``` -The version table provides: -- **Migration tracking**: Records which migrations have been applied -- **Timestamp tracking**: Shows when each migration was executed -- **Idempotency**: Prevents duplicate migration runs -- **Version identification**: Easily identify the current database schema version \ No newline at end of file +Optionally use `--stop-at-slot ` to limit how far back in history the marking goes. + +:::tip Typical workflow +1. **Before the fork**: run `fork-candidate` commands to validate the fork block +2. **After running the upgrade script**: run `verify-upgrade` to confirm the schema upgrade +3. **After the fork activates**: run `validate-fork` to confirm data integrity +4. **For full archive verification**: run the [Archive Replayer](/node-operators/archive-node/replayer) to replay all transactions and compare the resulting ledger against the official fork config +::: + +If you encounter any issues, reach out to the Mina community on [Discord](https://discord.gg/minaprotocol) or [GitHub Discussions](https://github.com/MinaProtocol/mina/discussions). + +## Full Archive Verification with the Replayer + +The toolbox commands above verify the schema and fork block integrity, but they do not verify every transaction in the archive. For complete end-to-end verification, use the **[Archive Replayer](/node-operators/archive-node/replayer)** — a tool that replays all transactions from genesis (or a checkpoint) through the fork point, recomputing the ledger state and comparing it against the official fork config. + +The replayer is not limited to upgrade-time use. It is an **ongoing verification tool** that can be run at any time to confirm your archive database faithfully represents the canonical chain. After the Mesa upgrade, you can continue running it against Mesa blocks to detect any data corruption or missing blocks. + +See [Archive Replayer](/node-operators/archive-node/replayer) for usage, flags, and examples. + +## Database Schema Changes + +For the full schema diff (new columns, modified tables, applied SQL), see [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes). diff --git a/docs/network-upgrades/mesa/fork-schedule.mdx b/docs/network-upgrades/mesa/fork-schedule.mdx new file mode 100644 index 000000000..326a3b17b --- /dev/null +++ b/docs/network-upgrades/mesa/fork-schedule.mdx @@ -0,0 +1,77 @@ +--- +title: Mesa Fork Schedule +sidebar_label: Fork Schedule +hide_title: true +description: Concrete fork-schedule values for the Mesa upgrade — Mesa genesis timestamp, stop-transaction-slot (slot_tx_end), stop-network-slot (slot_chain_end), and pre-fork / Mesa release versions per network. +keywords: + - Mesa + - fork schedule + - stop-transaction-slot + - stop-network-slot + - slot_tx_end + - slot_chain_end + - Mesa genesis timestamp +--- + +# Mesa Fork Schedule + +This page collects the concrete fork-schedule values for the Mesa upgrade per network. If you are integrating with Mina (exchange, custodian, indexer, Rosetta consumer, monitoring), this is the single source of truth for **when** to act and **which release** to run. + +The relevant moments are: + +- **`stop-transaction-slot`** (a.k.a. `slot_tx_end` in the daemon config) — the slot at which nodes stop accepting new user transactions. Block production continues with empty blocks until the network-slot. See the [glossary entry](/network-upgrades/mesa/glossary#stop-transaction-slot). +- **`stop-network-slot`** (a.k.a. `slot_chain_end`) — the slot at which block production halts entirely. The network is offline between this slot and the Mesa genesis timestamp. See the [glossary entry](/network-upgrades/mesa/glossary#stop-network-slot). +- **Mesa genesis timestamp** — the wall-clock time at which the first block on the Mesa chain is produced. Exactly 3 hours after the _stop-network-slot_. See the [glossary entry](/network-upgrades/mesa/glossary#mesa-genesis-timestamp). + +## Mainnet + +:::warning Values not yet published +The mainnet Mesa fork schedule has not been announced. The values below are placeholders. They will be filled in once o1Labs publishes the stop-slot release and the Mesa genesis timestamp. +::: + +| Field | Value | +|---|---| +| Pre-fork (stop-slot) release | _TBD — will be a `3.x.x` tag_ | +| Mesa release | _TBD — will be a `4.x.x` tag_ | +| `stop-transaction-slot` (`slot_tx_end`) | _TBD_ | +| `stop-network-slot` (`slot_chain_end`) | _TBD_ | +| Mesa genesis timestamp (UTC) | _TBD — exactly 3 hours after the stop-network-slot_ | +| State Finalization window | Exactly 100 slots = exactly 5 hours between `slot_tx_end` and `slot_chain_end` | + +Subscribe to the [MinaProtocol/mina releases](https://github.com/MinaProtocol/mina/releases?q=mesa) and watch the [Mina Foundation announcements channel](https://discord.gg/minaprotocol) for the schedule publication. + +## Devnet + +:::warning Values not yet published +Devnet typically forks ahead of mainnet to give integrators a final dress rehearsal. Schedule will be posted here once announced. +::: + +| Field | Value | +|---|---| +| Pre-fork (stop-slot) release | _TBD_ | +| Mesa release | _TBD_ | +| `stop-transaction-slot` (`slot_tx_end`) | _TBD_ | +| `stop-network-slot` (`slot_chain_end`) | _TBD_ | +| Mesa genesis timestamp (UTC) | _TBD_ | + +## Preflight + +The Mesa preflight network has already forked. The numbers below are historical; they will not change. + +| Field | Value | +|---|---| +| Pre-fork release | `4.0.0-preflight1-b649c79` | +| Stop-slot release | `4.0.0-preflight-stop-2967b39` | +| Mesa release | `4.0.0-preflight-3f038cb` | +| Mesa hard-fork time (UTC) | 2026-04-27 13:00 | +| Transaction protocol version after fork | 5.0.0 | +| o1js version targeting protocol v5.0.0 | `o1js@3.0.0-mesa.698ca` | + +For the full preflight context, see [Preflight Network](/network-upgrades/mesa/preflight-network). + +## How to use these values + +- **Exchanges**: schedule your deposit/withdrawal freeze to begin **before** `stop-transaction-slot` and re-enable **after** the Mesa genesis timestamp confirms block production. See the [Exchanges tab on Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) for the full checklist. +- **Block producers**: install the stop-slot release any time before `stop-transaction-slot`. If you are using [automode](/network-upgrades/mesa/upgrade-modes), no further action is needed — the dispatcher will swap to the Mesa binary automatically after the Mesa release is published. +- **Archive operators**: schedule the archive schema upgrade for any time on or after `stop-network-slot`. The script is backward compatible and can also be run earlier. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). +- **Rosetta operators**: follow the manual-mode timeline — Rosetta does not support automode. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). diff --git a/docs/network-upgrades/mesa/glossary.mdx b/docs/network-upgrades/mesa/glossary.mdx new file mode 100644 index 000000000..f55ab7afb --- /dev/null +++ b/docs/network-upgrades/mesa/glossary.mdx @@ -0,0 +1,147 @@ +--- +title: Glossary +sidebar_label: Glossary +hide_title: true +description: Definitions of key terms used throughout the Mesa upgrade documentation — stop-slots, automode, dispatcher, trustless/trustful upgrades, and more. +keywords: + - Mesa + - glossary + - terminology + - stop-slot + - automode + - hard fork +--- + +# Glossary + +This page defines the key terms used throughout the Mesa upgrade documentation. If you encounter an unfamiliar term in any of the upgrade guides, you can look it up here. + +:::tip +Bookmark this page for quick reference while reading the upgrade docs. +::: + +--- + +## Network and Fork Terms + +### Hard Fork + +A major, non-backward-compatible network upgrade. All node operators must upgrade their software before the fork activates. After the fork, nodes running the old software can no longer participate in the network. The Mesa upgrade is a hard fork. + +### Stop-Slot Release + +The Mina release (version 3.x.x) that node operators install **before** the fork. This release contains the stop-slot logic that gracefully halts the network at the designated time. In [automode](#automode), this release also bundles both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary). + +### Stop-Transaction-Slot {#stop-transaction-slot} + +A predefined global slot number baked into the [stop-slot release](#stop-slot-release). When the network reaches this slot, nodes stop accepting new user transactions. Block production continues with empty blocks (no user commands, no coinbase, no fee transfers) for exactly 100 more slots (5 hours) until the [stop-network-slot](#stop-network-slot). This is the point where exchanges must disable deposits and withdrawals. + +Referred to as `slot_tx_end` in the Mina codebase and daemon configuration. + +### Stop-Network-Slot {#stop-network-slot} + +A predefined global slot number that comes after the [stop-transaction-slot](#stop-transaction-slot). When the network reaches this slot, block production halts entirely. The network is now frozen, and the final state is used to build the Mesa release. In [automode](#automode), this is when the daemon automatically transitions to the [post-fork binary](#post-fork-binary). + +Referred to as `slot_chain_end` in the Mina codebase and daemon configuration. + +### State Finalization + +The stabilization period between the [stop-transaction-slot](#stop-transaction-slot) and the [stop-network-slot](#stop-network-slot) — 100 slots, which is exactly 5 hours. During this window, empty blocks are produced so that all nodes converge on the same final state. No operator action is required; nodes should remain running. See [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +### Mesa Genesis Timestamp + +The predefined time at which the first block is produced on the new Mesa chain — exactly 3 hours after the network reaches the [stop-network-slot](#stop-network-slot). This marks the start of the upgraded network. + +### Mesa Release + +The new Mina build (version 4.x.x) published after [state finalization](#state-finalization) is complete. It contains the Mesa chain configuration and genesis ledger derived from the final pre-fork state. In [manual mode](#manual-mode), operators install this release to join the Mesa network. + +### Preflight Network + +A testing environment for validating the Mesa upgrade before deployment to mainnet. Node operators and developers can test their infrastructure and applications against it. The preflight network may be unstable and is not intended for production use. See [Preflight Network](/network-upgrades/mesa/preflight-network). + +--- + +## Upgrade Mode Terms + +### Automode {#automode} + +The recommended upgrade path for daemon nodes (block producers, SNARK coordinators). In automode, the node handles the entire fork transition automatically — no manual intervention is required during the fork window. The [stop-slot release](#stop-slot-release) ships both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary), and the [dispatcher](#dispatcher) routes to the correct one based on the fork state. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). + +### Manual Mode {#manual-mode} + +The traditional upgrade path where operators manually stop their node after the network halts, install the [Mesa release](#mesa-release), and restart. This gives full control over every step but requires prompt action when the release is published. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). + +### Automode Image {#automode-image} + +The Docker image that bundles the [automode](#automode) runtime — both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary), plus the [dispatcher](#dispatcher). Published as `minaprotocol/mina-daemon-auto-hardfork:{version}-{codename}-{network}`. Use this image when running automode in Docker. The companion image `minaprotocol/mina-daemon-hardfork` is the manual-mode equivalent (no dispatcher). + +### Pre-Fork Binary {#pre-fork-binary} + +The binary that runs the current (Berkeley) chain up to the [stop-network-slot](#stop-network-slot). In [automode](#automode), this is shipped as part of the `mina-{network}-automode` package and installed at `/usr/lib/mina/berkeley/mina`. In Docker, the [automode image](#automode-image) includes this binary. + +### Post-Fork Binary {#post-fork-binary} + +The binary that runs the new Mesa chain after the fork activates. In [automode](#automode), this is shipped as part of the `mina-{network}-automode` package and installed at `/usr/lib/mina/mesa/mina`. The [dispatcher](#dispatcher) switches to this binary once the [activation file](#activation-file) is created. + +### Dispatcher {#dispatcher} + +A shell script wrapper (`mina-dispatch`) installed at `/usr/local/bin/` that routes commands to the correct binary — [pre-fork](#pre-fork-binary) or [post-fork](#post-fork-binary). For `daemon`, routing is based on whether the [activation file](#activation-file) exists. For `client`, commands are always routed to the [post-fork binary](#post-fork-binary). Other subcommands (e.g., `accounts list`) are not supported — use `mina-berkeley` or `mina-mesa` directly. See [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +### Activation File {#activation-file} + +A sentinel file created by the daemon when it reaches the [stop-network-slot](#stop-network-slot) during [automode](#automode). Its presence signals to the [dispatcher](#dispatcher) that the fork has completed and the [post-fork binary](#post-fork-binary) should be used on subsequent restarts. Located at `{config-directory}/auto-fork-mesa-{network}/activated`. + +--- + +## Archive Upgrade Terms + +### Trustless Upgrade {#trustless} + +An archive database upgrade method where operators run the `upgrade_to_mesa.sql` script on their existing database. The script is backward-compatible and can be applied before the fork while the Berkeley archive node is still running. It completes in under 1 minute. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +### Trustful Upgrade {#trustful} + +An archive database upgrade method where operators import an official SQL database dump published by o1Labs after the fork. This requires no pre-fork action — operators wait for the dump, create a fresh database, and import it. The trade-off is that you trust o1Labs' export rather than verifying the data yourself. + +### Archive Hardfork Toolbox {#hardfork-toolbox} + +A dedicated CLI tool (`mina-archive-hardfork-toolbox`) for verifying the integrity of archive database upgrades and fork transitions. It provides commands to validate the fork block, verify schema upgrades, and mark the canonical chain. Shipped with the Mina archive package. See [Archive Upgrade — Verification](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox). + +### Replayer {#replayer} + +A verification tool (`mina-replayer`) that replays all transactions from a Mina archive database to reconstruct the ledger state from scratch. It can be used to verify that the archive database faithfully represents the canonical chain — both during the fork and as an ongoing integrity check. See [Archive Replayer](/node-operators/archive-node/replayer). + +### Fork Config + +The official ledger checkpoint published with the [Mesa release](#mesa-release). Operators can compare the output of the [replayer](#replayer) against this config to verify their archive matches the canonical state used to build Mesa. + +--- + +## Mesa-Specific Changes and Conventions + +Mesa bundles four Mina Improvement Proposals (MIPs) that change protocol behavior. Full descriptions live in the [Mesa overview](/network-upgrades/mesa#what-mesa-introduces); the brief definitions below link each to its canonical spec. + +### Faster Blocks + +Mesa reduces the slot time to 90 seconds and unsets the default zkApp soft limit (previously 24 commands per block). Defined in [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md). + +### Expanded zkApp State + +Mesa increases the on-chain state available to zkApps from 8 fields (indexes `0–7`) to 32 fields (indexes `0–31`) per account. This applies to both the `zkapp_states` and `zkapp_states_nullable` database tables. Defined in [MIP7](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0007-increase-state-size-limit.md). See [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) for the SQL diff. + +### Larger Events and Actions + +Mesa raises the per-transaction limit on events and actions from 100 to 1024 field elements each, and removes the previous 16-field-element cap per individual event or action. Defined in [MIP8](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0008-increase-events-actions-limit.md). + +### Larger zkApp Transactions + +Mesa roughly triples the maximum number of account updates a single zkApp transaction can contain. Defined in [MIP9](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0009-increase-zkapp-account-update-limit.md). + +### Mesa Package Naming Convention + +The [automode](#automode) Debian package is published as `mina-{network}-automode` to distinguish the dual-binary automode variant from the standard manual-mode `mina-{network}` package. This allows operators to install the automode package side-by-side with the existing daemon without one replacing the other. `mina-{network}-automode` is an umbrella package: under the hood it depends on `mina-{network}-prefork-mesa` (the [pre-fork binary](#pre-fork-binary)) and `mina-{network}-postfork-mesa` (the [post-fork binary](#post-fork-binary) plus the [dispatcher](#dispatcher)), and apt resolves both transitively when you install it. + +### Node Status Service + +A telemetry feature that reports non-sensitive node data (e.g., version, sync status) to help monitor the amount of upgraded active stake during the upgrade. Enabled by default on the Mesa daemon; pass `--simplified-node-stats` to keep the report minimal, or `--disable-node-status` to opt out entirely. See [Help Monitor the Network](/network-upgrades/mesa/upgrade-steps/post-upgrade#help-monitor-the-network) for the full flag set. diff --git a/docs/network-upgrades/mesa/index.mdx b/docs/network-upgrades/mesa/index.mdx new file mode 100644 index 000000000..cddb0cfd1 --- /dev/null +++ b/docs/network-upgrades/mesa/index.mdx @@ -0,0 +1,199 @@ +--- +title: Mesa Upgrade +sidebar_label: Mesa Upgrade +hide_title: true +description: Overview of the Mesa mainnet upgrade for Mina Protocol — expanded zkApp state, automode upgrades, and protocol improvements. +keywords: + - Mesa + - upgrade + - hard fork + - overview + - mainnet +--- + +# Mesa Upgrade + +The Mesa upgrade is a major network upgrade (hard fork) for the Mina Protocol mainnet. It is not backward compatible: every consensus-participating Mina daemon (block producers, SNARK coordinators, archive nodes, Rosetta API nodes, and seed nodes) must run a Mesa-compatible release to remain on the network. Wallets, off-chain services, and downstream tooling do not run a daemon and only need to update if they pin to a specific transaction format or version. + +:::info New to the Mesa upgrade? +This documentation uses terms like _automode_, _stop-slot_, _trustless upgrade_, _dispatcher_, and more. If you encounter an unfamiliar term, check the **[Glossary](/network-upgrades/mesa/glossary)** for definitions. +::: + +## What Mesa Introduces + +Mesa bundles four Mina Improvement Proposals (MIPs) that change protocol behavior, plus two operational improvements to the upgrade flow itself. The canonical MIP specs live in the [MinaProtocol/MIPs](https://github.com/MinaProtocol/MIPs/tree/main/MIPS) repository; each section below links to the spec for the proposal it describes. + +### Faster Blocks — [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md) + +Mesa halves Mina's slot time from **180 seconds to 90 seconds**, doubling block production frequency. To keep the long-term token emission rate flat, the per-block **coinbase reward is also halved** (from 720 MINA to 360 MINA). The total Mina supplied per epoch is unchanged. + +Several second-order consequences flow from the slot-time change: + +- **Epoch duration halves** from ~14.9 days to ~7.4 days. Scripts and operational procedures tied to epoch boundaries (delegation cooldown, automated payouts, monitoring dashboards) trigger about twice as often. +- **Vesting schedules on active vesting accounts are automatically migrated** during the hard fork so they continue unlocking on the same real-world cadence. No operator action is required. +- **The default zkApp soft limit (24 commands per block) is unset**, allowing blocks to include more zkApp transactions when network demand is high. +- **SNARK coordinators handling maximum-cost zkApp transactions should deploy at least 4 workers** (≈4 CPU cores each) to keep up with the new 90-second slot timing. A prerequisite SNARK-worker parallelization is delivered via soft fork before Mesa. + +### Expanded zkApp State — [MIP7](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0007-increase-state-size-limit.md) + +Mesa raises the on-chain state available to a zkApp account from **8 field elements (indexes `0–7`) to 32 field elements (indexes `0–31`)**. zkApps can now store roughly four times more data directly on chain without off-chain workarounds. + +Implications: + +- **zkApp developers must redeploy** to take advantage of the new fields; older verification keys will not validate transactions touching indexes `8–31`. +- **Archive databases gain new columns** (`element8` … `element31`) on `zkapp_states` and `zkapp_states_nullable`. The Mesa archive upgrade script handles this automatically — see [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) for the full schema diff. +- **Transaction protocol version is bumped** because the on-chain serialization format changes. + +### Larger Events and Actions — [MIP8](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0008-increase-events-actions-limit.md) + +Mesa raises the per-transaction limit on events and actions from **100 field elements to 1024 field elements** for each. The previous cap of 16 field elements per individual event or action is also removed. This makes events and actions a viable channel for richer on-chain signaling and larger off-chain dispatch payloads in a single transaction. + +Mostly relevant to zkApp developers — node operators do not need to act on this change, but block producers will validate against the new limits automatically after the fork. + +### Larger zkApp Transactions — [MIP9](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0009-increase-zkapp-account-update-limit.md) + +Mesa **roughly triples** the maximum number of account updates a single zkApp transaction can contain. The previous weighted-cost formula (`10.26·np + 10.08·n2 + 9.14·n1 < 69.45`) is replaced with a simpler `np + n2 + n1 ≤ 16` rule that admits any balanced binary proof tree of height 4. + +This depends on the SNARK-worker parallelization shipped as a prerequisite for [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md) — without it, processing maximum-size transactions inside a 90-second slot would not be feasible. + +### Automode Upgrades + +For the first time in Mina's history, block producers can upgrade through a hard fork **without manual intervention**. The automode mechanism ships both the pre-fork and post-fork binaries in a single package, with a dispatcher that automatically transitions to the new chain when the fork activates. Archive node and Rosetta API operators still use manual mode — automode is not available for those node types. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for details. + +### Simplified Archive Upgrade + +Unlike the Berkeley upgrade (which required up to 48 hours for archive database conversion), the Mesa archive upgrade is a fast schema upgrade that completes in under a minute. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +## Upgrade Flow — End-to-End Timeline + +The Mesa upgrade follows four phases. The timeline below shows what happens at each stage, when it happens, and what **you** need to do. + +### Overview + +Mina's Mesa Upgrade — four-phase timeline from Pre-Upgrade through Post-Upgrade, with state finalization and network shutdown markers + +The upgrade moves through four phases — **Pre-Upgrade**, **State Finalization**, **Upgrade**, and **Post-Upgrade** — anchored by three key moments: + +| Milestone | When | What happens | +|---|---|---| +| **stop-transaction-slot** | Hours before the fork | Network stops accepting new transactions | +| **stop-network-slot** | Exactly 5 hours after stop-transaction-slot | Block production halts entirely | +| **Mesa genesis timestamp** | Exactly 3 hours after stop-network-slot | First Mesa block is produced | + +--- + +### Phase 1: Pre-Upgrade — weeks before the fork + +> **Goal:** Every participant is prepared and running the stop-slot release before the fork begins. + +The table below is a high-level summary. The concrete install commands (Debian, Docker, etc.) and hardware requirements live in the [Requirements](/network-upgrades/mesa/requirements) and [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) pages — start there once you know your role and deployment style. + +| Actor | What to do | +|---|---| +| **All operators** | Verify [hardware requirements](/network-upgrades/mesa/requirements) (32 GB RAM, 8-core CPU with AVX/BMI2). Back up keys. | +| **Block Producers** | Choose [upgrade mode](/network-upgrades/mesa/upgrade-modes): **automode** (recommended) or manual. Install stop-slot release [3.x.x](https://github.com/MinaProtocol/mina/releases). If automode, install the `mina-{network}-automode` package — see [Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). | +| **SNARK Coordinators** | A coordinator is a daemon node — follow the Block Producer path (automode or manual). | +| **Standalone SNARK Workers** | Workers spawned by a coordinator (`--run-snark-worker`) need no separate action — they inherit the coordinator's binary. Workers run as a separate `mina internal snark-worker` process or container must be redeployed with the Mesa release after the fork. | +| **Archive Nodes** | Install stop-slot release. Choose upgrade method: _trustless_ (run upgrade script now) or _trustful_ (import o1Labs dump later). If trustless, run the [archive upgrade script](/network-upgrades/mesa/archive-upgrade) — it is backward compatible and can be applied early. | +| **Exchanges** | Install stop-slot release. Update integrations (mina-signer, Rosetta API). Test on devnet. Plan deposit/withdrawal freeze window. | +| **zkApp Developers** | Update to the Mesa-compatible o1js version (`o1js@3.0.0-mesa.698ca` for the preflight chain), recompile your contract, and verify it on the [preflight network](/network-upgrades/mesa/preflight-network). Plan to redeploy on Mesa after the fork — every zkApp must be redeployed because the protocol version bump changes the verification key. {/* TODO(PR #1133): Link to the official o1js Mesa upgrade guide once published (cjjdespres #3220525842). */} | + +:::tip Readiness checklist +Review the [Requirements](/network-upgrades/mesa/requirements) and pick your [Upgrade Mode](/network-upgrades/mesa/upgrade-modes) before proceeding. +::: + +--- + +### Phase 2: State Finalization — hours before the fork (exactly 5 hours) + +> **Goal:** The network reaches consensus on a final state. No new transactions are accepted. + +At the predefined [**stop-transaction-slot**](/network-upgrades/mesa/fork-schedule), nodes stop accepting new user transactions. Block production continues for ~100 more slots with empty blocks (no coinbase, no fees) until the [**stop-network-slot**](/network-upgrades/mesa/fork-schedule) to ensure all nodes converge on the same state. The concrete slot numbers and timestamps live on the [Fork Schedule](/network-upgrades/mesa/fork-schedule) page. + +| Actor | What to do | +|---|---| +| **Block Producers** | **Keep your node running.** Block density during finalization is critical. Do not stop your node. | +| **SNARK Workers** | **Keep running.** Continue producing SNARK work. | +| **Archive Nodes** | Keep the archive node running to capture all finalized blocks. If doing trustless upgrade, run the upgrade script now if you haven't already. | +| **Exchanges** | **Disable MINA deposits and withdrawals.** Any transactions submitted after the stop-transaction-slot will not exist on the Mesa chain. | +| **zkApp Developers** | No action required. Monitor announcements. | + +:::danger For exchanges +Transactions submitted after the stop-transaction-slot **will not carry over** to the Mesa chain. Freeze all MINA activity before this point. +::: + +--- + +### Phase 3: Upgrade — fork day (network is down) + +> **Goal:** The network halts, state is exported, and the Mesa release is published. + +At the **stop-network-slot**, block production stops entirely. o1Labs exports the network state, builds the Mesa release with the final ledger baked in, and publishes packages and Docker images. + +| Actor | What to do | +|---|---| +| **Block Producers (automode)** | **Nothing.** Your node transitions to Mesa automatically. It will start producing blocks when the Mesa genesis timestamp arrives. | +| **Block Producers (manual)** | Stop your node. Wait for the Mesa release announcement. Install the new package. Restart with [updated flags](/network-upgrades/mesa/upgrade-steps/post-upgrade). | +| **SNARK Coordinators** | Same as block producers — automode transitions automatically, manual requires stop + install + restart. | +| **Standalone SNARK Workers** | Coordinator-spawned workers follow the coordinator automatically. Standalone workers (`mina internal snark-worker` process or container) must be stopped and redeployed with the Mesa binary; an older worker cannot submit work to a Mesa coordinator. | +| **Archive Nodes** | Install the Mesa archive node release. Point it at your upgraded database (trustless) or import the o1Labs SQL dump into a fresh database (trustful). | +| **Exchanges** | Install the Mesa release. Keep deposits/withdrawals disabled until Mesa block production begins. | + +--- + +### Phase 4: Post-Upgrade — after the fork + +> **Goal:** Block production resumes on the Mesa network. Normal operations return. + +Exactly **3 hours** after the _stop-network-slot_, at the predefined Mesa genesis timestamp, the first Mesa block is produced. + +| Actor | What to do | +|---|---| +| **Block Producers** | Verify your node is on the Mesa chain (`mina client status`). If automode, check that the `activated` file exists in your config directory (see [Troubleshooting](/network-upgrades/mesa/troubleshooting#how-do-i-know-which-binary-my-node-is-using)). Monitor block production. | +| **SNARK Workers** | Reconnect workers to the upgraded coordinator. Standalone workers must already be on the Mesa binary at this point. Verify SNARK work is being produced. | +| **Archive Nodes** | Verify the archive database is in sync. Run [validation checks](/network-upgrades/mesa/upgrade-steps/post-upgrade#in-depth-validation). Fix any missing blocks using archive tooling. | +| **Exchanges** | **Re-enable MINA deposits and withdrawals** once block production is confirmed and your systems are verified. | +| **zkApp Developers** | **Redeploy every zkApp** on Mesa — the protocol version bump invalidates Berkeley verification keys, so a redeploy is required before any zkApp can accept transactions. You can also take advantage of the expanded 32-field on-chain state at this point. | + +--- + +For step-by-step instructions per phase, see [Upgrade Steps](/network-upgrades/mesa/upgrade-steps). For end-to-end walkthroughs by role (block producer, archive node, zkApp developer, exchange), see [Examples](/network-upgrades/mesa/upgrade-steps/examples). + +## Upgrade Modes + +The Mesa upgrade supports two modes for daemon node operators: **[Automode](/network-upgrades/mesa/upgrade-modes)** (recommended — node handles the fork transition automatically) and **[Manual](/network-upgrades/mesa/upgrade-modes)** (operator stops the node, installs the Mesa release, and restarts). + +See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for the full comparison, requirements, and the persistent-filesystem / process-restart constraints that automode imposes. For low-level details on the dispatcher and dual-binary architecture, see [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +## Examples + +End-to-end walkthroughs of the four phases for different roles (block producer automode/manual, archive node, zkApp developer, exchange) live on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. +## Quick Reference by Operator Type + +| Operator Type | Key Pages | +|---|---| +| **Block Producers** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Modes](/network-upgrades/mesa/upgrade-modes), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **SNARK Workers / Coordinators** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Archive Node Operators** | [Requirements](/network-upgrades/mesa/requirements), [Archive Upgrade](/network-upgrades/mesa/archive-upgrade), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Rosetta API Operators** | [Requirements](/network-upgrades/mesa/requirements), [Archive Upgrade](/network-upgrades/mesa/archive-upgrade), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Exchanges** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade), [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) | + +## Network Details + +The values below describe the **current Mainnet (pre-fork)** chain. They will change after the Mesa fork activates — the post-fork Chain ID, Git SHA-1, and node build link will be published in the Mesa release announcement. + +``` +Chain ID (Mainnet, pre-fork) +a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1 + +Git SHA-1 (Mainnet, pre-fork) +ae112d3a96fe71b4ccccf3c54e7b7494db4898a4 + +Seed List +https://bootnodes.minaprotocol.com/networks/mainnet.txt + +Node build +https://github.com/MinaProtocol/mina/releases?q=mesa +``` + +{/* TODO(PR #1133): When the Mesa mainnet release is cut, add the post-fork Chain ID, Git SHA-1, and the specific release tag URL alongside the pre-fork values above. For the current preflight values, see [Preflight Network](/network-upgrades/mesa/preflight-network). */} diff --git a/docs/network-upgrades/mesa/preflight-network.mdx b/docs/network-upgrades/mesa/preflight-network.mdx index 9894f6a79..b900f7319 100644 --- a/docs/network-upgrades/mesa/preflight-network.mdx +++ b/docs/network-upgrades/mesa/preflight-network.mdx @@ -107,7 +107,7 @@ docker run --name mina-mesa-preflight -d \ gcr.io/o1labs-192920/mina-daemon:4.0.0-preflight-3f038cb-bookworm-mesa \ daemon \ --peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt \ - --libp2p-keypair /root/.mina-config/keys/libp2p-key + --libp2p-keypair /data/.mina-config/keys/libp2p-key ``` ### Example: Running with Debian Package @@ -144,7 +144,7 @@ mina daemon \ :::info Upgrading from Berkeley to Mesa If you have an existing Berkeley archive database that you want to upgrade to Mesa, please refer to the comprehensive [Archive Upgrade](archive-upgrade) guide for detailed instructions on: -- Migration prerequisites and requirements +- Upgrade prerequisites and requirements - Running the upgrade script - Rollback procedures - Database schema changes diff --git a/docs/network-upgrades/mesa/requirements.mdx b/docs/network-upgrades/mesa/requirements.mdx new file mode 100644 index 000000000..e83a6a79e --- /dev/null +++ b/docs/network-upgrades/mesa/requirements.mdx @@ -0,0 +1,49 @@ +--- +title: Requirements +sidebar_label: Requirements +hide_title: true +description: Hardware and connectivity requirements for daemon node operators upgrading to Mesa — block producers, SNARK coordinators, archive nodes, Rosetta API nodes, and seed nodes. +keywords: + - Mesa + - upgrade + - hardware requirements +--- + +# Requirements + +## Hardware Requirements + +Please note the following are the hardware requirements for each node type after the upgrade: + +| Node Type | Memory | CPU | Storage | Network | +|--|--|--|--|--| +| Mina Daemon Node | 32 GB RAM | 8 core processor with BMI2 and AVX CPU instruction set are required | 16 GB | 1 Mbps Internet Connection | +| SNARK Coordinator | 32 GB RAM | 8 core processor | 16 GB | 1 Mbps Internet Connection | +| SNARK Worker (per worker) | 8 GB RAM | 6 core/12 threads with BMI2 and AVX CPU instruction set are required | 1 GB | 1 Mbps Internet Connection | +| Archive Node | 32 GB RAM | 8 core processor | 64 GB | 1 Mbps Internet Connection | +| Rosetta API standalone Docker image | 8 GB RAM | 2 core processor | 16 GB | 1 Mbps Internet Connection | +| Rosetta API + Archive Node | 32 GB RAM | 8 core processor | 64 GB | 1 Mbps Internet Connection | +| Mina Seed Node | 64 GB RAM | 8 core processor | 16 GB | 1 Mbps Internet Connection | + +## Mina Daemon Requirements + +The Mesa daemon inherits the same networking and process-management requirements as the Berkeley daemon. Rather than restate them here, refer to the canonical sections in the Validator Node docs: + +- **IP and port configuration** — `--external-ip`, `--external-port`, default port `8302` (libp2p). See [Validator Node Requirements](/node-operators/validator-node/requirements#networking). +- **Auto-restart on crash** — systemd `Restart=always`, Docker `--restart=always`. See [Running mina node as a service](/node-operators/validator-node/connecting-to-the-network#running-mina-node-as-a-service). + +The hardware table above is Mesa-specific; everything else lives once in the Validator Node docs. + +## Seed Peer Requirements + +### Generation of libp2p keypair + +To ensure connectivity across the network, it is essential that all seed nodes start with the **same** `libp2p` keypair. +This consistency allows other nodes in the network to reliably connect. +Although the same libp2p keys can be reused from before the upgrade, if you need to manually generate new libp2p keys, use the following command: + +``` +mina libp2p generate-keypair --privkey-path +``` + +Further information on [generating a libp2p key pair](/node-operators/seed-peers/generating-a-libp2p-keypair) on Mina Protocol. diff --git a/docs/network-upgrades/mesa/troubleshooting.mdx b/docs/network-upgrades/mesa/troubleshooting.mdx new file mode 100644 index 000000000..8ef1b29c3 --- /dev/null +++ b/docs/network-upgrades/mesa/troubleshooting.mdx @@ -0,0 +1,75 @@ +--- +title: Troubleshooting +sidebar_label: Troubleshooting +hide_title: true +description: Troubleshooting the Mesa upgrade — checking which binary your node is running, dispatcher behavior, and debug mode for automode operators. +keywords: + - Mesa + - upgrade + - troubleshooting + - automode + - dispatcher + - debug +--- + +# Troubleshooting + +This page collects common operator questions and debugging tips for the Mesa upgrade. For low-level architecture and dispatcher internals, see [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +## How do I know which binary my node is using? + +Check whether the activation state file exists. The file is located at: + +`{config-directory}/auto-fork-mesa-{network_id}/activated` + +Where: +- `{config-directory}` is the path passed via `--config-directory` (defaults to `~/.mina-config` on the host, or `/root/.mina-config` in Docker) +- `{network_id}` is the network name (e.g., `mainnet` or `devnet`) + +For example, on a typical mainnet setup: + +```bash +ls ~/.mina-config/auto-fork-mesa-mainnet/activated +``` + +- **File does not exist**: your node is using the pre-fork (Berkeley) binary +- **File exists**: your node has transitioned to the Mesa binary + +## Can I run non-daemon commands or use a specific binary version? + +As described in [Dispatcher Limitations](/network-upgrades/mesa/appendix/upgrade-modes-details#dispatcher-limitations), the dispatcher supports `daemon`, `client`, and `--version`. For any other command — including `accounts list`, `ledger export`, and other CLI operations — **you must use the version-specific binary directly**: + +| Binary | Description | Full path | +|---|---|---| +| `mina-berkeley` | Pre-fork binary (current chain) | `/usr/lib/mina/berkeley/mina` | +| `mina-mesa` | Post-fork binary (Mesa chain) | `/usr/lib/mina/mesa/mina` | + +```bash +# Use mina-mesa for all non-daemon commands after the fork +mina-mesa client status +mina-mesa accounts list +mina-mesa ledger export + +# Use mina-berkeley for pre-fork queries +mina-berkeley client status +``` + +These symlinks (`mina-berkeley`, `mina-mesa`) are installed globally by the automode packages and are always available. They bypass the dispatcher entirely and run the binary directly, so they work regardless of the activation state. + +:::caution Do not manipulate the activation state file +While it is technically possible to force the dispatcher to use a specific runtime by creating or removing the `activated` file, **this is strongly discouraged**. Manually manipulating the state file while the network is live can put your node on the wrong chain. If you need to run a specific version, use `mina-berkeley` or `mina-mesa` directly instead. +::: + +## Debug mode + +Set `MINA_DISPATCHER_DEBUG=1` in your environment to see which binary and arguments the dispatcher is using: + +```bash +MINA_DISPATCHER_DEBUG=1 mina daemon ... +``` + +For a dry run (print the command without executing): + +```bash +MINA_DISPATCHER_DRYRUN=1 mina daemon ... +``` diff --git a/docs/network-upgrades/mesa/upgrade-modes.mdx b/docs/network-upgrades/mesa/upgrade-modes.mdx new file mode 100644 index 000000000..0bdbc3cf7 --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-modes.mdx @@ -0,0 +1,195 @@ +--- +title: Upgrade Modes +sidebar_label: Upgrade Modes +hide_title: true +description: Automode and Manual upgrade paths for the Mesa hard fork. Automode handles the transition automatically for daemon nodes; Manual gives operators full control. +keywords: + - Mesa + - upgrade + - automode + - manual + - daemon +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Upgrade Modes + +The Mesa upgrade supports two modes for daemon node operators: **Automode** (recommended — the daemon handles the fork transition automatically) and **Manual Mode** (the operator stops, installs the Mesa release, and restarts). Both modes reach the same end state — a node running on the Mesa network. + +**Archive node operators, Rosetta API operators, and standalone SNARK workers must use Manual mode** — Automode is not available for those node types. + +## Comparison + +| Aspect | Automode | Manual | +|---|---|---| +| Operator intervention during fork | None | Stop, install, restart | +| Downtime | Minimal (automatic transition) | Depends on operator response time | +| Applies to | Daemon nodes only (block producers, SNARK coordinators) | All node types (incl. archive, Rosetta, standalone SNARK workers) | +| Control | Automated | Full manual control | +| Risk | Lower (fewer manual steps) | Higher (depends on operator timing) | + +## Choose your upgrade mode + +Pick the tab that matches the mode you intend to run. If you are unsure, start with the Automode tab — it is the recommended path for daemon nodes. + + + + +Automode is the recommended upgrade path for the Mina daemon — the process that participates in consensus. This includes block producers and SNARK coordinators (a SNARK coordinator runs as a daemon). In automode, the node handles the entire fork transition automatically. + +### Requirements + +- Install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) **before** the _stop-transaction-slot_ +- Ensure your node remains running through the State Finalization phase +- Meet the [hardware requirements](/network-upgrades/mesa/requirements) + +### Who should use Automode + +- Block producers who want a hands-off upgrade experience +- SNARK coordinators (run as a Mina daemon) +- Operators who want to minimize downtime and operational risk + +Automode is **not** available for the following — operators of these node types must switch to the Manual Mode tab: + +- **Archive nodes** — schema upgrade is a separate manual step (see [Archive Upgrade](/network-upgrades/mesa/archive-upgrade)) +- **Rosetta API nodes** — upgrade alongside the archive node they depend on +- **Standalone SNARK workers** — `mina internal snark-worker` processes/containers run separately from any daemon; they must be stopped and redeployed with the Mesa binary after the fork. (Workers spawned by a coordinator inherit the coordinator's upgrade automatically.) + +### How automode works during the fork + +1. You install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) during the Pre-Upgrade phase. +2. Your node participates normally in block production through the State Finalization phase. +3. When the network reaches the _stop-network-slot_, the node automatically: + - Stops producing blocks on the old chain + - Transitions to the Mesa network configuration + - Begins producing blocks on the Mesa network once the genesis timestamp is reached +4. No manual intervention is required during the fork. + +:::danger Automode requires a persistent config directory and an automatic restart policy +At the fork, the daemon **exits cleanly** (exit code 0) after writing the Mesa configuration and an `activated` marker to the config directory. It does **not** restart itself. Your process manager must restart it, and the config directory must survive that restart for the dispatcher to find the `activated` marker. + +- **Persistent config directory**: `~/.mina-config` (host) or a named Docker volume / Kubernetes `PersistentVolumeClaim` mounted at `/root/.mina-config` (container). An ephemeral `emptyDir` will lose the `activated` file and break the transition. +- **Automatic restart policy**: `Restart=always` on systemd (default in the Mina unit), `--restart=always` on Docker, `restartPolicy: Always` on Kubernetes. + +Failing to meet either requirement means your node will not join the Mesa network at genesis. See [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details#restart-and-filesystem-requirements) for the underlying mechanism. +::: + +### Installing automode + +Pick the install method that matches how you run your daemon today. + + + + +Install the automode package — it ships the pre-fork binary, the post-fork binary, and the dispatcher together so the daemon routes to the correct binary before and after the fork: + +```bash +sudo apt-get update +sudo apt-get install mina-{network}-automode=4.x.x +``` + +The package installs the pre-fork binary at `/usr/lib/mina/berkeley/mina`, the post-fork binary at `/usr/lib/mina/mesa/mina`, the dispatcher at `/usr/local/bin/mina-dispatch`, and the dispatcher configuration in `/etc/default/mina-dispatch`. + +`mina-{network}-automode` is an umbrella package — it depends on `mina-{network}-prefork-mesa` and `mina-{network}-postfork-mesa`, which apt pulls in automatically. You do not need to install them by hand. + +Start your node as usual — the dispatcher routes to the correct binary based on fork state. See the Debian automode example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + +Use the `mina-daemon-auto-hardfork` image (the regular `mina-daemon` image is manual-mode only): + +``` +minaprotocol/mina-daemon-auto-hardfork:{version}-{codename}-{network} +``` + +The image bundles both binaries and the dispatcher, and sets `MINA_APP=/usr/local/bin/mina-dispatch` so the entrypoint uses the dispatcher automatically. Start it with your normal flags **plus** `--restart=always` and a named volume mounted at `/root/.mina-config` — see the Docker automode example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + +For docker-compose users, see the [docker-compose quickstart](/network-upgrades/mesa/appendix/automode-docker-compose-quickstart) (under Appendix) for a ready-to-edit compose file. + + + + + + + +Manual mode gives operators full control over each step of the upgrade process. **Archive node operators and Rosetta API operators must use manual mode** — Automode is not available for those node types. + +### Requirements + +- Install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) **before** the _stop-transaction-slot_ +- Be prepared to act promptly when the Mesa release is published to minimize downtime +- Meet the [hardware requirements](/network-upgrades/mesa/requirements) + +### Who should use Manual mode + +- Operators who need full control over the upgrade process +- Operators with custom deployment pipelines that require explicit upgrade steps +- Operators who want to validate the Mesa build before restarting +- **Archive node operators** (automode is not available — schema upgrade is a separate manual step) +- **Rosetta API operators** (Rosetta upgrades alongside the archive node it depends on) + +### How manual mode works during the fork + +1. You install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) during the Pre-Upgrade phase. +2. Your node participates normally through the State Finalization phase. +3. When the network halts at the _stop-network-slot_, you: + - Stop your node + - Wait for the Mesa release to be published + - Install the Mesa release + - Restart your node with the [updated flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) +4. Your node begins participating in the Mesa network once the genesis timestamp is reached. + +### Installing manual mode + + + + +Install only the stop-slot release before the fork: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet=3.x.x +``` + +After the fork, when the Mesa release is published, stop your node and install the Mesa package: + +```bash +sudo systemctl stop mina +sudo apt-get install mina-mainnet=4.x.x +sudo systemctl start mina +``` + +See the Debian manual example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + +Use the regular `mina-daemon` image (not `mina-daemon-auto-hardfork`): + +``` +minaprotocol/mina-daemon:-bullseye-mainnet +``` + +After the fork, stop the container and start a new one with the Mesa image: + +```bash +docker stop mina && docker rm mina +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon ... +``` + +See the Docker manual example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + + + + +For in-depth technical details on how the upgrade mechanism works internally — dispatcher routing, dual-binary architecture, fork-state file — see [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details). diff --git a/docs/network-upgrades/mesa/upgrade-steps/examples.mdx b/docs/network-upgrades/mesa/upgrade-steps/examples.mdx new file mode 100644 index 000000000..6b54d1b1d --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-steps/examples.mdx @@ -0,0 +1,296 @@ +--- +title: Examples +sidebar_label: Examples +hide_title: true +description: End-to-end Mesa upgrade walkthroughs by role and deployment style — block producer (automode/manual), archive node, zkApp developer, and exchange. +keywords: + - Mesa + - upgrade + - examples + - block producer + - archive node + - zkApp + - exchange +--- + +# Examples + +The walkthroughs below follow concrete operators through the four Mesa upgrade phases (Pre-Upgrade, [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization), [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade)) for different roles and setups. Click to expand the example that matches your situation. + +
+Example: Block Producer Upgrading with Automode (Debian) + +Imagine you are **Alice**, a block producer running on a Debian server. Here is what your upgrade looks like end to end. + +**Weeks before the fork** — Alice checks [hardware requirements](/network-upgrades/mesa/requirements) and installs the automode packages: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet-automode=4.x.x +``` + +She starts her node normally. The automode dispatcher runs the pre-fork binary until the fork activates. + +> For the full preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork (State Finalization)** — The network reaches the _stop-transaction-slot_. Alice's node keeps producing blocks — she does **nothing**. Empty blocks are produced for 100 slots (exactly 5 hours) until all nodes agree on the final state. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts at the _stop-network-slot_. Alice's daemon generates the Mesa configuration, writes the `activated` marker file, and **shuts down cleanly** (exit code 0). Because Alice uses systemd with `Restart=always`, the daemon restarts automatically. On restart, the dispatcher detects the `activated` file and launches the Mesa binary. Alice does **nothing** — this all happens automatically. + +> For what happens if you chose manual mode instead, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). For details on the restart mechanism, see [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details#restart-and-filesystem-requirements). + +**After the fork (Post-Upgrade)** — Exactly 3 hours after the _stop-network-slot_, the first Mesa block is produced. Alice verifies: + +```bash +# Check if the activated file exists (path depends on your config directory and network ID) +ls ~/.mina-config/auto-fork-mesa-mainnet/activated + +# Confirm Mesa chain ID +mina client status +``` + +:::note Dispatcher and non-daemon commands +The automode dispatcher only supports the `daemon` subcommand and `client status` command — it cannot determine the active runtime for other commands because they don't pass the config directory. For commands like `accounts list`, or `ledger export`, use the correct version-specific binary directly: `mina-mesa` (after the fork) or `mina-berkeley` (before the fork). The full paths `/usr/lib/mina/mesa/mina` and `/usr/lib/mina/berkeley/mina` also work. This limitation may be removed in a future release. +::: + +She's done. Her node is producing blocks on Mesa. + +> For post-upgrade flags and monitoring, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Block Producer — Manual Mode (Docker) + +**Carlos** runs a block producer using Docker and prefers manual control over the upgrade. + +**Weeks before the fork** — Carlos pulls the stop-slot Docker image and starts his node: + +```bash +docker pull minaprotocol/mina-daemon:-bullseye-mainnet + +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon \ + --block-producer-key /keys/my-wallet \ + --config-directory /root/.mina-config \ + --libp2p-keypair /keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +> For the full preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork (State Finalization)** — Carlos keeps his node running. He does **nothing** during this phase. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts at the _stop-network-slot_. Carlos waits for the Mesa release announcement, then swaps to the new image. + +> Throughout these examples, `` is a placeholder for the Mesa release tag announced for your target network. For the **preflight** network the current value is published on [Preflight Network](/network-upgrades/mesa/preflight-network); for **devnet/mainnet** the value will be published with the corresponding release announcement. + +```bash +docker stop mina && docker rm mina + +docker pull minaprotocol/mina-daemon:-bullseye-mainnet + +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon \ + --block-producer-key /keys/my-wallet \ + --config-directory /root/.mina-config \ + --libp2p-keypair /keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork (Post-Upgrade)** — Exactly 3 hours after the _stop-network-slot_, block production starts. Carlos verifies: + +```bash +docker exec mina mina client status # confirms Mesa chain ID +``` + +> For post-upgrade flags and monitoring, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Archive Node / Explorer Operator + +**Eve** runs an archive node, a Rosetta API instance, and a block explorer. She needs to upgrade both the daemon and the database. + +**Weeks before the fork** — Eve installs the stop-slot release and decides on her upgrade method: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet=3.x.x +``` + +She chooses the **trustless** path — running the upgrade script early, while her archive is still online: + +```bash +# Back up the database first +pg_dump -U archive_db > berkeley-archive-backup.sql + +# Download and run the upgrade script (completes in under 1 minute) +curl -O https://raw.githubusercontent.com/MinaProtocol/mina/refs/heads/mesa/src/app/archive/upgrade_to_mesa.sql +psql -U -d archive_db -f upgrade_to_mesa.sql + +# Verify +psql -U -d archive_db -c "SELECT * FROM version;" +``` + +The script is backward-compatible — her existing Berkeley archive node keeps working normally after the upgrade. + +> For the full archive upgrade guide, see [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +**Hours before the fork (State Finalization)** — Eve keeps her archive node running to capture all finalized blocks. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts. Eve installs the Mesa archive release and points it at her already-upgraded database: + +```bash +sudo systemctl stop mina-archive +sudo systemctl stop mina + +sudo apt-get update +sudo apt-get install mina-archive-mainnet=4.x.x mina-mainnet=4.x.x + +# Start archive process pointing to the upgraded DB +mina-archive run \ + --postgres-uri postgres://:@localhost:5432/archive_db \ + --server-port 3086 \ + --log-json --log-level DEBUG + +# Start the non-block-producing daemon connected to the archive +mina daemon \ + --archive-address localhost:3086 \ + --config-directory ~/.mina-config \ + --libp2p-keypair ~/keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +She also restarts Rosetta: + +```bash +docker run --name rosetta --rm -d \ + -p 3088:3088 \ + --entrypoint '' \ + minaprotocol/mina-rosetta:-bullseye-mainnet \ + /usr/local/bin/mina-rosetta \ + --archive-uri "postgres://:@localhost:5432/archive_db" \ + --graphql-uri "http://localhost:3085/graphql" \ + --log-json --port 3088 +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork (Post-Upgrade)** — Block production resumes. Eve verifies data integrity: + +```bash +# Check the archive is in sync +mina client status + +# Run the verification toolbox +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri postgres://:@localhost:5432/archive_db \ + --protocol-version \ + --migration-version +``` + +She checks her explorer UI to confirm new Mesa blocks are appearing and the historical data is intact. + +> For the full validation workflow, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) and the [Archive Hardfork Toolbox](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox). + +
+ +
+Example: zkApp Developer + +**Frank** maintains a zkApp deployed on mainnet. His contract uses on-chain state and he wants to take advantage of Mesa's expanded 32-field state. + +**Weeks before the fork** — Frank updates his o1js dependency to the Mesa-compatible version and tests his zkApp on the [preflight network](/network-upgrades/mesa/preflight-network): + +```bash +npm install o1js@3.0.0-mesa.698ca +``` + +This is the first o1js release that targets the Mesa transaction protocol. `o1js@latest` currently resolves to a pre-Mesa version and will produce transactions the post-fork preflight network rejects. See [Preflight Network](/network-upgrades/mesa/preflight-network) for the matching daemon/archive/rosetta release. + +He verifies that: +- His contract recompiles cleanly against Mesa-compatible o1js and deploys on the preflight Mesa network +- Transactions execute end-to-end against the redeployed contract +- If he plans to use the expanded state fields (indexes `8–31`), his updated contract version compiles and deploys on preflight + +> For details on testing with the preflight network, see [Preflight Network](/network-upgrades/mesa/preflight-network). + +**Hours before the fork (State Finalization)** — Frank does **nothing**. His deployed zkApp keeps running on the Berkeley chain. No transactions can be submitted during this phase anyway. + +**Fork day (Upgrade)** — Frank does **nothing during the network halt**. His zkApp account and on-chain state values carry over to the Mesa chain automatically (including state fields at indexes `0–7`). However, the verification key generated for Berkeley is no longer valid under the Mesa protocol — the contract cannot process transactions on Mesa until it is redeployed. + +**After the fork (Post-Upgrade)** — Block production resumes on Mesa. Frank **must redeploy his zkApp** before it can accept transactions, regardless of whether he uses the new state fields: + +```bash +# Required for every zkApp — the Mesa protocol version bump invalidates the Berkeley verification key +zkapp deploy --network mainnet +``` + +If he is also adopting the expanded `8–31` state slots, this same deployment step ships the updated contract version that declares the new fields. + +> For post-upgrade details, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Exchange Upgrading to Mesa + +**Bob** is an exchange operator. His main concern is avoiding lost deposits. + +**Weeks before the fork** — Bob tests his integration (Rosetta API, mina-signer) on the [preflight network](/network-upgrades/mesa/preflight-network). He reviews [schema changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) and installs the stop-slot release: + +```bash +sudo apt-get install mina-mainnet=3.x.x +``` + +> For the full exchange preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork** — Before the _stop-transaction-slot_ arrives, Bob **disables MINA deposits and withdrawals** on his platform and notifies customers about the maintenance window. + +:::danger +Any transactions submitted after the stop-transaction-slot **will not exist on the Mesa chain**. This is the most critical step for exchanges. +::: + +**Fork day** — The network halts. Bob waits for the Mesa release announcement, then upgrades: + +```bash +sudo systemctl stop mina +sudo apt-get update +sudo apt-get install mina-mainnet=4.x.x +sudo systemctl start mina +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork** — Block production resumes. Bob verifies his node is on the Mesa chain, confirms Rosetta API is working, then **re-enables MINA deposits and withdrawals**. + +```bash +mina client status # verify Mesa chain ID +# test a small internal transfer before opening to customers +``` + +> For post-upgrade flags and configurations, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
diff --git a/docs/network-upgrades/mesa/upgrade-steps/index.mdx b/docs/network-upgrades/mesa/upgrade-steps/index.mdx new file mode 100644 index 000000000..33ac23b6f --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-steps/index.mdx @@ -0,0 +1,45 @@ +--- +title: Upgrade Steps +sidebar_label: Upgrade Steps +hide_title: true +description: Step-by-step Mesa upgrade instructions with per-role timelines and examples for block producers, SNARK workers, archive nodes, and exchanges. +keywords: + - Mesa + - upgrade + - steps + - phases + - block producer + - SNARK worker + - archive node + - exchange +--- + +# Upgrade Steps + +The Mesa upgrade proceeds through four sequential phases. Each phase has specific actions for different node operator types. + +| Phase | Description | +|---|---| +| [Pre-Upgrade](/network-upgrades/mesa/requirements) | Prepare infrastructure, upgrade to the stop-slot release, run archive upgrade scripts | +| [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization) | 100-slot stabilization period — no new transactions accepted, block production continues | +| [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade) | Network halts, state is exported, Mesa build is published | +| [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | Block production resumes on Mesa, flags and configurations for the new network | + +**Please note:** A simplified Node Status service will be part of the upgrade tooling and enabled by default in the Pre-Upgrade release with stop-slots ([3.x.x](https://github.com/MinaProtocol/mina/releases)). This feature allows for a safe upgrade by monitoring the amount of upgraded active stake. Only non-sensitive data is reported. + +Before proceeding, make sure you have reviewed the [Requirements](/network-upgrades/mesa/requirements) and chosen your [Upgrade Mode](/network-upgrades/mesa/upgrade-modes). + +## Walkthroughs by role + +The per-phase pages above describe what every role does at each step. For end-to-end narrative walkthroughs with concrete commands by role and deployment style (block producer automode/manual, archive node, zkApp developer, exchange), see the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + +Role-specific quick references: + +- **Block Producers & SNARK Coordinators** — pick a mode on [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). Install commands for automode are in [Upgrade Modes — Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). +- **SNARK Workers** — coordinator-spawned workers inherit the coordinator's binary; standalone `mina internal snark-worker` deployments must be redeployed with the Mesa binary after the fork. +- **Archive Node Operators** — archive nodes do **not** support automode. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade) for the schema upgrade and trustless/trustful paths. +- **Exchanges** — disable MINA deposits and withdrawals before the _stop-transaction-slot_ and keep them disabled until Mesa block production resumes. See [Exchanges](/network-upgrades/mesa/upgrade-steps/post-upgrade) on the Post-Upgrade page for the re-enable checklist. + +:::danger For exchanges +Any transactions submitted after the _stop-transaction-slot_ **will not exist on the Mesa chain**. Disable deposits and withdrawals **before** [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization) begins and keep them disabled until block production resumes on Mesa. +::: diff --git a/docs/network-upgrades/mesa/upgrade-steps/post-upgrade.mdx b/docs/network-upgrades/mesa/upgrade-steps/post-upgrade.mdx new file mode 100644 index 000000000..8dfa2e37a --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-steps/post-upgrade.mdx @@ -0,0 +1,420 @@ +--- +title: Post-Upgrade +sidebar_label: Post-Upgrade +hide_title: true +description: Post-upgrade verification, health checks, and monitoring for the Mesa mainnet network. +keywords: + - Mesa + - upgrade + - post-upgrade + - health check + - monitoring + - verification +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Post-Upgrade + +Exactly 3 hours after the network reaches the _stop-network-slot_, at the predefined Mesa genesis timestamp, block production starts and the network is successfully upgraded. + +This page helps you **verify your node is healthy** and running on the Mesa chain. + +The checks are split into separate tabs for **Rosetta API** and **Exchange** because the two roles overlap but are not the same. _Rosetta API_ here means an operator running the `mina-rosetta` integration layer (typically alongside an archive node). _Exchange_ means an operator running the customer-facing platform that ingests Mina balances and submits payments. Most exchanges use Rosetta, so they will work through both tabs; a handful operate without Rosetta (custom GraphQL integrations) and can skip the Rosetta tab. + +## Verify Your Node + +First, confirm your node is on the Mesa chain — this check is identical for every role: + +```bash +# If using automode, check the activation file exists +ls ~/.mina-config/auto-fork-mesa-mainnet/activated + +# Check node status +mina client status +``` + +You should see: +- **Sync status**: `Synced` +- **Chain ID** matching the Mesa chain ID for your network. The chain ID changes at the fork and is network-specific (mainnet, devnet, and preflight each get their own) — the exact value is published in the Mesa release announcement alongside the first Mesa packages. +- **Git SHA-1** matching the published Mesa daemon commit +- **Genesis timestamp** matching the published Mesa genesis timestamp +- **Block height** advancing + +Then run the role-specific checks below. + + + + +#### Verify block production + +Verify block production the same way you would on Berkeley — see [Block Producer Node](/node-operators/block-producer-node) for the relevant `mina client status` fields and checks. If you are a Delegation Program participant, your uptime continues to be tracked on the Mesa chain. + +#### Check logs for errors + +```bash +# Look for any errors in recent logs +journalctl -u mina --since "1 hour ago" --no-pager | grep -i error + +# For Docker +docker logs mina --since 1h 2>&1 | grep -i error +``` + + + + +#### Verify SNARK workers are connected + +```bash +# Check that workers are producing proofs +mina client status | grep -i snark +``` + +SNARK workers are **not compatible across the fork** — the transaction SNARK changed in Mesa, so a Berkeley-era worker cannot produce valid work for a Mesa coordinator. After the fork, every worker must run a Mesa-compatible release. Workers spawned by the coordinator pick up the new binary automatically; standalone workers must be redeployed. + +To restart a standalone worker against the coordinator: + +```bash +/usr/lib/mina/mesa/mina internal snark-worker \ + --proof-level full \ + --shutdown-on-disconnect false \ + --daemon-address : +``` + +The automode dispatcher does not route `mina internal` subcommands, and `mina-mesa` is not on `PATH` in the Docker image — invoke the Mesa worker binary by its full install path (`/usr/lib/mina/mesa/mina`) as shown above. + + + + +#### 1. Verify archive data integrity + +```bash +# Check the schema-version table +psql -U -d -c "SELECT * FROM migration_history;" + +# Run the hardfork toolbox verification +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri postgres://:@localhost:5432/ \ + --protocol-version \ + --migration-version +``` + +#### 2. Check for missing blocks + +Query the archive database to see if blocks are being captured: + +```bash +psql -U -d -c "SELECT height, state_hash FROM blocks ORDER BY height DESC LIMIT 5;" +``` + +If blocks are missing, use the archive tooling to backfill. See [In-Depth Validation](#in-depth-validation) below. + + + + +#### 1. Complete the Archive Node checks above first + +Rosetta depends on a healthy archive database. + +#### 2. Verify Rosetta is responding + +```bash +curl -s http://localhost:3088/network/list \ + -H 'Content-Type: application/json' \ + -d '{"metadata":{}}' | jq . +``` + +You should see `mainnet` in the response. + +#### 3. Test a balance lookup + +```bash +curl -s http://localhost:3088/account/balance \ + -H 'Content-Type: application/json' \ + -d '{ + "network_identifier": {"blockchain":"mina","network":"mainnet"}, + "account_identifier": {"address":""} + }' | jq . +``` + + + + +#### 1. Verify your integration stack + +- Confirm Rosetta API is responding (if used) +- Confirm archive database is up to date (if used directly) +- Test a small internal MINA transfer before re-enabling customer-facing operations + +#### 2. Re-enable deposits and withdrawals + +Only re-enable MINA deposits and withdrawals after: +- Block production is confirmed (blocks are advancing) +- Your integration is verified end to end +- You have confirmed balances match expectations + + + + +## In-Depth Validation + +The checks in the tabs above cover basic health. This section provides deeper validation procedures for operators who want thorough verification. + + + + +1. **Verify signature kind** — query the GraphQL endpoint to confirm the correct signature kind: + ```graphql + query { + signatureKind + } + ``` + For mainnet, this should return `mainnet`. + +2. **Verify connectivity** — ensure your node has peers and is connected to the network. Check that the node is receiving and gossiping blocks. + + + + +Use the `mina-archive-hardfork-toolbox` to verify the upgrade. All commands require `--postgres-uri postgresql://:@:/`. See the [Archive Upgrade](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox) page for full toolbox documentation. + +**Verify schema upgrade:** +```bash +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri \ + --protocol-version \ + --migration-version +``` + +You can also verify manually: +```sql +SELECT * FROM migration_history; +``` + +**Validate fork block integrity** (after the fork activates): +```bash +mina-archive-hardfork-toolbox validate-fork \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` + +**Verify no commands after fork point:** +```bash +mina-archive-hardfork-toolbox fork-candidate no-commands-after \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` + +**Verify extended zkApp state columns exist:** +```sql +SELECT column_name FROM information_schema.columns +WHERE table_name = 'zkapp_states_nullable' +AND column_name LIKE 'element%' +ORDER BY column_name; +``` +Confirm columns `element0` through `element31` exist. + +**Check for missing blocks:** +```bash +mina-missing-blocks-auditor --archive-uri postgres://:@
:/ +``` + +**Compare block heights** — the archive height should match or be close to the daemon's reported height: +```sql +SELECT MAX(height) FROM blocks; +``` + + + + +Use `rosetta-cli` to verify the API conforms to the Rosetta specification: + +```bash +# Spec check +rosetta-cli check:spec --configuration-file config.json + +# Data check — aim for reconciliation coverage above 60% +rosetta-cli check:data --configuration-file config.json + +# Construction check (after block production starts) +rosetta-cli check:construction --configuration-file config.json --start-block 2 +``` + + + + +1. **Verify seed connectivity** — confirm all seeds listed in `https://bootnodes.minaprotocol.com/networks/mainnet.txt` are connectable. + +2. **Verify block production** — monitor that blocks are being produced at the expected cadence via block explorers and your node's logs. + +3. **Verify empty blocks during finalization** — confirm that all blocks between the _stop-transaction-slot_ and _stop-network-slot_ were empty (no user transactions, no coinbase, no fee transfers): + ```sql + SELECT b.height, b.state_hash + FROM blocks b + WHERE b.global_slot_since_genesis > + AND b.global_slot_since_genesis <= + AND ( + EXISTS (SELECT 1 FROM blocks_user_commands buc WHERE buc.block_id = b.id) + OR EXISTS (SELECT 1 FROM blocks_internal_commands bic WHERE bic.block_id = b.id) + ); + ``` + This query should return zero rows. + +:::note +In rare cases a non-empty block can appear after the _stop-transaction-slot_ — for example if a block producer and an archive operator both ran a build without the [stop-slot release](/network-upgrades/mesa/glossary#stop-slot-release). As long as the majority of stake upgraded, such a block lands on a short fork and does not affect the hard fork block, so it is not a problem for the upgrade. +::: + + + + +--- + +## Help Monitor the Network + +The Node Status reporting service is **enabled by default** on the Mesa daemon. If you want to keep it on and point it at the o1Labs collection endpoints, pass: + +```bash +--simplified-node-stats +--node-status-url https://nodestats.minaprotocol.com/submit/stats +--node-error-url https://nodestats.minaprotocol.com/submit/stats +``` + +- `--simplified-node-stats` (no value) makes the reported payload a minimal, non-sensitive subset (version, sync status, peer count). +- `--node-status-url` is the endpoint that receives status reports. +- `--node-error-url` is the endpoint that receives crash reports the daemon emits just before terminating. + +To opt out of the Node Status service entirely, pass `--disable-node-status` (no value). Note: the previously-documented `--node-stats-type full|none` argument was never accepted by the daemon — `--simplified-node-stats` and `--disable-node-status` are the actual flag names in the Mesa release. + +## Report Issues + +If you encounter any problems after the upgrade: + +- Report bugs on [GitHub](https://github.com/MinaProtocol/mina/issues) with the label `mesa` +- Reach out on [Discord](https://discord.gg/minaprotocol) in the appropriate channel + +--- + +## Flag and Configuration Reference + +The flags below are **unchanged from Berkeley** — if your node was running correctly before the fork, the same flags will work on Mesa. This section is provided as a reference for operators setting up fresh nodes or verifying their configuration. + +For full details, see the [Mesa release notes](https://github.com/MinaProtocol/mina/releases?q=mesa). {/* TODO(PR #1133): Replace with the specific Mesa release tag URL once published (e.g. `/releases/tag/4.0.0`). */} + +:::info What changed in Mesa +- **Only if you manually manage your genesis config or genesis ledgers, repoint `--genesis-ledger-dir` and `-config-file` at the Mesa-specific files.** Operators who rely on the package-installed genesis config — including all automode users — do not need to change anything. If you do override them, both ship inside the Mesa Debian/Docker package: `--genesis-ledger-dir` should be `/var/lib/coda/mesa` (or the equivalent path your package installed), and `-config-file` should be the Mesa runtime config (typically `/etc/mina/mesa/daemon.json`). +- **All other flags carry over unchanged** — `--block-producer-key`, `--libp2p-keypair`, `--peer-list-url`, `--external-ip/port`, log flags, etc. behave the same on Mesa as on Berkeley. +::: + +
+Block Producer flags + +``` +mina daemon + --block-producer-key + --config-directory + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt + +ENVIRONMENT VARIABLES + RAYON_NUM_THREADS=6 + MINA_LIBP2P_PASS + MINA_PRIVKEY_PASS +``` + +
+ +
+SNARK Coordinator flags + +``` +mina daemon + --config-directory + --enable-peer-exchange true + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt + --run-snark-coordinator + --snark-worker-fee 0.001 + --work-selection [seq|rand|roffset] + +ENVIRONMENT VARIABLES + MINA_LIBP2P_PASS +``` + +
+ +
+SNARK Worker flags + +``` +mina internal snark-worker + --proof-level full + --shutdown-on-disconnect false + --daemon-address + +ENVIRONMENT VARIABLES + RAYON_NUM_THREADS=8 +``` + +
+ +
+Archive Node flags + +Running an Archive Node involves a non-block-producing daemon connected to the archive process and a PostgreSQL database. For more information, see [Archive Node](/node-operators/archive-node). + +**Daemon (non-block-producing):** + +``` +mina daemon + --archive-address :3086 + --config-directory + --enable-peer-exchange true + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt + +ENVIRONMENT VARIABLES + MINA_LIBP2P_PASS +``` + +**Archive process:** + +``` +mina-archive run + --metrics-port + --postgres-uri postgres://:@
:/ + --server-port 3086 + --log-json + --log-level DEBUG +``` + +
+ +
+Rosetta API + +Once your Archive Node stack is running: + +```bash +docker run \ + --name rosetta --rm \ + -p 3088:3088 \ + --entrypoint '' \ + minaprotocol/mina-rosetta:-bullseye-mainnet \ + /usr/local/bin/mina-rosetta \ + --archive-uri "${PG_CONNECTION_STRING}" \ + --graphql-uri "${GRAPHQL_URL}" \ + --log-json \ + --log-level ${LOG_LEVEL} \ + --port 3088 +``` + +
diff --git a/docs/network-upgrades/mesa/upgrade-steps/state-finalization.mdx b/docs/network-upgrades/mesa/upgrade-steps/state-finalization.mdx new file mode 100644 index 000000000..4efd5d1b1 --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-steps/state-finalization.mdx @@ -0,0 +1,47 @@ +--- +title: State Finalization +sidebar_label: State Finalization +hide_title: true +description: State finalization phase of the Mesa upgrade — stabilization period between stop-transaction-slot and stop-network-slot. +keywords: + - Mesa + - upgrade + - state finalization + - stop-transaction-slot + - stop-network-slot +--- + +# State Finalization + +Between the predefined _stop-transaction-slot_ and _stop-network-slot_, a stabilization period of 100 slots will occur. During this phase, the network consensus will not accept new blocks with transactions in them, including coinbase transactions. This means all blocks produced during this period will be completely empty — no user commands, no coinbase rewards, and no fee transfers. The state finalization period ensures all nodes reach a consensus on the latest network state before the upgrade. + +During the state finalization slots, it is crucial to maintain a high block density. Therefore, block producers and SNARK workers shall continue running their nodes to support the network's stability and security. + +Archive nodes should also continue to execute to ensure finalized blocks are in the database and can be carried over to the upgraded schema, preserving the integrity and accessibility of the network's history. + +## Block Producers and SNARK Workers + +1. It is crucial for the network's successful upgrade that all block producers and SNARK workers maintain their block-producing nodes up and running throughout the state finalization phase. +2. If you are running multiple daemons like is common with many operators, you can run one single node at this stage. +3. If you are a Delegation Program operator, remember that your uptime data will continue to be tracked during the state finalization phase and will be considered for the delegation grant in the following epoch. + +:::info +During this phase, both Automode and Manual mode operators simply keep their nodes running. No special action is needed regardless of your upgrade mode. +::: + +## Archive Node Operators and Rosetta Operators + +**If you plan to do the _trustful_ upgrade, you can skip this step.** + +If you are doing the trustless upgrade, then: + +1. Continue to execute the archive node to ensure finalized blocks are in the database. +2. Execute the archive node upgrade script, which is backward compatible. +3. Continue to run archive node until after the network stops at the stop-network slot. +4. For more information on the archive node upgrade process, please refer to the [Archive Upgrade](/network-upgrades/mesa/archive-upgrade) section. + +## Exchanges + +Exchanges shall disable MINA deposits and withdrawals during the state finalization period (the period between _stop-transaction-slot_ and _stop-network-slot_) since any transactions after the _stop-transaction-slot_ will not be part of the upgraded chain. + +Note that this assumes the majority of block producers are running the [stop-slot release](/network-upgrades/mesa/glossary#stop-slot-release), which is what enforces the transaction cutoff. If your own node is still on a pre-stop-slot build, you might technically be able to submit transactions, but the block producers running the stop-slot release will discard any blocks containing them. diff --git a/docs/network-upgrades/mesa/upgrade-steps/upgrade.mdx b/docs/network-upgrades/mesa/upgrade-steps/upgrade.mdx new file mode 100644 index 000000000..b1c395208 --- /dev/null +++ b/docs/network-upgrades/mesa/upgrade-steps/upgrade.mdx @@ -0,0 +1,39 @@ +--- +title: Upgrade +sidebar_label: Upgrade +hide_title: true +description: Upgrade phase of the Mesa hard fork — network halt, state export, and Mesa build publication. +keywords: + - Mesa + - upgrade + - network halt + - state export +--- + +# Upgrade + +Starting at the _stop-network-slot_ the network will not produce nor accept new blocks, resulting in halting the network. During the upgrade period, o1Labs will use automated tooling to export the network state based on the block at the slot just before the _stop-transaction-slot_. The exported state will then be baked into the new Mesa build, which will be used to initiate the upgraded network. It is during the upgrade window that the Mesa network infrastructure will be bootstrapped, and seed nodes will become available. o1Labs will also finalize the archive node upgrade and publish the PostgreSQL database dumps for import by the archive node operators who wish to bootstrap their archives in a trustful manner. + +There is a tool available to validate that the Mesa node was built from the pre-upgrade network state. See [In-Depth Validation](/network-upgrades/mesa/upgrade-steps/post-upgrade#in-depth-validation) for the `mina-archive-hardfork-toolbox` commands (`validate-fork`, `fork-candidate no-commands-after`) used to verify fork-block integrity. {/* TODO(PR #1133): If/when an authoritative `upgrading-to-mesa.md` lands in MinaProtocol/mina, link to it here. The previous link to `mina/blob/mesa/docs/upgrading-to-mesa.md` returned 404 — the `mesa` branch does not exist on the main repo. */} + +## Block Producers and SNARK Workers + +:::info +If you are using [Automode](/network-upgrades/mesa/upgrade-modes), your node handles this phase automatically. It will transition to the Mesa network without manual intervention. Skip to [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) for monitoring guidance. +::: + +If you are using **Manual mode**: + +1. During the upgrade phase (between _stop-network-slot_ and the publishing of the Mesa release), block producers can shut down their nodes. +2. After the publication of the Mesa node release, block producers and SNARK workers should upgrade their nodes and be prepared for block production at the genesis timestamp, which is the slot when the first Mesa block will be produced. +3. It is possible to continue using the same libp2p key after the upgrade. Pass it to the Mesa daemon with the [`--libp2p-keypair`](/network-upgrades/mesa/upgrade-steps/post-upgrade#flag-and-configuration-reference) flag (see the Flag and Configuration Reference on the Post-Upgrade page). + +## Archive Node Operators and Rosetta Operators + +1. Upon publishing the archive node Mesa release, archive node operators and Rosetta operators should upgrade their systems. There will be both Docker images and archive node releases available to choose from. +2. Depending on the chosen upgrade method: + - _Trustless_ + - Operators should direct their Mesa archive process to the previously upgraded database. + - _Trustful_ + - Operators shall import the SQL dump file provided by o1Labs to a freshly created database. + - Operators should direct their Mesa archive process to the newly created database. diff --git a/docs/node-operators/archive-node/replayer.mdx b/docs/node-operators/archive-node/replayer.mdx new file mode 100644 index 000000000..6baa323e1 --- /dev/null +++ b/docs/node-operators/archive-node/replayer.mdx @@ -0,0 +1,274 @@ +--- +title: Archive Replayer +sidebar_label: Archive Replayer +hide_title: true +description: Use the mina-replayer tool to verify a Mina archive database by replaying its transactions and reconstructing the ledger state. +keywords: + - replayer + - archive + - verification + - ledger + - integrity + - hard fork +--- + +# Archive Replayer + +`mina-replayer` replays all transactions from a Mina archive database, applying them sequentially to reconstruct the ledger state. It is an **ongoing verification tool**: run it at any time to confirm that your archive database faithfully represents the canonical chain. It also plays a role at hard-fork time — see [Hard Fork Replay](#hard-fork-replay) at the end of this page. + +## What the Replayer Does + +The replayer reads transactions from the archive database in order (respecting global slot and sequence number) and applies them to a starting ledger. At each step, it verifies that the computed Merkle root matches what the archive recorded. This catches data corruption, missing transactions, or schema issues in your archive. + +## Prerequisites + +- A PostgreSQL database with Mina archive data +- The `mina-replayer` binary (ships with the Mina archive Debian package and the archive/daemon Docker images) +- An input JSON file specifying the starting ledger + +### Getting the binary + +`mina-replayer` ships with the Mina archive Debian package and the archive/daemon Docker images — there is no need to build it from source: + +- **Debian:** install the archive package, which puts `mina-replayer` on your `PATH`. +- **Docker:** run it straight from the archive image, for example: + ```bash + docker run --rm minaprotocol/mina-archive:-bullseye-mainnet mina-replayer --help + ``` + +## Basic Usage + +The replayer requires two arguments: an input file and a database connection. + +```bash +mina-replayer \ + --archive-uri postgres://:@:/ \ + --input-file input.json +``` + +### Required Flags + +| Flag | Description | +|---|---| +| `--archive-uri` | PostgreSQL connection string for the archive database | +| `--input-file` | JSON file specifying the starting ledger and target state | + +### Optional Flags + +| Flag | Description | +|---|---| +| `--output-file` | Write the final ledger state to this file | +| `--continue-on-error` | Don't stop on transaction application errors | +| `--checkpoint-interval` | Create intermediate checkpoint files every N blocks | +| `--checkpoint-output-folder` | Directory for checkpoint files | +| `--checkpoint-file-prefix` | Filename prefix for checkpoint files | +| `--genesis-ledger-dir` | Directory containing the genesis ledger | +| `--log-json` | Output logs in JSON format | +| `--log-level` | Console log level (e.g., `info`, `debug`, `spam`) | +| `--log-file` | Write logs to a file | + +## Input File Format + +The input file tells the replayer where to start. For a full replay from genesis: + +```json +{ + "genesis_ledger": { + "add_genesis_winner": false, + "s3_data_hash": "", + "hash": "" + } +} +``` + +If resuming from a previous checkpoint, include `start_slot_since_genesis` and any prior epoch data. + +## Replay Modes + +### Full Replay (from genesis) + +This is the most thorough verification. It replays every transaction from genesis, catching any inconsistency in the entire archive history. + +```bash +mina-replayer \ + --archive-uri \ + --input-file genesis-input.json \ + --output-file output.json +``` + +**Use this mode when:** you want full confidence that your archive is correct. + +### Replay from Checkpoint + +If you already have a checkpoint, you can replay just the portion after it instead of re-replaying the entire history. + +```bash +mina-replayer \ + --archive-uri \ + --input-file checkpoint.json +``` + +**Use this mode when:** you want to verify recent blocks without re-replaying the full history. + +## Using Checkpoints for Long Replays + +For mainnet, a full replay from genesis can take a long time. Use checkpoints to break it into resumable segments: + +```bash +mina-replayer \ + --archive-uri \ + --input-file input.json \ + --checkpoint-interval 10000 \ + --checkpoint-output-folder ./checkpoints \ + --checkpoint-file-prefix mainnet-replay +``` + +This creates a checkpoint file every 10,000 blocks. If the replay is interrupted, restart from the latest checkpoint by using it as the `--input-file`. + +## Troubleshooting + +### Merkle root mismatch + +The replayer verifies the computed ledger Merkle root against the archive at each block. A mismatch means your archive has missing or incorrect data. Check for: + +- Missing blocks (`mina-missing-blocks-auditor`) +- Database corruption +- Incomplete schema upgrade + +### Continue on error + +If you want to see all errors rather than stopping at the first one: + +```bash +mina-replayer --continue-on-error --archive-uri --input-file input.json +``` + +--- + +## Hard Fork Replay + +:::info Extra — only relevant during a hard fork +This section applies only while a hard fork is in progress; on a normal day you do not need it. +::: + +At a hard fork, the replayer does additional work: it replays the archive up to the fork point and exports the final ledger state as the genesis ledger for the new chain. Specifically, it: + +1. Replays all transactions from genesis (or a checkpoint) up to the fork point +2. Stops at the `slot_chain_end` (the stop-network-slot where the old chain halts) +3. Exports the final ledger state as a JSON checkpoint — the genesis ledger for the new chain +4. Lets you compare that checkpoint against the official fork config to verify your archive matches the canonical state + +### Hard fork flags + +| Flag | Description | +|---|---| +| `--hard-fork-target ` | The target hard fork (e.g. `mesa`) | +| `--stop-slot-config-file` | JSON file with the fork parameters (stop slots, epoch data) | +| `--hard-fork-output-file` | Output file for the post-fork genesis ledger checkpoint | + +### Stop-slot configuration file + +The stop-slot config defines the fork parameters. This file uses the same format as the daemon runtime config: + +```json +{ + "genesis": { + "genesis_state_timestamp": "2026-02-24T19:30:00Z" + }, + "ledger": { + "add_genesis_winner": false, + "s3_data_hash": "", + "hash": "" + }, + "daemon": { + "slot_tx_end": 1900, + "slot_chain_end": 1920, + "hard_fork_genesis_slot_delta": 40 + }, + "epoch_data": { + "staking": { + "seed": "", + "s3_data_hash": "", + "hash": "" + }, + "next": { + "seed": "", + "s3_data_hash": "", + "hash": "" + } + } +} +``` + +Key fields in `daemon`: + +- **`slot_tx_end`** — the stop-transaction-slot (no more transactions accepted after this slot) +- **`slot_chain_end`** — the stop-network-slot (chain halts here, this is the fork point) +- **`hard_fork_genesis_slot_delta`** — slot offset for the new genesis relative to the fork point + +### Running the hard fork replay + +```bash +mina-replayer \ + --archive-uri postgres://:@:/ \ + --input-file input.json \ + --hard-fork-target \ + --stop-slot-config-file stop-slot-config.json \ + --hard-fork-output-file output.json \ + --log-json \ + --log-level info +``` + +The replayer will: + +1. Start from the genesis ledger (or checkpoint) specified in `input.json` +2. Replay all transactions from the archive (user commands, internal commands, and zkApp transactions) +3. Stop at the `slot_chain_end` — blocks at or beyond this slot are excluded +4. Apply the hard fork migration to produce the post-fork ledger +5. Write the result to `output.json` + +### Output format + +The output file contains the genesis configuration for the new chain: + +```json +{ + "start_slot_since_genesis": 1960, + "genesis_ledger": { + "hash": "", + "s3_data_hash": "", + "add_genesis_winner": false + } +} +``` + +The `start_slot_since_genesis` is the new genesis slot, computed from the fork point plus `hard_fork_genesis_slot_delta`. + +### Verifying against the official fork config + +After producing the replayer output, compare it to the official fork configuration published with the release: + +```bash +# Compare ledger hashes (ignoring s3_data_hash, which may differ due to non-deterministic RocksDB metadata) +diff \ + <(jq -S 'del(.genesis_ledger.s3_data_hash)' output.json) \ + <(jq -S 'del(.genesis_ledger.s3_data_hash)' official-fork-config.json) +``` + +If the diff is empty, your archive database correctly represents the canonical chain state at the fork point. + +:::tip Why ignore s3_data_hash? +The `s3_data_hash` is a SHA3-256 hash of the gzipped RocksDB ledger directory. RocksDB includes non-deterministic metadata (timestamps, sequence numbers, compaction state) and gzip headers may also differ across runs. The logical ledger contents are identical even when this hash differs — the `hash` field (the Merkle root) is the authoritative check. +::: + +### Hard fork troubleshooting + +**"No blocks found before slot_chain_end"** — the replayer could not find any blocks in the archive before the fork point. Verify: + +- Your archive database contains blocks up to the fork slot +- The `slot_chain_end` in your stop-slot config is correct + +## Further Reading + +- [Archive Node](/node-operators/archive-node) — running and maintaining an archive node +- [Replayer source code](https://github.com/MinaProtocol/mina/tree/compatible/src/app/replayer) diff --git a/docusaurus.config.js b/docusaurus.config.js index d410eeaf0..bc32c21d5 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -280,6 +280,42 @@ module.exports = { from: '/mesa-upgrade/archive-upgrade', to: '/network-upgrades/mesa/archive-upgrade', }, + { + from: '/mesa-upgrade/flags-configs', + to: '/network-upgrades/mesa/upgrade-steps/post-upgrade', + }, + { + from: '/mesa-upgrade/appendix', + to: '/network-upgrades/mesa/appendix/archive-node-schema-changes', + }, + { + from: '/network-upgrades/mesa/appendix', + to: '/network-upgrades/mesa/appendix/archive-node-schema-changes', + }, + { + from: '/network-upgrades/mesa/upgrade-modes-details', + to: '/network-upgrades/mesa/appendix/upgrade-modes-details', + }, + { + from: '/network-upgrades/mesa/docker-compose-quickstart', + to: '/network-upgrades/mesa/appendix/automode-docker-compose-quickstart', + }, + { + from: '/network-upgrades/mesa/replayer', + to: '/node-operators/archive-node/replayer', + }, + { + from: '/mesa-upgrade/upgrade-steps/state-finalization', + to: '/network-upgrades/mesa/upgrade-steps/state-finalization', + }, + { + from: '/mesa-upgrade/upgrade-steps/upgrade', + to: '/network-upgrades/mesa/upgrade-steps/upgrade', + }, + { + from: '/mesa-upgrade/upgrade-steps/post-upgrade', + to: '/network-upgrades/mesa/upgrade-steps/post-upgrade', + }, { from: '/node-operators/requirements', to: '/node-operators/validator-node/requirements', diff --git a/sidebars.js b/sidebars.js index 84fa9ed2f..d5bf7510e 100644 --- a/sidebars.js +++ b/sidebars.js @@ -19,9 +19,10 @@ module.exports = { label: 'Berkeley Upgrade', link: { type: 'doc', - id: 'network-upgrades/berkeley/requirements', + id: 'network-upgrades/berkeley/index', }, items: [ + 'network-upgrades/berkeley/requirements', { type: 'category', label: 'Archive Migration', @@ -48,9 +49,41 @@ module.exports = { { type: 'category', label: 'Mesa Upgrade', + link: { + type: 'doc', + id: 'network-upgrades/mesa/index', + }, items: [ - 'network-upgrades/mesa/preflight-network', + 'network-upgrades/mesa/requirements', + 'network-upgrades/mesa/upgrade-modes', + 'network-upgrades/mesa/fork-schedule', + { + type: 'category', + label: 'Upgrade Steps', + link: { + type: 'doc', + id: 'network-upgrades/mesa/upgrade-steps/index', + }, + items: [ + 'network-upgrades/mesa/upgrade-steps/state-finalization', + 'network-upgrades/mesa/upgrade-steps/upgrade', + 'network-upgrades/mesa/upgrade-steps/post-upgrade', + 'network-upgrades/mesa/upgrade-steps/examples', + ], + }, 'network-upgrades/mesa/archive-upgrade', + 'network-upgrades/mesa/preflight-network', + 'network-upgrades/mesa/troubleshooting', + { + type: 'category', + label: 'Appendix', + items: [ + 'network-upgrades/mesa/appendix/archive-node-schema-changes', + 'network-upgrades/mesa/appendix/upgrade-modes-details', + 'network-upgrades/mesa/appendix/automode-docker-compose-quickstart', + ], + }, + 'network-upgrades/mesa/glossary', ], }, ], @@ -1529,6 +1562,7 @@ module.exports = { 'node-operators/archive-node/getting-started', 'node-operators/archive-node/archive-redundancy', 'node-operators/archive-node/docker-compose', + 'node-operators/archive-node/replayer', ], }, { diff --git a/static/img/network-upgrades/mesa/automode-flow.svg b/static/img/network-upgrades/mesa/automode-flow.svg new file mode 100644 index 000000000..52c926a61 --- /dev/null +++ b/static/img/network-upgrades/mesa/automode-flow.svg @@ -0,0 +1,125 @@ + + + + + + + + + + + + + + + + + + Automode Upgrade Flow + + + + + + + PRE-UPGRADE + Weeks before the fork + + + Install automode packages + prefork-mesa + postfork-mesa + + + Dispatcher routes to Berkeley binary + Node runs normally, produces blocks + + + + + + + + + STATE FINALIZATION + ~5 hours (100 slots) + + + stop-transaction-slot + + + Node keeps producing empty blocks + No transactions, no coinbase. Still on Berkeley binary. + + + + + + + + + UPGRADE (AUTOMATIC) + Fork day + + + stop-network-slot + + + + + 1 + Daemon detects stop-network-slot + + + + 2 + Generates Mesa config + genesis ledgers + + + + 3 + Writes activated marker file to config directory + + + + 4 + Daemon shuts down (exit code 0) + + + + + Process manager restarts daemon + systemd / Docker / Kubernetes + + + + + Dispatcher finds activated -> routes to Mesa binary + + + + + + + + + POST-UPGRADE + ~1 hour after release + + + Mesa genesis timestamp + + + Block production resumes on Mesa + Verify: auto-fork-mesa-*/activated exists in config dir + + + + + REQUIREMENTS + + + Persistent config directory (must survive restarts) + + + Restart policy (Restart=always or equivalent) + \ No newline at end of file diff --git a/static/img/network-upgrades/mesa/manual-flow.svg b/static/img/network-upgrades/mesa/manual-flow.svg new file mode 100644 index 000000000..8cd819c44 --- /dev/null +++ b/static/img/network-upgrades/mesa/manual-flow.svg @@ -0,0 +1,118 @@ + + + + + + + + + + + + + + + Manual Mode Upgrade Flow + + + + + + + PRE-UPGRADE + Weeks before the fork + + + Install stop-slot release (4.0.0) + Single binary (pre-fork only) + + + Node runs normally + Produces blocks, earns rewards + + + + + + + + + STATE FINALIZATION + ~5 hours (100 slots) + + + stop-transaction-slot + + + Keep your node running + Empty blocks, no transactions. No action required. + + + + + + + + + UPGRADE (MANUAL) + Fork day — operator action required + + + stop-network-slot + + Network halts. No more blocks produced. + + + + + 1 + Stop your node + + + + 2 + Wait for Mesa release announcement + + + + 3 + Install the Mesa release + New binary + config + genesis ledgers + + + + 4 + Update startup flags + See Post-Upgrade Flags page + + + + 5 + Start your node with the Mesa binary + + + + Every minute between the Mesa genesis timestamp and your restart = missed blocks + + + + + + + + + POST-UPGRADE + ~1 hour after release + + + Mesa genesis timestamp + + + Block production resumes on Mesa + Verify: mina client status shows Mesa chain ID + + + + + With automode, steps 1–5 happen automatically with no downtime at the fork. + \ No newline at end of file diff --git a/static/img/network-upgrades/mesa/mesa-upgrade-timeline.png b/static/img/network-upgrades/mesa/mesa-upgrade-timeline.png new file mode 100644 index 000000000..65c6dbe4d Binary files /dev/null and b/static/img/network-upgrades/mesa/mesa-upgrade-timeline.png differ diff --git a/static/img/network-upgrades/mesa/simplified-schedule-archive.png b/static/img/network-upgrades/mesa/simplified-schedule-archive.png new file mode 100644 index 000000000..2ca535939 Binary files /dev/null and b/static/img/network-upgrades/mesa/simplified-schedule-archive.png differ diff --git a/static/img/network-upgrades/mesa/simplified-schedule-echanges.png b/static/img/network-upgrades/mesa/simplified-schedule-echanges.png new file mode 100644 index 000000000..39f1d3474 Binary files /dev/null and b/static/img/network-upgrades/mesa/simplified-schedule-echanges.png differ diff --git a/static/img/network-upgrades/mesa/simplified-schedule-manual.png b/static/img/network-upgrades/mesa/simplified-schedule-manual.png new file mode 100644 index 000000000..3542f025f Binary files /dev/null and b/static/img/network-upgrades/mesa/simplified-schedule-manual.png differ diff --git a/static/img/network-upgrades/mesa/simplified-schedule.png b/static/img/network-upgrades/mesa/simplified-schedule.png new file mode 100644 index 000000000..4f1059d4d Binary files /dev/null and b/static/img/network-upgrades/mesa/simplified-schedule.png differ diff --git a/static/llms-full.txt b/static/llms-full.txt index eb441ca52..b96d46b17 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -5202,6 +5202,40 @@ minaprotocol/mina-rosetta:3.1.0-ae112d3-bullseye-mainnet \ --port 3088 ``` +--- +url: /network-upgrades/berkeley +--- + +# Berkeley Upgrade + +The Berkeley upgrade was the most significant network upgrade in Mina's history, transitioning mainnet from the legacy proof system to the Berkeley proof system. It was completed in June 2024. + +## What Berkeley Introduced + +### zkApp Programmability + +Berkeley brought full zkApp (zero-knowledge application) support to mainnet. Developers can deploy smart contracts that execute off-chain computation and generate zero-knowledge proofs verified on-chain, enabling privacy-preserving applications with minimal on-chain footprint. + +### Recursive Proofs + +The upgrade enabled recursive proof composition, allowing proofs to verify other proofs. This is foundational to Mina's constant-size blockchain — the entire chain state can be verified with a single proof regardless of history length. + +### New Transaction Model + +Berkeley introduced a new transaction format supporting zkApp commands alongside traditional payment and delegation transactions. This included on-chain state storage for smart contracts, events, and actions. + +### Archive Database Migration + +The upgrade required a full archive database migration from the legacy schema to the Berkeley schema. This was the most operationally intensive part of the upgrade for archive node operators, taking up to 48 hours for the trustless migration path. + +## Upgrade Details + +- **Release**: [3.0.0](https://github.com/MinaProtocol/mina/releases/tag/3.0.0) (Berkeley mainnet release) +- **Upgrade mode**: Manual only (automode was not available for Berkeley) +- **Archive migration**: Trustless (48h) or trustful (o1Labs database export) + +For the operational details of the Berkeley upgrade, see the sub-pages below. These are preserved for historical reference. + --- url: /network-upgrades/berkeley/requirements --- @@ -5395,7 +5429,7 @@ url: /network-upgrades # Network Upgrades -Mina protocol evolves through network upgrades (hard forks) that introduce new features and improvements. Each upgrade requires node operators to update their software to remain compatible with the network. +Mina Protocol evolves through network upgrades (hard forks). Each upgrade introduces new protocol features, performance improvements, or governance changes. Hard forks are not backward compatible — all node operators must upgrade before the fork activates. :::tip Mesa preflight hard fork completed @@ -5414,6 +5448,329 @@ See [Preflight Network](/network-upgrades/mesa/preflight-network) for the full u | [Berkeley](/network-upgrades/berkeley/requirements) | Completed | June 2024 | zkApp programmability, recursive proofs, new transaction model, archive database migration | | [Mesa](/network-upgrades/mesa/preflight-network) | Preflight hard fork completed | 2026-04-27 13:00 UTC (preflight) | Transaction protocol v5.0.0, automode upgrades, simplified archive migration | +--- +url: /network-upgrades/mesa/appendix/archive-node-schema-changes +--- + +# Archive Node Schema Changes + +## Upgrading archive nodes from Berkeley to Mesa + +Below we present details of what changed in the archive node database schema between Berkeley and Mesa versions. + +### Extended zkApp State Fields + +Both zkApp state tables have been modified to support additional state elements: + +**zkapp_states_nullable table** + - Added columns `element8` through `element31` (nullable integer fields) + - Each new column references `zkapp_field(id)` + - These fields allow zkApps to store additional state information beyond the original 8 elements + +```sql +ALTER TABLE zkapp_states_nullable ADD COLUMN IF NOT EXISTS element8 INT REFERENCES zkapp_field(id); +... +ALTER TABLE zkapp_states_nullable ADD COLUMN IF NOT EXISTS element31 INT REFERENCES zkapp_field(id); +``` + +**zkapp_states table** + - Added columns `element8` through `element31` (non-nullable integer fields) + - Each new column references `zkapp_field(id)` with a default value pointing to the zero field + - Unlike the nullable version, these fields are required and default to the zero field ID + +```sql +ALTER TABLE zkapp_states ADD COLUMN IF NOT EXISTS element8 INT DEFAULT NOT NULL REFERENCES zkapp_field(id); +... +ALTER TABLE zkapp_states ADD COLUMN IF NOT EXISTS element31 INT DEFAULT NOT NULL REFERENCES zkapp_field(id); +``` + +This expansion allows zkApps to store up to 32 state elements instead of the previous 8, significantly increasing the state storage capacity for complex smart contracts. + +### Version Tracking + +The upgrade introduces a new `migration_history` table to keep track of the database schema version. The purpose of this table is to help with future database migrations. The table tracks which migration scripts were applied and when. + +**migration_history table** +```sql +CREATE TABLE IF NOT EXISTS migration_history ( + commit_start_at timestamptz NOT NULL DEFAULT now() PRIMARY KEY, + protocol_version text NOT NULL, + migration_version text NOT NULL, + description text NOT NULL, + status migration_status NOT NULL +); + +``` + +The migration_history table provides: +- **Migration tracking**: Records which migrations have been applied +- **Timestamp tracking**: Shows when each migration was executed +- **Idempotency**: Prevents duplicate migration runs +- **Version identification**: Easily identify the current database schema version + +The table is created if it does not exist already. Rollback and upgrade scripts will insert a new row with the version number and timestamp when the script was applied. + +--- +url: /network-upgrades/mesa/appendix/automode-docker-compose-quickstart +--- + +# Automode Docker Compose Quickstart + +```yaml +services: + mina_node: + image: 'minaprotocol/mina-daemon-auto-hardfork:-noble-mainnet' + restart: always + environment: + MINA_CLIENT_TRUSTLIST: "0.0.0.0/0" + entrypoint: [] + command: > + bash -c ' + mina daemon \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --insecure-rest-server \ + --rest-port 3085 \ + --hardfork-handling migrate-exit + ' + volumes: + - './mina-config:/root/.mina-config' + ports: + - '3085:3085' + - '8302:8302' +``` + +Replace `` with the published stop-slot release from the [Mina releases page](https://github.com/MinaProtocol/mina/releases). + +:::tip About `--hardfork-handling` +The automode image still expects `--hardfork-handling migrate-exit`, so you always pass it here. The automode dispatcher consumes the flag and routes to the correct binary, so you do not need any conditional logic around it. Remove it only when you switch off the automode image (see [Switching back to normal Docker](#switching-back-to-normal-docker)). +::: + +:::tip Docker image tag +Use the stop-slot release image for your target network. For devnet, replace the image tag suffix `mainnet` with `devnet` and use the corresponding devnet release tag. +::: + +:::tip Peer list URL +The `--peer-list-url` above points to the mainnet bootnodes. For devnet, replace it with `https://bootnodes.minaprotocol.com/networks/devnet.txt`. +::: + +:::tip Additional environment variables +Add any extra configuration (e.g. `MINA_PRIVKEY_PASS`) under the `environment` section. +::: + +## Switching back to normal Docker + +After your node has completed the Mesa transition and you want to move off `mina-daemon-auto-hardfork`, update your Compose config as follows: + +- Remove `--hardfork-handling migrate-exit`. +- Set the Docker image to the post-fork artifact, for example `minaprotocol/mina-daemon:-noble-mainnet`. + +```bash +docker compose up -d +``` + +:::caution +`restart: always` and the persistent `mina-config` volume are required for automode. If the volume is wiped between restarts, the node cannot transition to Mesa. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for details. +::: + +--- +url: /network-upgrades/mesa/appendix/upgrade-modes-details +--- + +# Upgrade Modes - Details + +This page explains **how the two upgrade modes work under the hood**. If you just want to know which mode to pick and what to do, see [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). This page is for operators who want to understand the mechanism before trusting it with their nodes. + +## The Big Picture + +During the Mesa hard fork, the network transitions from the current chain (referred to internally as "berkeley") to the new Mesa chain. Both upgrade modes use the same **stop-slot mechanism** to halt the old chain at a predetermined slot. The difference is what happens next: + +- **Automode**: the node automatically switches to the Mesa binary and resumes. No operator action needed. +- **Manual mode**: the operator stops the old node, installs the Mesa release, and starts it themselves. + +Both modes reach the same end state — a node running on the Mesa network, producing blocks after the Mesa genesis timestamp. + +## Stop-Slot Mechanism + +Both modes rely on two critical slot numbers baked into the stop-slot release (3.x.x): + +| Slot | What happens | +|---|---| +| **stop-transaction-slot** | The network stops accepting transactions. Blocks produced after this slot are empty — no user commands, no coinbase rewards, no fee transfers. This begins the State Finalization period. | +| **stop-network-slot** | The network stops producing and accepting blocks entirely. The chain halts. This is the point where the fork happens. | + +The gap between these two slots is exactly 100 slots (5 hours) — the **State Finalization** period. It ensures all nodes converge on the same final state before the fork. + +:::tip For block producers +You **will not earn block rewards** during State Finalization (no coinbase), but you **must keep your node running** to maintain network stability and block density. If you are in the Delegation Program, uptime tracking continues during this phase. +::: + +## Automode — How It Works Internally + +### Dual-Binary Architecture + +The automode release ships **two complete sets of Mina binaries plus the dispatcher** in a single `mina-{network}-automode` Debian package: + +| Component | Path | Purpose | +|---|---|---| +| **Pre-fork binary** | `{RUNTIMES_BASE_PATH}/berkeley/mina` | Runs the current chain up to the stop-network-slot | +| **Post-fork binary** | `{RUNTIMES_BASE_PATH}/mesa/mina` | Runs the Mesa chain after the fork | +| **Dispatcher** (`mina-dispatch`) | `/usr/local/bin/mina-dispatch` | Routes your commands to the correct binary | + +When you install the automode package (or use the [automode Docker image](/network-upgrades/mesa/glossary#automode-image)), all three components are installed together. + +### The Dispatcher + +The dispatcher (`mina-dispatch`) is a shell script that acts as a transparent wrapper around the real `mina` binary. When you run a `mina` command, the dispatcher decides which binary to execute: + +- **`daemon` subcommand** — routes based on the activation state file: + ``` + Does the activation state file exist? + → NO: route to the pre-fork (berkeley) binary + → YES: route to the post-fork (mesa) binary + ``` +- **`client` subcommand** — always routes to the post-fork (mesa) binary, regardless of activation state. This works because the GraphQL schema did not change between Berkeley and Mesa. +- **`--version`** — passed through without processing. + +The **activation state file** is located at `{MINA_HARDFORK_STATE_DIR}/auto-fork-mesa-{network_id}/activated` (e.g., `~/.mina-config/auto-fork-mesa-mainnet/activated` on the host, or `/root/.mina-config/auto-fork-mesa-mainnet/activated` in Docker). This file is created by the daemon itself when it detects that the network has reached the stop-network-slot. + +### Dispatcher Limitations + +:::info Current implementation — may change in future releases +This limitation exists because, for most subcommands, the dispatcher does not receive the config directory location as an argument. Without access to the config directory, it cannot check for the `activated` state file and therefore cannot determine whether the node is running Berkeley or Mesa. +::: + +The dispatcher supports the following subcommands: + +| Subcommand | Routing behavior | +|---|---| +| `daemon` | Routes to pre-fork or post-fork binary based on activation state | +| `client` | Always routes to the post-fork (mesa) binary | +| `--version` | Passed through without processing | + +Any other subcommand (e.g., `accounts list`, `ledger export`) will fail with an error: + +``` +mina-dispatch ERROR: unsupported subcommand 'accounts' for automatic hardfork handling +``` + +For unsupported subcommands, invoke the correct version-specific binary directly: + +| Binary | When to use | Path | +|---|---|---| +| `mina-berkeley` | Before the fork (or to query pre-fork state) | `/usr/lib/mina/berkeley/mina` | +| `mina-mesa` | After the fork | `/usr/lib/mina/mesa/mina` | + +```bash +# These work at any time — they bypass the dispatcher +mina-berkeley client status +mina-mesa accounts list +mina-mesa ledger export + +# Full paths also work for any binary in either runtime +/usr/lib/mina/mesa/mina client status +``` + +:::note About `client` routing +The `client` subcommand (e.g., `mina client status`) is always routed to the mesa binary because it communicates with the running daemon over GraphQL, and the GraphQL schema did not change between Berkeley and Mesa. This means `mina client status` works through the dispatcher at any point — before or after the fork — as long as a daemon is running. +::: + +### What the Dispatcher Does for `daemon` Commands + +When the dispatcher routes a `daemon` command to the Mesa binary, it automatically adjusts your command-line arguments: + +1. **Config files**: Your existing `-config-file` arguments are kept, and the Mesa-specific configuration is **appended as the last** `-config-file` entry. This ensures Mesa settings take precedence. +2. **Genesis ledger directory**: Any `--genesis-ledger-dir` argument is rewritten to point to the Mesa ledger directory. +3. **Hardfork handling flag**: The `--hardfork-handling` argument is **removed** (it is not supported on the Mesa chain). + +### Automode Timeline + +Here is what happens from a block producer's perspective when using automode: + +Automode upgrade flow diagram showing four phases: pre-upgrade, state finalization, automatic upgrade with process restart, and post-upgrade + +### Restart and Filesystem Requirements + +The automode transition involves a **process restart**. When the daemon reaches the stop-network-slot, it: + +1. Generates the Mesa configuration (`daemon.json`) and genesis ledger tarballs in the config directory +2. Writes an `activated` sentinel file to mark the fork as complete +3. **Exits with code 0** (clean shutdown) + +The daemon does **not** restart itself. Your process manager must detect the exit and restart the process. On restart, the dispatcher sees the `activated` file and launches the Mesa binary with the auto-generated config. + +**This means two things are critical:** + +**Persistent config directory** — The config directory (typically `~/.mina-config` or `/root/.mina-config` in Docker) **must survive across restarts**. It contains the `activated` file and the generated Mesa configuration. If this directory is ephemeral or gets wiped on restart, the node will restart into the Berkeley binary and fail to join the Mesa network. + +**Automatic restart on clean exit** — Your process manager must be configured to restart the daemon after exit code 0: + +| Platform | Configuration | Notes | +|---|---|---| +| **systemd** | `Restart=always` | Default in the Mina systemd unit (`mina.service`). Restarts after 30 seconds. | +| **Docker** | `--restart=always` or `--restart=unless-stopped` | Set when creating the container. The default (`no`) will **not** restart. | +| **Kubernetes** | `restartPolicy: Always` | The k8s default for pods. Ensure your liveness probes and Helm chart configuration do not treat exit code 0 as a failure that triggers a volume wipe or full pod replacement. The config directory volume **must** be a `PersistentVolumeClaim`, not `emptyDir`. | + +:::danger For Kubernetes / Helm users +If your Helm chart or pod spec uses `emptyDir` for the config directory, the `activated` file and generated Mesa config will be lost when the pod restarts. Use a `PersistentVolumeClaim` instead. Also verify that your restart logic does not re-initialize the config directory from scratch — the auto-generated Mesa config must be preserved. +::: + +### How automode is packaged + +Operator install instructions (Debian packages, Docker image) live on [Upgrade Modes — Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). The detail worth keeping here is what each artifact contributes to the mechanism described above: + +- The `mina-{network}-automode` Debian package is an umbrella package whose dependencies pull in the pre-fork binary, post-fork binary, dispatcher, and dispatcher configuration. Internally it depends on `mina-{network}-prefork-mesa` (supplies `/usr/lib/mina/berkeley/mina`) and `mina-{network}-postfork-mesa` (supplies `/usr/lib/mina/mesa/mina`, plus `/usr/local/bin/mina-dispatch` and `/etc/default/mina-dispatch`). Operators only need to install `mina-{network}-automode`; apt resolves the rest. +- The `minaprotocol/mina-daemon-auto-hardfork` Docker image bundles the same artifacts and sets `MINA_APP=/usr/local/bin/mina-dispatch` and `MINA_HARDFORK_STATE_DIR=/root/.mina-config` in the entrypoint. + +:::caution +The dispatcher reads its configuration from `/etc/default/mina-dispatch`. This file must exist and be owned by root. Do not modify it unless you know what you are doing. +::: + +## Manual Mode — How It Works + +Manual mode is the traditional upgrade approach, similar to the Berkeley upgrade. You are in full control of every step. + +### Manual Mode Timeline + +Manual mode upgrade flow diagram showing four phases: pre-upgrade, state finalization, manual upgrade with 5 operator steps, and post-upgrade + +### Manual Mode — What Gets Installed + +When you install the Mesa release manually, the package includes: +- The Mesa `mina` binary +- A new runtime configuration JSON for the Mesa network +- New genesis and epoch ledger tarballs + +These replace the pre-fork components. There is no dispatcher involved — you are running the Mesa binary directly. + +### Manual Mode — Updating Your Flags + +When switching from the pre-fork to the Mesa binary, you need to update your startup flags: + +1. **Remove** `--hardfork-handling` if you were using it +2. **Update** your `--genesis-ledger-dir` to point to the Mesa ledger directory (included in the package) +3. **Update** your `-config-file` to use the Mesa configuration +4. **Keep** your existing `--block-producer-key`, `--libp2p-keypair`, and other operator-specific flags + +See [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) for exact flag values. + +### Docker (Manual Mode) + +For manual mode with Docker, use the **hardfork** image (not auto-hardfork): + +``` +minaprotocol/mina-daemon-hardfork:{version}-{codename}-{network} +``` + +This image includes both pre-fork and post-fork packages but uses a dedicated hardfork entrypoint that requires manual intervention to complete the transition. + +## Which mode should I pick? + +The "who should use" criteria for each mode and the side-by-side comparison live on [Upgrade Modes](/network-upgrades/mesa/upgrade-modes#comparison) — this page intentionally stays focused on the underlying mechanism so the two documents do not drift. + +## Operator troubleshooting + +Operator-facing troubleshooting (how to tell which binary is active, dispatcher debug mode, when to use `mina-berkeley` vs `mina-mesa` directly) has moved to its own page: [Troubleshooting](/network-upgrades/mesa/troubleshooting). + --- url: /network-upgrades/mesa/archive-upgrade --- @@ -5441,7 +5798,7 @@ The commands below use `` as a placeholder for the archive build y To successfully upgrade the archive database to Mesa, you must ensure that your environment meets the foundational requirements. -## Migration host +## Upgrade host - PostgreSQL database for database server - If you use Docker, then any of the supported OS by Mina (bullseye, focal, noble, bookworm or jammy) with at least 32 GB of RAM @@ -5458,9 +5815,9 @@ you can always download it from the O1Labs Google Cloud bucket. ### Upgrade script Assuming that you have a PostgreSQL database with Mainnet archive data, in order to upgrade it to Mesa version, you need to run SQL upgrade script. -We put all efforts to make the upgrade process as smooth as possible. Script can be run on archive node which is online or offline. +The script can be run on an archive node that is either online or offline. Script can be run multiple times, it will skip steps that were already completed. It also performs sanity checks before each step to ensure that the upgrade process is successful. -Finally it creates new table (version) in the database to keep track of the upgrade process. +Finally it creates a new table (`migration_history`) in the database to keep track of the upgrade process. #### Getting the script @@ -5513,8 +5870,8 @@ Make sure to replace `` and `` with your actual PostgreSQL u #### Rollback You can rollback the upgrade process by restoring the database from a backup taken before running the upgrade script. -Another is to run rollback script which is part of the upgrade script. It will drop all tables and other database objects created by the upgrade script. -It will also update the version table to reflect the rollback. +Another alternative is to run the rollback script, which is part of the upgrade script. It will drop all tables and other database objects created by the upgrade script. +It will also update the `migration_history` table to reflect the rollback. ##### Running the rollback script @@ -5532,133 +5889,587 @@ Make sure to replace `` and `` with your actual PostgreSQL u After successfully running the upgrade script, you DO NOT need to restart your archive node or Rosetta API. Changes in upgrade script are backward compatible and will be picked up by the archive node and Rosetta API automatically. -### Verification -To verify that the upgrade was successful, you can check the version table in the PostgreSQL database. - -You can do this by running the following command: -```bash -psql -U -d -c "SELECT * FROM version;" -``` -Make sure to replace `` and `` with your actual PostgreSQL username and database name. +### Verification with the Archive Hardfork Toolbox -If the upgrade was successful, you should see the new version number in the output. +The `mina-archive-hardfork-toolbox` is a dedicated CLI tool for verifying the integrity of archive database upgrades and fork transitions. It is shipped with the Mina archive package and available as a standalone binary. -We put a lot of effort into making the upgrade process as smooth as possible. -However, if you encounter any issues or need assistance, please reach out to the Mina community on [Discord](https://discord.gg/minaprotocol) or [GitHub Discussions](https://github.com/MinaProtocol/mina/discussions). +For full details, see the [toolbox README](https://github.com/MinaProtocol/mina/blob/compatible/src/app/archive_hardfork_toolbox/README.md). -## Appendix: Database Schema Changes +All commands below require a `--postgres-uri` flag in the format: +``` +postgresql://:@:/ +``` -Below we present details of what was changed in the archive node database schema between Berkeley and Mesa versions. +#### Step 1: Pre-fork validation (before the fork) -### Zkapp_state_nullable Additional Columns +Verify the fork block candidate is valid before the network halts. -The `zkapp_state_nullable` table has been modified to include new columns `element8` through `element31` which are nullable and can store additional state information for zkApps. +**Check that the fork block is in the best chain:** +```bash +mina-archive-hardfork-toolbox fork-candidate is-in-best-chain \ + --postgres-uri \ + --fork-state-hash \ + --fork-height \ + --fork-slot +``` -```sql -, element8 int REFERENCES zkapp_field(id) -... -, element31 int REFERENCES zkapp_field(id) -); +**Verify the fork block has enough confirmations:** +```bash +mina-archive-hardfork-toolbox fork-candidate confirmations \ + --postgres-uri \ + --latest-state-hash \ + --fork-slot \ + --required-confirmations ``` -This expansion allows zkApps to store up to 31 state elements instead of the previous 8, significantly increasing the state storage capacity for complex smart contracts. +**Verify no commands were executed after the fork block** (ensures a clean fork point): +```bash +mina-archive-hardfork-toolbox fork-candidate no-commands-after \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` -### Version Table +**Find the last block with transactions** (useful for identifying the fork point): +```bash +mina-archive-hardfork-toolbox fork-candidate last-filled-block \ + --postgres-uri +``` -We also introduced a new table `version` to keep track of the database schema version. -The purpose of this table is to help with future database migrations. The table tracks which migration scripts were applied and when. -Ultimately it helps to determine the current version of the database schema and helps to avoid applying the same migration script multiple times. +#### Step 2: Verify the schema upgrade -This table is created if it does not exist already. Rollback and upgrade scripts will insert a new row with the version number and timestamp when the script was applied. +After running `upgrade-to-mesa.sql`, verify the database schema was upgraded correctly: -```sql -CREATE TABLE IF NOT EXISTS version ( - version_num INT PRIMARY KEY, - applied_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP -); +```bash +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri \ + --protocol-version \ + --migration-version ``` -The version table provides: -- **Migration tracking**: Records which migrations have been applied -- **Timestamp tracking**: Shows when each migration was executed -- **Idempotency**: Prevents duplicate migration runs -- **Version identification**: Easily identify the current database schema version +You can also verify manually by checking the `migration_history` table: +```bash +psql -U -d -c "SELECT * FROM migration_history;" +``` ---- -url: /network-upgrades/mesa/preflight-network ---- +If the upgrade was successful, the `migration_history` table will contain an entry with the expected migration version and a recent timestamp. -# Connect to Mesa Preflight Network +#### Step 3: Post-fork validation -The Mesa preflight network is a testing environment for validating the Mesa upgrade before deployment to devnet and mainnet. This network allows node operators and developers to test their infrastructure and applications with the new Mesa release. +After the fork activates and the Mesa chain is running, validate the fork block and its ancestry: -:::tip Mesa preflight hard fork completed +```bash +mina-archive-hardfork-toolbox validate-fork \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` -The Mesa preflight network hard-forked at **2026-04-27 13:00 UTC**. The post-fork chain is now running the Mesa release **`4.0.0-preflight-3f038cb`**, which bumps the **transaction protocol version to 5.0.0**. +#### Step 4: Mark the canonical chain (if needed) -The transaction protocol bump is a breaking wire-format change: any node still running a pre-fork build (including the stop-slot release `4.0.0-preflight-stop-2967b39` and the original preflight build `4.0.0-preflight1-b649c79`) will reject the new transaction format and is no longer compatible with the network. Upgrade to **`4.0.0-preflight-3f038cb`** to resume participation. +If you need to mark the chain leading to a specific block as canonical for the new protocol version: -zkApp developers must rebuild and redeploy against **`o1js@3.0.0-mesa.698ca`** — the first o1js release that targets transaction protocol v5.0.0. Transactions produced by earlier o1js versions will be rejected by post-fork nodes. +```bash +mina-archive-hardfork-toolbox convert-chain-to-canonical \ + --postgres-uri \ + --target-block-hash \ + --protocol-version .. +``` -A dedicated **Upgrade Steps** page covering the post-fork operator checklist will be linked from the Mesa Upgrade sidebar once it ships. +Optionally use `--stop-at-slot ` to limit how far back in history the marking goes. +:::tip Typical workflow +1. **Before the fork**: run `fork-candidate` commands to validate the fork block +2. **After running the upgrade script**: run `verify-upgrade` to confirm the schema upgrade +3. **After the fork activates**: run `validate-fork` to confirm data integrity +4. **For full archive verification**: run the [Archive Replayer](/node-operators/archive-node/replayer) to replay all transactions and compare the resulting ledger against the official fork config ::: -:::caution Preflight Network Notice +If you encounter any issues, reach out to the Mina community on [Discord](https://discord.gg/minaprotocol) or [GitHub Discussions](https://github.com/MinaProtocol/mina/discussions). -The preflight network is intended for testing purposes only. This network may experience instability, breaking changes, and unexpected behavior. Data on this network should not be considered persistent or reliable. +## Full Archive Verification with the Replayer -::: +The toolbox commands above verify the schema and fork block integrity, but they do not verify every transaction in the archive. For complete end-to-end verification, use the **[Archive Replayer](/node-operators/archive-node/replayer)** — a tool that replays all transactions from genesis (or a checkpoint) through the fork point, recomputing the ledger state and comparing it against the official fork config. -## Available Build Versions +The replayer is not limited to upgrade-time use. It is an **ongoing verification tool** that can be run at any time to confirm your archive database faithfully represents the canonical chain. After the Mesa upgrade, you can continue running it against Mesa blocks to detect any data corruption or missing blocks. -The Mesa preflight network uses the following build versions: +See [Archive Replayer](/node-operators/archive-node/replayer) for usage, flags, and examples. -**Current version (install this):** `4.0.0-preflight-3f038cb` — post-hard-fork release. Implements transaction protocol version **5.0.0** and is the only build compatible with the post-fork chain. +## Database Schema Changes -**Compatible o1js release:** `o1js@3.0.0-mesa.698ca` — the matching SDK for transaction protocol v5.0.0. Install with `npm install o1js@3.0.0-mesa.698ca` (or the equivalent for your package manager) when developing or redeploying zkApps against the post-fork preflight network. +For the full schema diff (new columns, modified tables, applied SQL), see [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes). -**Previous versions (historical, do not install on new nodes):** +--- +url: /network-upgrades/mesa/fork-schedule +--- -- `4.0.0-preflight-stop-2967b39` — stop-slot release used to carry nodes up to the fork. Pre-fork transaction protocol; incompatible with the post-fork chain. -- `4.0.0-preflight1-b649c79` — original preflight build. Listed for reference only. +# Mesa Fork Schedule -### Docker Images +This page collects the concrete fork-schedule values for the Mesa upgrade per network. If you are integrating with Mina (exchange, custodian, indexer, Rosetta consumer, monitoring), this is the single source of truth for **when** to act and **which release** to run. -Docker images are available for both `amd64` and `arm64` architectures, on the `bookworm` and `noble` base distributions: +The relevant moments are: -- **Mina Daemon:** `gcr.io/o1labs-192920/mina-daemon:4.0.0-preflight-3f038cb-bookworm-mesa` -- **Archive Node:** `gcr.io/o1labs-192920/mina-archive:4.0.0-preflight-3f038cb-bookworm-mesa` -- **Rosetta:** `gcr.io/o1labs-192920/mina-rosetta:4.0.0-preflight-3f038cb-bookworm-mesa` +- **`stop-transaction-slot`** (a.k.a. `slot_tx_end` in the daemon config) — the slot at which nodes stop accepting new user transactions. Block production continues with empty blocks until the network-slot. See the [glossary entry](/network-upgrades/mesa/glossary#stop-transaction-slot). +- **`stop-network-slot`** (a.k.a. `slot_chain_end`) — the slot at which block production halts entirely. The network is offline between this slot and the Mesa genesis timestamp. See the [glossary entry](/network-upgrades/mesa/glossary#stop-network-slot). +- **Mesa genesis timestamp** — the wall-clock time at which the first block on the Mesa chain is produced. Exactly 3 hours after the _stop-network-slot_. See the [glossary entry](/network-upgrades/mesa/glossary#mesa-genesis-timestamp). -For `noble`, swap `bookworm` for `noble` in the tag (e.g. `mina-daemon:4.0.0-preflight-3f038cb-noble-mesa`). +## Mainnet -### Debian Packages +:::warning Values not yet published +The mainnet Mesa fork schedule has not been announced. The values below are placeholders. They will be filled in once o1Labs publishes the stop-slot release and the Mesa genesis timestamp. +::: -Debian packages are available from the unstable repository: +| Field | Value | +|---|---| +| Pre-fork (stop-slot) release | _TBD — will be a `3.x.x` tag_ | +| Mesa release | _TBD — will be a `4.x.x` tag_ | +| `stop-transaction-slot` (`slot_tx_end`) | _TBD_ | +| `stop-network-slot` (`slot_chain_end`) | _TBD_ | +| Mesa genesis timestamp (UTC) | _TBD — exactly 3 hours after the stop-network-slot_ | +| State Finalization window | Exactly 100 slots = exactly 5 hours between `slot_tx_end` and `slot_chain_end` | -**Repository:** `unstable.apt.packages.minaprotocol.com` -**Channel:** `preflight` -**Codenames:** `bookworm`, `noble` +Subscribe to the [MinaProtocol/mina releases](https://github.com/MinaProtocol/mina/releases?q=mesa) and watch the [Mina Foundation announcements channel](https://discord.gg/minaprotocol) for the schedule publication. -Available packages (each at version `4.0.0-preflight-3f038cb`): +## Devnet -- `mina-mesa` -- `mina-archive-mesa` -- `mina-rosetta-mesa` +:::warning Values not yet published +Devnet typically forks ahead of mainnet to give integrators a final dress rehearsal. Schedule will be posted here once announced. +::: -For detailed information about the Debian repository structure and configuration, please refer to the [Debian Repository documentation](https://unstable.apt.packages.minaprotocol.com/). +| Field | Value | +|---|---| +| Pre-fork (stop-slot) release | _TBD_ | +| Mesa release | _TBD_ | +| `stop-transaction-slot` (`slot_tx_end`) | _TBD_ | +| `stop-network-slot` (`slot_chain_end`) | _TBD_ | +| Mesa genesis timestamp (UTC) | _TBD_ | -## Connecting to the Network +## Preflight -To connect your node to the Mesa preflight network, you must use the preflight network seed peer list. +The Mesa preflight network has already forked. The numbers below are historical; they will not change. -### Required Parameter +| Field | Value | +|---|---| +| Pre-fork release | `4.0.0-preflight1-b649c79` | +| Stop-slot release | `4.0.0-preflight-stop-2967b39` | +| Mesa release | `4.0.0-preflight-3f038cb` | +| Mesa hard-fork time (UTC) | 2026-04-27 13:00 | +| Transaction protocol version after fork | 5.0.0 | +| o1js version targeting protocol v5.0.0 | `o1js@3.0.0-mesa.698ca` | -Add the following parameter to your mina daemon command: +For the full preflight context, see [Preflight Network](/network-upgrades/mesa/preflight-network). -```bash ---peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt +## How to use these values + +- **Exchanges**: schedule your deposit/withdrawal freeze to begin **before** `stop-transaction-slot` and re-enable **after** the Mesa genesis timestamp confirms block production. See the [Exchanges tab on Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) for the full checklist. +- **Block producers**: install the stop-slot release any time before `stop-transaction-slot`. If you are using [automode](/network-upgrades/mesa/upgrade-modes), no further action is needed — the dispatcher will swap to the Mesa binary automatically after the Mesa release is published. +- **Archive operators**: schedule the archive schema upgrade for any time on or after `stop-network-slot`. The script is backward compatible and can also be run earlier. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). +- **Rosetta operators**: follow the manual-mode timeline — Rosetta does not support automode. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). + +--- +url: /network-upgrades/mesa/glossary +--- + +# Glossary + +This page defines the key terms used throughout the Mesa upgrade documentation. If you encounter an unfamiliar term in any of the upgrade guides, you can look it up here. + +:::tip +Bookmark this page for quick reference while reading the upgrade docs. +::: + +--- + +## Network and Fork Terms + +### Hard Fork + +A major, non-backward-compatible network upgrade. All node operators must upgrade their software before the fork activates. After the fork, nodes running the old software can no longer participate in the network. The Mesa upgrade is a hard fork. + +### Stop-Slot Release + +The Mina release (version 3.x.x) that node operators install **before** the fork. This release contains the stop-slot logic that gracefully halts the network at the designated time. In [automode](#automode), this release also bundles both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary). + +### Stop-Transaction-Slot {#stop-transaction-slot} + +A predefined global slot number baked into the [stop-slot release](#stop-slot-release). When the network reaches this slot, nodes stop accepting new user transactions. Block production continues with empty blocks (no user commands, no coinbase, no fee transfers) for exactly 100 more slots (5 hours) until the [stop-network-slot](#stop-network-slot). This is the point where exchanges must disable deposits and withdrawals. + +Referred to as `slot_tx_end` in the Mina codebase and daemon configuration. + +### Stop-Network-Slot {#stop-network-slot} + +A predefined global slot number that comes after the [stop-transaction-slot](#stop-transaction-slot). When the network reaches this slot, block production halts entirely. The network is now frozen, and the final state is used to build the Mesa release. In [automode](#automode), this is when the daemon automatically transitions to the [post-fork binary](#post-fork-binary). + +Referred to as `slot_chain_end` in the Mina codebase and daemon configuration. + +### State Finalization + +The stabilization period between the [stop-transaction-slot](#stop-transaction-slot) and the [stop-network-slot](#stop-network-slot) — 100 slots, which is exactly 5 hours. During this window, empty blocks are produced so that all nodes converge on the same final state. No operator action is required; nodes should remain running. See [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +### Mesa Genesis Timestamp + +The predefined time at which the first block is produced on the new Mesa chain — exactly 3 hours after the network reaches the [stop-network-slot](#stop-network-slot). This marks the start of the upgraded network. + +### Mesa Release + +The new Mina build (version 4.x.x) published after [state finalization](#state-finalization) is complete. It contains the Mesa chain configuration and genesis ledger derived from the final pre-fork state. In [manual mode](#manual-mode), operators install this release to join the Mesa network. + +### Preflight Network + +A testing environment for validating the Mesa upgrade before deployment to mainnet. Node operators and developers can test their infrastructure and applications against it. The preflight network may be unstable and is not intended for production use. See [Preflight Network](/network-upgrades/mesa/preflight-network). + +--- + +## Upgrade Mode Terms + +### Automode {#automode} + +The recommended upgrade path for daemon nodes (block producers, SNARK coordinators). In automode, the node handles the entire fork transition automatically — no manual intervention is required during the fork window. The [stop-slot release](#stop-slot-release) ships both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary), and the [dispatcher](#dispatcher) routes to the correct one based on the fork state. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). + +### Manual Mode {#manual-mode} + +The traditional upgrade path where operators manually stop their node after the network halts, install the [Mesa release](#mesa-release), and restart. This gives full control over every step but requires prompt action when the release is published. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). + +### Automode Image {#automode-image} + +The Docker image that bundles the [automode](#automode) runtime — both the [pre-fork](#pre-fork-binary) and [post-fork binary](#post-fork-binary), plus the [dispatcher](#dispatcher). Published as `minaprotocol/mina-daemon-auto-hardfork:{version}-{codename}-{network}`. Use this image when running automode in Docker. The companion image `minaprotocol/mina-daemon-hardfork` is the manual-mode equivalent (no dispatcher). + +### Pre-Fork Binary {#pre-fork-binary} + +The binary that runs the current (Berkeley) chain up to the [stop-network-slot](#stop-network-slot). In [automode](#automode), this is shipped as part of the `mina-{network}-automode` package and installed at `/usr/lib/mina/berkeley/mina`. In Docker, the [automode image](#automode-image) includes this binary. + +### Post-Fork Binary {#post-fork-binary} + +The binary that runs the new Mesa chain after the fork activates. In [automode](#automode), this is shipped as part of the `mina-{network}-automode` package and installed at `/usr/lib/mina/mesa/mina`. The [dispatcher](#dispatcher) switches to this binary once the [activation file](#activation-file) is created. + +### Dispatcher {#dispatcher} + +A shell script wrapper (`mina-dispatch`) installed at `/usr/local/bin/` that routes commands to the correct binary — [pre-fork](#pre-fork-binary) or [post-fork](#post-fork-binary). For `daemon`, routing is based on whether the [activation file](#activation-file) exists. For `client`, commands are always routed to the [post-fork binary](#post-fork-binary). Other subcommands (e.g., `accounts list`) are not supported — use `mina-berkeley` or `mina-mesa` directly. See [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +### Activation File {#activation-file} + +A sentinel file created by the daemon when it reaches the [stop-network-slot](#stop-network-slot) during [automode](#automode). Its presence signals to the [dispatcher](#dispatcher) that the fork has completed and the [post-fork binary](#post-fork-binary) should be used on subsequent restarts. Located at `{config-directory}/auto-fork-mesa-{network}/activated`. + +--- + +## Archive Upgrade Terms + +### Trustless Upgrade {#trustless} + +An archive database upgrade method where operators run the `upgrade_to_mesa.sql` script on their existing database. The script is backward-compatible and can be applied before the fork while the Berkeley archive node is still running. It completes in under 1 minute. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +### Trustful Upgrade {#trustful} + +An archive database upgrade method where operators import an official SQL database dump published by o1Labs after the fork. This requires no pre-fork action — operators wait for the dump, create a fresh database, and import it. The trade-off is that you trust o1Labs' export rather than verifying the data yourself. + +### Archive Hardfork Toolbox {#hardfork-toolbox} + +A dedicated CLI tool (`mina-archive-hardfork-toolbox`) for verifying the integrity of archive database upgrades and fork transitions. It provides commands to validate the fork block, verify schema upgrades, and mark the canonical chain. Shipped with the Mina archive package. See [Archive Upgrade — Verification](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox). + +### Replayer {#replayer} + +A verification tool (`mina-replayer`) that replays all transactions from a Mina archive database to reconstruct the ledger state from scratch. It can be used to verify that the archive database faithfully represents the canonical chain — both during the fork and as an ongoing integrity check. See [Archive Replayer](/node-operators/archive-node/replayer). + +### Fork Config + +The official ledger checkpoint published with the [Mesa release](#mesa-release). Operators can compare the output of the [replayer](#replayer) against this config to verify their archive matches the canonical state used to build Mesa. + +--- + +## Mesa-Specific Changes and Conventions + +Mesa bundles four Mina Improvement Proposals (MIPs) that change protocol behavior. Full descriptions live in the [Mesa overview](/network-upgrades/mesa#what-mesa-introduces); the brief definitions below link each to its canonical spec. + +### Faster Blocks + +Mesa reduces the slot time to 90 seconds and unsets the default zkApp soft limit (previously 24 commands per block). Defined in [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md). + +### Expanded zkApp State + +Mesa increases the on-chain state available to zkApps from 8 fields (indexes `0–7`) to 32 fields (indexes `0–31`) per account. This applies to both the `zkapp_states` and `zkapp_states_nullable` database tables. Defined in [MIP7](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0007-increase-state-size-limit.md). See [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) for the SQL diff. + +### Larger Events and Actions + +Mesa raises the per-transaction limit on events and actions from 100 to 1024 field elements each, and removes the previous 16-field-element cap per individual event or action. Defined in [MIP8](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0008-increase-events-actions-limit.md). + +### Larger zkApp Transactions + +Mesa roughly triples the maximum number of account updates a single zkApp transaction can contain. Defined in [MIP9](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0009-increase-zkapp-account-update-limit.md). + +### Mesa Package Naming Convention + +The [automode](#automode) Debian package is published as `mina-{network}-automode` to distinguish the dual-binary automode variant from the standard manual-mode `mina-{network}` package. This allows operators to install the automode package side-by-side with the existing daemon without one replacing the other. `mina-{network}-automode` is an umbrella package: under the hood it depends on `mina-{network}-prefork-mesa` (the [pre-fork binary](#pre-fork-binary)) and `mina-{network}-postfork-mesa` (the [post-fork binary](#post-fork-binary) plus the [dispatcher](#dispatcher)), and apt resolves both transitively when you install it. + +### Node Status Service + +A telemetry feature that reports non-sensitive node data (e.g., version, sync status) to help monitor the amount of upgraded active stake during the upgrade. Enabled by default on the Mesa daemon; pass `--simplified-node-stats` to keep the report minimal, or `--disable-node-status` to opt out entirely. See [Help Monitor the Network](/network-upgrades/mesa/upgrade-steps/post-upgrade#help-monitor-the-network) for the full flag set. + +--- +url: /network-upgrades/mesa +--- + +# Mesa Upgrade + +The Mesa upgrade is a major network upgrade (hard fork) for the Mina Protocol mainnet. It is not backward compatible: every consensus-participating Mina daemon (block producers, SNARK coordinators, archive nodes, Rosetta API nodes, and seed nodes) must run a Mesa-compatible release to remain on the network. Wallets, off-chain services, and downstream tooling do not run a daemon and only need to update if they pin to a specific transaction format or version. + +:::info New to the Mesa upgrade? +This documentation uses terms like _automode_, _stop-slot_, _trustless upgrade_, _dispatcher_, and more. If you encounter an unfamiliar term, check the **[Glossary](/network-upgrades/mesa/glossary)** for definitions. +::: + +## What Mesa Introduces + +Mesa bundles four Mina Improvement Proposals (MIPs) that change protocol behavior, plus two operational improvements to the upgrade flow itself. The canonical MIP specs live in the [MinaProtocol/MIPs](https://github.com/MinaProtocol/MIPs/tree/main/MIPS) repository; each section below links to the spec for the proposal it describes. + +### Faster Blocks — [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md) + +Mesa halves Mina's slot time from **180 seconds to 90 seconds**, doubling block production frequency. To keep the long-term token emission rate flat, the per-block **coinbase reward is also halved** (from 720 MINA to 360 MINA). The total Mina supplied per epoch is unchanged. + +Several second-order consequences flow from the slot-time change: + +- **Epoch duration halves** from ~14.9 days to ~7.4 days. Scripts and operational procedures tied to epoch boundaries (delegation cooldown, automated payouts, monitoring dashboards) trigger about twice as often. +- **Vesting schedules on active vesting accounts are automatically migrated** during the hard fork so they continue unlocking on the same real-world cadence. No operator action is required. +- **The default zkApp soft limit (24 commands per block) is unset**, allowing blocks to include more zkApp transactions when network demand is high. +- **SNARK coordinators handling maximum-cost zkApp transactions should deploy at least 4 workers** (≈4 CPU cores each) to keep up with the new 90-second slot timing. A prerequisite SNARK-worker parallelization is delivered via soft fork before Mesa. + +### Expanded zkApp State — [MIP7](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0007-increase-state-size-limit.md) + +Mesa raises the on-chain state available to a zkApp account from **8 field elements (indexes `0–7`) to 32 field elements (indexes `0–31`)**. zkApps can now store roughly four times more data directly on chain without off-chain workarounds. + +Implications: + +- **zkApp developers must redeploy** to take advantage of the new fields; older verification keys will not validate transactions touching indexes `8–31`. +- **Archive databases gain new columns** (`element8` … `element31`) on `zkapp_states` and `zkapp_states_nullable`. The Mesa archive upgrade script handles this automatically — see [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) for the full schema diff. +- **Transaction protocol version is bumped** because the on-chain serialization format changes. + +### Larger Events and Actions — [MIP8](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0008-increase-events-actions-limit.md) + +Mesa raises the per-transaction limit on events and actions from **100 field elements to 1024 field elements** for each. The previous cap of 16 field elements per individual event or action is also removed. This makes events and actions a viable channel for richer on-chain signaling and larger off-chain dispatch payloads in a single transaction. + +Mostly relevant to zkApp developers — node operators do not need to act on this change, but block producers will validate against the new limits automatically after the fork. + +### Larger zkApp Transactions — [MIP9](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0009-increase-zkapp-account-update-limit.md) + +Mesa **roughly triples** the maximum number of account updates a single zkApp transaction can contain. The previous weighted-cost formula (`10.26·np + 10.08·n2 + 9.14·n1 < 69.45`) is replaced with a simpler `np + n2 + n1 ≤ 16` rule that admits any balanced binary proof tree of height 4. + +This depends on the SNARK-worker parallelization shipped as a prerequisite for [MIP6](https://github.com/MinaProtocol/MIPs/blob/main/MIPS/mip-0006-slot-reduction-90s.md) — without it, processing maximum-size transactions inside a 90-second slot would not be feasible. + +### Automode Upgrades + +For the first time in Mina's history, block producers can upgrade through a hard fork **without manual intervention**. The automode mechanism ships both the pre-fork and post-fork binaries in a single package, with a dispatcher that automatically transitions to the new chain when the fork activates. Archive node and Rosetta API operators still use manual mode — automode is not available for those node types. See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for details. + +### Simplified Archive Upgrade + +Unlike the Berkeley upgrade (which required up to 48 hours for archive database conversion), the Mesa archive upgrade is a fast schema upgrade that completes in under a minute. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +## Upgrade Flow — End-to-End Timeline + +The Mesa upgrade follows four phases. The timeline below shows what happens at each stage, when it happens, and what **you** need to do. + +### Overview + +Mina's Mesa Upgrade — four-phase timeline from Pre-Upgrade through Post-Upgrade, with state finalization and network shutdown markers + +The upgrade moves through four phases — **Pre-Upgrade**, **State Finalization**, **Upgrade**, and **Post-Upgrade** — anchored by three key moments: + +| Milestone | When | What happens | +|---|---|---| +| **stop-transaction-slot** | Hours before the fork | Network stops accepting new transactions | +| **stop-network-slot** | Exactly 5 hours after stop-transaction-slot | Block production halts entirely | +| **Mesa genesis timestamp** | Exactly 3 hours after stop-network-slot | First Mesa block is produced | + +--- + +### Phase 1: Pre-Upgrade — weeks before the fork + +> **Goal:** Every participant is prepared and running the stop-slot release before the fork begins. + +The table below is a high-level summary. The concrete install commands (Debian, Docker, etc.) and hardware requirements live in the [Requirements](/network-upgrades/mesa/requirements) and [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) pages — start there once you know your role and deployment style. + +| Actor | What to do | +|---|---| +| **All operators** | Verify [hardware requirements](/network-upgrades/mesa/requirements) (32 GB RAM, 8-core CPU with AVX/BMI2). Back up keys. | +| **Block Producers** | Choose [upgrade mode](/network-upgrades/mesa/upgrade-modes): **automode** (recommended) or manual. Install stop-slot release [3.x.x](https://github.com/MinaProtocol/mina/releases). If automode, install the `mina-{network}-automode` package — see [Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). | +| **SNARK Coordinators** | A coordinator is a daemon node — follow the Block Producer path (automode or manual). | +| **Standalone SNARK Workers** | Workers spawned by a coordinator (`--run-snark-worker`) need no separate action — they inherit the coordinator's binary. Workers run as a separate `mina internal snark-worker` process or container must be redeployed with the Mesa release after the fork. | +| **Archive Nodes** | Install stop-slot release. Choose upgrade method: _trustless_ (run upgrade script now) or _trustful_ (import o1Labs dump later). If trustless, run the [archive upgrade script](/network-upgrades/mesa/archive-upgrade) — it is backward compatible and can be applied early. | +| **Exchanges** | Install stop-slot release. Update integrations (mina-signer, Rosetta API). Test on devnet. Plan deposit/withdrawal freeze window. | +| **zkApp Developers** | Update to the Mesa-compatible o1js version (`o1js@3.0.0-mesa.698ca` for the preflight chain), recompile your contract, and verify it on the [preflight network](/network-upgrades/mesa/preflight-network). Plan to redeploy on Mesa after the fork — every zkApp must be redeployed because the protocol version bump changes the verification key. {/* TODO(PR #1133): Link to the official o1js Mesa upgrade guide once published (cjjdespres #3220525842). */} | + +:::tip Readiness checklist +Review the [Requirements](/network-upgrades/mesa/requirements) and pick your [Upgrade Mode](/network-upgrades/mesa/upgrade-modes) before proceeding. +::: + +--- + +### Phase 2: State Finalization — hours before the fork (exactly 5 hours) + +> **Goal:** The network reaches consensus on a final state. No new transactions are accepted. + +At the predefined [**stop-transaction-slot**](/network-upgrades/mesa/fork-schedule), nodes stop accepting new user transactions. Block production continues for ~100 more slots with empty blocks (no coinbase, no fees) until the [**stop-network-slot**](/network-upgrades/mesa/fork-schedule) to ensure all nodes converge on the same state. The concrete slot numbers and timestamps live on the [Fork Schedule](/network-upgrades/mesa/fork-schedule) page. + +| Actor | What to do | +|---|---| +| **Block Producers** | **Keep your node running.** Block density during finalization is critical. Do not stop your node. | +| **SNARK Workers** | **Keep running.** Continue producing SNARK work. | +| **Archive Nodes** | Keep the archive node running to capture all finalized blocks. If doing trustless upgrade, run the upgrade script now if you haven't already. | +| **Exchanges** | **Disable MINA deposits and withdrawals.** Any transactions submitted after the stop-transaction-slot will not exist on the Mesa chain. | +| **zkApp Developers** | No action required. Monitor announcements. | + +:::danger For exchanges +Transactions submitted after the stop-transaction-slot **will not carry over** to the Mesa chain. Freeze all MINA activity before this point. +::: + +--- + +### Phase 3: Upgrade — fork day (network is down) + +> **Goal:** The network halts, state is exported, and the Mesa release is published. + +At the **stop-network-slot**, block production stops entirely. o1Labs exports the network state, builds the Mesa release with the final ledger baked in, and publishes packages and Docker images. + +| Actor | What to do | +|---|---| +| **Block Producers (automode)** | **Nothing.** Your node transitions to Mesa automatically. It will start producing blocks when the Mesa genesis timestamp arrives. | +| **Block Producers (manual)** | Stop your node. Wait for the Mesa release announcement. Install the new package. Restart with [updated flags](/network-upgrades/mesa/upgrade-steps/post-upgrade). | +| **SNARK Coordinators** | Same as block producers — automode transitions automatically, manual requires stop + install + restart. | +| **Standalone SNARK Workers** | Coordinator-spawned workers follow the coordinator automatically. Standalone workers (`mina internal snark-worker` process or container) must be stopped and redeployed with the Mesa binary; an older worker cannot submit work to a Mesa coordinator. | +| **Archive Nodes** | Install the Mesa archive node release. Point it at your upgraded database (trustless) or import the o1Labs SQL dump into a fresh database (trustful). | +| **Exchanges** | Install the Mesa release. Keep deposits/withdrawals disabled until Mesa block production begins. | + +--- + +### Phase 4: Post-Upgrade — after the fork + +> **Goal:** Block production resumes on the Mesa network. Normal operations return. + +Exactly **3 hours** after the _stop-network-slot_, at the predefined Mesa genesis timestamp, the first Mesa block is produced. + +| Actor | What to do | +|---|---| +| **Block Producers** | Verify your node is on the Mesa chain (`mina client status`). If automode, check that the `activated` file exists in your config directory (see [Troubleshooting](/network-upgrades/mesa/troubleshooting#how-do-i-know-which-binary-my-node-is-using)). Monitor block production. | +| **SNARK Workers** | Reconnect workers to the upgraded coordinator. Standalone workers must already be on the Mesa binary at this point. Verify SNARK work is being produced. | +| **Archive Nodes** | Verify the archive database is in sync. Run [validation checks](/network-upgrades/mesa/upgrade-steps/post-upgrade#in-depth-validation). Fix any missing blocks using archive tooling. | +| **Exchanges** | **Re-enable MINA deposits and withdrawals** once block production is confirmed and your systems are verified. | +| **zkApp Developers** | **Redeploy every zkApp** on Mesa — the protocol version bump invalidates Berkeley verification keys, so a redeploy is required before any zkApp can accept transactions. You can also take advantage of the expanded 32-field on-chain state at this point. | + +--- + +For step-by-step instructions per phase, see [Upgrade Steps](/network-upgrades/mesa/upgrade-steps). For end-to-end walkthroughs by role (block producer, archive node, zkApp developer, exchange), see [Examples](/network-upgrades/mesa/upgrade-steps/examples). + +## Upgrade Modes + +The Mesa upgrade supports two modes for daemon node operators: **[Automode](/network-upgrades/mesa/upgrade-modes)** (recommended — node handles the fork transition automatically) and **[Manual](/network-upgrades/mesa/upgrade-modes)** (operator stops the node, installs the Mesa release, and restarts). + +See [Upgrade Modes](/network-upgrades/mesa/upgrade-modes) for the full comparison, requirements, and the persistent-filesystem / process-restart constraints that automode imposes. For low-level details on the dispatcher and dual-binary architecture, see [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +## Examples + +End-to-end walkthroughs of the four phases for different roles (block producer automode/manual, archive node, zkApp developer, exchange) live on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. +## Quick Reference by Operator Type + +| Operator Type | Key Pages | +|---|---| +| **Block Producers** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Modes](/network-upgrades/mesa/upgrade-modes), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **SNARK Workers / Coordinators** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade Flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Archive Node Operators** | [Requirements](/network-upgrades/mesa/requirements), [Archive Upgrade](/network-upgrades/mesa/archive-upgrade), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Rosetta API Operators** | [Requirements](/network-upgrades/mesa/requirements), [Archive Upgrade](/network-upgrades/mesa/archive-upgrade), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | +| **Exchanges** | [Requirements](/network-upgrades/mesa/requirements), [Upgrade Steps](/network-upgrades/mesa/upgrade-steps), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade), [Archive Node Schema Changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) | + +## Network Details + +The values below describe the **current Mainnet (pre-fork)** chain. They will change after the Mesa fork activates — the post-fork Chain ID, Git SHA-1, and node build link will be published in the Mesa release announcement. + +``` +Chain ID (Mainnet, pre-fork) +a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1 + +Git SHA-1 (Mainnet, pre-fork) +ae112d3a96fe71b4ccccf3c54e7b7494db4898a4 + +Seed List +https://bootnodes.minaprotocol.com/networks/mainnet.txt + +Node build +https://github.com/MinaProtocol/mina/releases?q=mesa +``` + +{/* TODO(PR #1133): When the Mesa mainnet release is cut, add the post-fork Chain ID, Git SHA-1, and the specific release tag URL alongside the pre-fork values above. For the current preflight values, see [Preflight Network](/network-upgrades/mesa/preflight-network). */} + +--- +url: /network-upgrades/mesa/preflight-network +--- + +# Connect to Mesa Preflight Network + +The Mesa preflight network is a testing environment for validating the Mesa upgrade before deployment to devnet and mainnet. This network allows node operators and developers to test their infrastructure and applications with the new Mesa release. + +:::tip Mesa preflight hard fork completed + +The Mesa preflight network hard-forked at **2026-04-27 13:00 UTC**. The post-fork chain is now running the Mesa release **`4.0.0-preflight-3f038cb`**, which bumps the **transaction protocol version to 5.0.0**. + +The transaction protocol bump is a breaking wire-format change: any node still running a pre-fork build (including the stop-slot release `4.0.0-preflight-stop-2967b39` and the original preflight build `4.0.0-preflight1-b649c79`) will reject the new transaction format and is no longer compatible with the network. Upgrade to **`4.0.0-preflight-3f038cb`** to resume participation. + +zkApp developers must rebuild and redeploy against **`o1js@3.0.0-mesa.698ca`** — the first o1js release that targets transaction protocol v5.0.0. Transactions produced by earlier o1js versions will be rejected by post-fork nodes. + +A dedicated **Upgrade Steps** page covering the post-fork operator checklist will be linked from the Mesa Upgrade sidebar once it ships. + +::: + +:::caution Preflight Network Notice + +The preflight network is intended for testing purposes only. This network may experience instability, breaking changes, and unexpected behavior. Data on this network should not be considered persistent or reliable. + +::: + +## Available Build Versions + +The Mesa preflight network uses the following build versions: + +**Current version (install this):** `4.0.0-preflight-3f038cb` — post-hard-fork release. Implements transaction protocol version **5.0.0** and is the only build compatible with the post-fork chain. + +**Compatible o1js release:** `o1js@3.0.0-mesa.698ca` — the matching SDK for transaction protocol v5.0.0. Install with `npm install o1js@3.0.0-mesa.698ca` (or the equivalent for your package manager) when developing or redeploying zkApps against the post-fork preflight network. + +**Previous versions (historical, do not install on new nodes):** + +- `4.0.0-preflight-stop-2967b39` — stop-slot release used to carry nodes up to the fork. Pre-fork transaction protocol; incompatible with the post-fork chain. +- `4.0.0-preflight1-b649c79` — original preflight build. Listed for reference only. + +### Docker Images + +Docker images are available for both `amd64` and `arm64` architectures, on the `bookworm` and `noble` base distributions: + +- **Mina Daemon:** `gcr.io/o1labs-192920/mina-daemon:4.0.0-preflight-3f038cb-bookworm-mesa` +- **Archive Node:** `gcr.io/o1labs-192920/mina-archive:4.0.0-preflight-3f038cb-bookworm-mesa` +- **Rosetta:** `gcr.io/o1labs-192920/mina-rosetta:4.0.0-preflight-3f038cb-bookworm-mesa` + +For `noble`, swap `bookworm` for `noble` in the tag (e.g. `mina-daemon:4.0.0-preflight-3f038cb-noble-mesa`). + +### Debian Packages + +Debian packages are available from the unstable repository: + +**Repository:** `unstable.apt.packages.minaprotocol.com` +**Channel:** `preflight` +**Codenames:** `bookworm`, `noble` + +Available packages (each at version `4.0.0-preflight-3f038cb`): + +- `mina-mesa` +- `mina-archive-mesa` +- `mina-rosetta-mesa` + +For detailed information about the Debian repository structure and configuration, please refer to the [Debian Repository documentation](https://unstable.apt.packages.minaprotocol.com/). + +## Connecting to the Network + +To connect your node to the Mesa preflight network, you must use the preflight network seed peer list. + +### Required Parameter + +Add the following parameter to your mina daemon command: + +```bash +--peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt ``` ### Example: Running with Docker @@ -5684,7 +6495,7 @@ docker run --name mina-mesa-preflight -d \ gcr.io/o1labs-192920/mina-daemon:4.0.0-preflight-3f038cb-bookworm-mesa \ daemon \ --peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt \ - --libp2p-keypair /root/.mina-config/keys/libp2p-key + --libp2p-keypair /data/.mina-config/keys/libp2p-key ``` ### Example: Running with Debian Package @@ -5721,7 +6532,7 @@ mina daemon \ :::info Upgrading from Berkeley to Mesa If you have an existing Berkeley archive database that you want to upgrade to Mesa, please refer to the comprehensive [Archive Upgrade](archive-upgrade) guide for detailed instructions on: -- Migration prerequisites and requirements +- Upgrade prerequisites and requirements - Running the upgrade script - Rollback procedures - Database schema changes @@ -5733,153 +6544,1245 @@ If you have an existing Berkeley archive database that you want to upgrade to Me An archive node stores the full history of the blockchain and provides a GraphQL API for querying historical data. It requires a PostgreSQL database. -**Installation with Debian:** +**Installation with Debian:** + +```bash +# Step 1: Install dependencies +sudo apt-get install -y lsb-release ca-certificates wget gnupg postgresql postgresql-contrib + +# Step 2: Import the GPG key +wget -q https://unstable.apt.packages.minaprotocol.com/repo-signing-key.gpg \ + -O /etc/apt/trusted.gpg.d/minaprotocol.gpg + +# Step 3: Add the repository +echo "deb https://unstable.apt.packages.minaprotocol.com $(lsb_release -cs) preflight" | \ + sudo tee /etc/apt/sources.list.d/mina-preflight.list + +# Step 4: Install mina-archive-mesa +sudo apt-get update +sudo apt-get install -y mina-archive-mesa=4.0.0-preflight-3f038cb + +# Step 5: Create PostgreSQL database +sudo -u postgres createdb archive +sudo -u postgres createuser archive_user +sudo -u postgres psql -c "ALTER USER archive_user WITH PASSWORD 'your-secure-password';" +sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE archive TO archive_user;" + +# Step 6: Start the archive node +mina-archive run \ + --postgres-uri postgresql://archive_user:your-secure-password@localhost:5432/archive \ + --server-port 3086 +``` + +**Using Docker:** + +```bash +# Start the archive node (assumes you have a PostgreSQL server running) +docker run --name mina-archive-mesa -d \ + -p 3086:3086 \ + --restart=always \ + gcr.io/o1labs-192920/mina-archive:4.0.0-preflight-3f038cb-bookworm-mesa \ + mina-archive run \ + --postgres-uri postgresql://archive_user:your-secure-password@postgres-host:5432/archive \ + --server-port 3086 +``` + +**Connecting Mina Daemon to Archive:** + +Configure your mina daemon to send blocks to the archive node: + +```bash +mina daemon \ + --peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt \ + --archive-address localhost:3086 +``` + +### Installing and Running Rosetta + +Rosetta is a standardized API for blockchain integration. The Rosetta API requires both a mina daemon and an archive node. + +**Installation with Debian:** + +```bash +# Step 1: Install dependencies (if not already done) +sudo apt-get install -y lsb-release ca-certificates wget gnupg + +# Step 2: Import the GPG key (if not already done) +wget -q https://unstable.apt.packages.minaprotocol.com/repo-signing-key.gpg \ + -O /etc/apt/trusted.gpg.d/minaprotocol.gpg + +# Step 3: Add the repository (if not already done) +echo "deb https://unstable.apt.packages.minaprotocol.com $(lsb_release -cs) preflight" | \ + sudo tee /etc/apt/sources.list.d/mina-preflight.list + +# Step 4: Install mina-rosetta-mesa +sudo apt-get update +sudo apt-get install -y mina-rosetta-mesa=4.0.0-preflight-3f038cb + +# Step 5: Start Rosetta +# Rosetta connects to both the mina daemon and archive node +mina-rosetta \ + --port 3087 \ + --archive-uri http://localhost:3086/graphql \ + --graphql-uri http://localhost:3085/graphql +``` + +**Using Docker:** + +```bash +docker run --name mina-rosetta-mesa -d \ + -p 3087:3087 \ + --restart=always \ + gcr.io/o1labs-192920/mina-rosetta:4.0.0-preflight-3f038cb-bookworm-mesa \ + --port 3087 \ + --archive-uri http://archive-host:3086/graphql \ + --graphql-uri http://mina-daemon-host:3085/graphql +``` + +**Note:** Rosetta requires: +- A running mina daemon (GraphQL endpoint, typically port 3085) +- A running archive node (GraphQL endpoint, typically port 3086) +- Both must be fully synced for Rosetta to function properly + +## Nightly Builds + +In addition to the preflight release builds, nightly builds are also available for testing the latest changes. + +**Repository:** `nightly.apt.packages.minaprotocol.com` + +Nightly builds are available for both Debian packages and Docker images. These builds represent the latest development state and may be even more unstable than preflight releases. + +For more information about nightly builds and repository configuration, see the [Debian Repository documentation](https://nightly.apt.packages.minaprotocol.com/). + +## Verification + +After starting your node, verify connectivity to the preflight network: + +### Check Node Status + +```bash +# For Docker +docker exec -it mina-mesa-preflight mina client status + +# For Debian installation +mina client status +``` + +### Monitor Logs + +```bash +# For Docker +docker logs -f mina-mesa-preflight + +# For systemd service (Debian) +journalctl -u mina -f +``` + +## Support and Feedback + +If you encounter issues or have feedback about the Mesa preflight network: + +1. Check the [Mina Protocol Discord](https://discord.gg/minaprotocol) for community support +2. Report issues on the [Mina GitHub repository](https://github.com/MinaProtocol/mina/issues) +3. Join the Mesa upgrade discussions in the community channels + +## Next Steps + +- Explore additional Mesa upgrade documentation +- Test your applications and infrastructure against the preflight network +- Provide feedback to help improve the Mesa upgrade process + +--- +url: /network-upgrades/mesa/requirements +--- + +# Requirements + +## Hardware Requirements + +Please note the following are the hardware requirements for each node type after the upgrade: + +| Node Type | Memory | CPU | Storage | Network | +|--|--|--|--|--| +| Mina Daemon Node | 32 GB RAM | 8 core processor with BMI2 and AVX CPU instruction set are required | 16 GB | 1 Mbps Internet Connection | +| SNARK Coordinator | 32 GB RAM | 8 core processor | 16 GB | 1 Mbps Internet Connection | +| SNARK Worker (per worker) | 8 GB RAM | 6 core/12 threads with BMI2 and AVX CPU instruction set are required | 1 GB | 1 Mbps Internet Connection | +| Archive Node | 32 GB RAM | 8 core processor | 64 GB | 1 Mbps Internet Connection | +| Rosetta API standalone Docker image | 8 GB RAM | 2 core processor | 16 GB | 1 Mbps Internet Connection | +| Rosetta API + Archive Node | 32 GB RAM | 8 core processor | 64 GB | 1 Mbps Internet Connection | +| Mina Seed Node | 64 GB RAM | 8 core processor | 16 GB | 1 Mbps Internet Connection | + +## Mina Daemon Requirements + +The Mesa daemon inherits the same networking and process-management requirements as the Berkeley daemon. Rather than restate them here, refer to the canonical sections in the Validator Node docs: + +- **IP and port configuration** — `--external-ip`, `--external-port`, default port `8302` (libp2p). See [Validator Node Requirements](/node-operators/validator-node/requirements#networking). +- **Auto-restart on crash** — systemd `Restart=always`, Docker `--restart=always`. See [Running mina node as a service](/node-operators/validator-node/connecting-to-the-network#running-mina-node-as-a-service). + +The hardware table above is Mesa-specific; everything else lives once in the Validator Node docs. + +## Seed Peer Requirements + +### Generation of libp2p keypair + +To ensure connectivity across the network, it is essential that all seed nodes start with the **same** `libp2p` keypair. +This consistency allows other nodes in the network to reliably connect. +Although the same libp2p keys can be reused from before the upgrade, if you need to manually generate new libp2p keys, use the following command: + +``` +mina libp2p generate-keypair --privkey-path +``` + +Further information on [generating a libp2p key pair](/node-operators/seed-peers/generating-a-libp2p-keypair) on Mina Protocol. + +--- +url: /network-upgrades/mesa/troubleshooting +--- + +# Troubleshooting + +This page collects common operator questions and debugging tips for the Mesa upgrade. For low-level architecture and dispatcher internals, see [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +## How do I know which binary my node is using? + +Check whether the activation state file exists. The file is located at: + +`{config-directory}/auto-fork-mesa-{network_id}/activated` + +Where: +- `{config-directory}` is the path passed via `--config-directory` (defaults to `~/.mina-config` on the host, or `/root/.mina-config` in Docker) +- `{network_id}` is the network name (e.g., `mainnet` or `devnet`) + +For example, on a typical mainnet setup: + +```bash +ls ~/.mina-config/auto-fork-mesa-mainnet/activated +``` + +- **File does not exist**: your node is using the pre-fork (Berkeley) binary +- **File exists**: your node has transitioned to the Mesa binary + +## Can I run non-daemon commands or use a specific binary version? + +As described in [Dispatcher Limitations](/network-upgrades/mesa/appendix/upgrade-modes-details#dispatcher-limitations), the dispatcher supports `daemon`, `client`, and `--version`. For any other command — including `accounts list`, `ledger export`, and other CLI operations — **you must use the version-specific binary directly**: + +| Binary | Description | Full path | +|---|---|---| +| `mina-berkeley` | Pre-fork binary (current chain) | `/usr/lib/mina/berkeley/mina` | +| `mina-mesa` | Post-fork binary (Mesa chain) | `/usr/lib/mina/mesa/mina` | + +```bash +# Use mina-mesa for all non-daemon commands after the fork +mina-mesa client status +mina-mesa accounts list +mina-mesa ledger export + +# Use mina-berkeley for pre-fork queries +mina-berkeley client status +``` + +These symlinks (`mina-berkeley`, `mina-mesa`) are installed globally by the automode packages and are always available. They bypass the dispatcher entirely and run the binary directly, so they work regardless of the activation state. + +:::caution Do not manipulate the activation state file +While it is technically possible to force the dispatcher to use a specific runtime by creating or removing the `activated` file, **this is strongly discouraged**. Manually manipulating the state file while the network is live can put your node on the wrong chain. If you need to run a specific version, use `mina-berkeley` or `mina-mesa` directly instead. +::: + +## Debug mode + +Set `MINA_DISPATCHER_DEBUG=1` in your environment to see which binary and arguments the dispatcher is using: + +```bash +MINA_DISPATCHER_DEBUG=1 mina daemon ... +``` + +For a dry run (print the command without executing): + +```bash +MINA_DISPATCHER_DRYRUN=1 mina daemon ... +``` + +--- +url: /network-upgrades/mesa/upgrade-modes +--- + +# Upgrade Modes + +The Mesa upgrade supports two modes for daemon node operators: **Automode** (recommended — the daemon handles the fork transition automatically) and **Manual Mode** (the operator stops, installs the Mesa release, and restarts). Both modes reach the same end state — a node running on the Mesa network. + +**Archive node operators, Rosetta API operators, and standalone SNARK workers must use Manual mode** — Automode is not available for those node types. + +## Comparison + +| Aspect | Automode | Manual | +|---|---|---| +| Operator intervention during fork | None | Stop, install, restart | +| Downtime | Minimal (automatic transition) | Depends on operator response time | +| Applies to | Daemon nodes only (block producers, SNARK coordinators) | All node types (incl. archive, Rosetta, standalone SNARK workers) | +| Control | Automated | Full manual control | +| Risk | Lower (fewer manual steps) | Higher (depends on operator timing) | + +## Choose your upgrade mode + +Pick the tab that matches the mode you intend to run. If you are unsure, start with the Automode tab — it is the recommended path for daemon nodes. + + + + +Automode is the recommended upgrade path for the Mina daemon — the process that participates in consensus. This includes block producers and SNARK coordinators (a SNARK coordinator runs as a daemon). In automode, the node handles the entire fork transition automatically. + +### Requirements + +- Install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) **before** the _stop-transaction-slot_ +- Ensure your node remains running through the State Finalization phase +- Meet the [hardware requirements](/network-upgrades/mesa/requirements) + +### Who should use Automode + +- Block producers who want a hands-off upgrade experience +- SNARK coordinators (run as a Mina daemon) +- Operators who want to minimize downtime and operational risk + +Automode is **not** available for the following — operators of these node types must switch to the Manual Mode tab: + +- **Archive nodes** — schema upgrade is a separate manual step (see [Archive Upgrade](/network-upgrades/mesa/archive-upgrade)) +- **Rosetta API nodes** — upgrade alongside the archive node they depend on +- **Standalone SNARK workers** — `mina internal snark-worker` processes/containers run separately from any daemon; they must be stopped and redeployed with the Mesa binary after the fork. (Workers spawned by a coordinator inherit the coordinator's upgrade automatically.) + +### How automode works during the fork + +1. You install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) during the Pre-Upgrade phase. +2. Your node participates normally in block production through the State Finalization phase. +3. When the network reaches the _stop-network-slot_, the node automatically: + - Stops producing blocks on the old chain + - Transitions to the Mesa network configuration + - Begins producing blocks on the Mesa network once the genesis timestamp is reached +4. No manual intervention is required during the fork. + +:::danger Automode requires a persistent config directory and an automatic restart policy +At the fork, the daemon **exits cleanly** (exit code 0) after writing the Mesa configuration and an `activated` marker to the config directory. It does **not** restart itself. Your process manager must restart it, and the config directory must survive that restart for the dispatcher to find the `activated` marker. + +- **Persistent config directory**: `~/.mina-config` (host) or a named Docker volume / Kubernetes `PersistentVolumeClaim` mounted at `/root/.mina-config` (container). An ephemeral `emptyDir` will lose the `activated` file and break the transition. +- **Automatic restart policy**: `Restart=always` on systemd (default in the Mina unit), `--restart=always` on Docker, `restartPolicy: Always` on Kubernetes. + +Failing to meet either requirement means your node will not join the Mesa network at genesis. See [Upgrade Modes — Details](/network-upgrades/mesa/appendix/upgrade-modes-details#restart-and-filesystem-requirements) for the underlying mechanism. +::: + +### Installing automode + +Pick the install method that matches how you run your daemon today. + + + + +Install the automode package — it ships the pre-fork binary, the post-fork binary, and the dispatcher together so the daemon routes to the correct binary before and after the fork: + +```bash +sudo apt-get update +sudo apt-get install mina-{network}-automode=4.x.x +``` + +The package installs the pre-fork binary at `/usr/lib/mina/berkeley/mina`, the post-fork binary at `/usr/lib/mina/mesa/mina`, the dispatcher at `/usr/local/bin/mina-dispatch`, and the dispatcher configuration in `/etc/default/mina-dispatch`. + +`mina-{network}-automode` is an umbrella package — it depends on `mina-{network}-prefork-mesa` and `mina-{network}-postfork-mesa`, which apt pulls in automatically. You do not need to install them by hand. + +Start your node as usual — the dispatcher routes to the correct binary based on fork state. See the Debian automode example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + +Use the `mina-daemon-auto-hardfork` image (the regular `mina-daemon` image is manual-mode only): + +``` +minaprotocol/mina-daemon-auto-hardfork:{version}-{codename}-{network} +``` + +The image bundles both binaries and the dispatcher, and sets `MINA_APP=/usr/local/bin/mina-dispatch` so the entrypoint uses the dispatcher automatically. Start it with your normal flags **plus** `--restart=always` and a named volume mounted at `/root/.mina-config` — see the Docker automode example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + +For docker-compose users, see the [docker-compose quickstart](/network-upgrades/mesa/appendix/automode-docker-compose-quickstart) (under Appendix) for a ready-to-edit compose file. + + + + + + + +Manual mode gives operators full control over each step of the upgrade process. **Archive node operators and Rosetta API operators must use manual mode** — Automode is not available for those node types. + +### Requirements + +- Install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) **before** the _stop-transaction-slot_ +- Be prepared to act promptly when the Mesa release is published to minimize downtime +- Meet the [hardware requirements](/network-upgrades/mesa/requirements) + +### Who should use Manual mode + +- Operators who need full control over the upgrade process +- Operators with custom deployment pipelines that require explicit upgrade steps +- Operators who want to validate the Mesa build before restarting +- **Archive node operators** (automode is not available — schema upgrade is a separate manual step) +- **Rosetta API operators** (Rosetta upgrades alongside the archive node it depends on) + +### How manual mode works during the fork + +1. You install the stop-slot release ([3.x.x](https://github.com/MinaProtocol/mina/releases)) during the Pre-Upgrade phase. +2. Your node participates normally through the State Finalization phase. +3. When the network halts at the _stop-network-slot_, you: + - Stop your node + - Wait for the Mesa release to be published + - Install the Mesa release + - Restart your node with the [updated flags](/network-upgrades/mesa/upgrade-steps/post-upgrade) +4. Your node begins participating in the Mesa network once the genesis timestamp is reached. + +### Installing manual mode + + + + +Install only the stop-slot release before the fork: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet=3.x.x +``` + +After the fork, when the Mesa release is published, stop your node and install the Mesa package: + +```bash +sudo systemctl stop mina +sudo apt-get install mina-mainnet=4.x.x +sudo systemctl start mina +``` + +See the Debian manual example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + +Use the regular `mina-daemon` image (not `mina-daemon-auto-hardfork`): + +``` +minaprotocol/mina-daemon:-bullseye-mainnet +``` + +After the fork, stop the container and start a new one with the Mesa image: + +```bash +docker stop mina && docker rm mina +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon ... +``` + +See the Docker manual example on the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + + + + + + + +For in-depth technical details on how the upgrade mechanism works internally — dispatcher routing, dual-binary architecture, fork-state file — see [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details). + +--- +url: /network-upgrades/mesa/upgrade-steps/examples +--- + +# Examples + +The walkthroughs below follow concrete operators through the four Mesa upgrade phases (Pre-Upgrade, [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization), [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade), [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade)) for different roles and setups. Click to expand the example that matches your situation. + +
+Example: Block Producer Upgrading with Automode (Debian) + +Imagine you are **Alice**, a block producer running on a Debian server. Here is what your upgrade looks like end to end. + +**Weeks before the fork** — Alice checks [hardware requirements](/network-upgrades/mesa/requirements) and installs the automode packages: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet-automode=4.x.x +``` + +She starts her node normally. The automode dispatcher runs the pre-fork binary until the fork activates. + +> For the full preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork (State Finalization)** — The network reaches the _stop-transaction-slot_. Alice's node keeps producing blocks — she does **nothing**. Empty blocks are produced for 100 slots (exactly 5 hours) until all nodes agree on the final state. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts at the _stop-network-slot_. Alice's daemon generates the Mesa configuration, writes the `activated` marker file, and **shuts down cleanly** (exit code 0). Because Alice uses systemd with `Restart=always`, the daemon restarts automatically. On restart, the dispatcher detects the `activated` file and launches the Mesa binary. Alice does **nothing** — this all happens automatically. + +> For what happens if you chose manual mode instead, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). For details on the restart mechanism, see [Upgrade Modes - Details](/network-upgrades/mesa/appendix/upgrade-modes-details#restart-and-filesystem-requirements). + +**After the fork (Post-Upgrade)** — Exactly 3 hours after the _stop-network-slot_, the first Mesa block is produced. Alice verifies: + +```bash +# Check if the activated file exists (path depends on your config directory and network ID) +ls ~/.mina-config/auto-fork-mesa-mainnet/activated + +# Confirm Mesa chain ID +mina client status +``` + +:::note Dispatcher and non-daemon commands +The automode dispatcher only supports the `daemon` subcommand and `client status` command — it cannot determine the active runtime for other commands because they don't pass the config directory. For commands like `accounts list`, or `ledger export`, use the correct version-specific binary directly: `mina-mesa` (after the fork) or `mina-berkeley` (before the fork). The full paths `/usr/lib/mina/mesa/mina` and `/usr/lib/mina/berkeley/mina` also work. This limitation may be removed in a future release. +::: + +She's done. Her node is producing blocks on Mesa. + +> For post-upgrade flags and monitoring, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Block Producer — Manual Mode (Docker) + +**Carlos** runs a block producer using Docker and prefers manual control over the upgrade. + +**Weeks before the fork** — Carlos pulls the stop-slot Docker image and starts his node: + +```bash +docker pull minaprotocol/mina-daemon:-bullseye-mainnet + +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon \ + --block-producer-key /keys/my-wallet \ + --config-directory /root/.mina-config \ + --libp2p-keypair /keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +> For the full preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork (State Finalization)** — Carlos keeps his node running. He does **nothing** during this phase. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts at the _stop-network-slot_. Carlos waits for the Mesa release announcement, then swaps to the new image. + +> Throughout these examples, `` is a placeholder for the Mesa release tag announced for your target network. For the **preflight** network the current value is published on [Preflight Network](/network-upgrades/mesa/preflight-network); for **devnet/mainnet** the value will be published with the corresponding release announcement. + +```bash +docker stop mina && docker rm mina + +docker pull minaprotocol/mina-daemon:-bullseye-mainnet + +docker run --name mina -d \ + --restart=always \ + -v mina-config:/root/.mina-config \ + minaprotocol/mina-daemon:-bullseye-mainnet \ + daemon \ + --block-producer-key /keys/my-wallet \ + --config-directory /root/.mina-config \ + --libp2p-keypair /keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork (Post-Upgrade)** — Exactly 3 hours after the _stop-network-slot_, block production starts. Carlos verifies: + +```bash +docker exec mina mina client status # confirms Mesa chain ID +``` + +> For post-upgrade flags and monitoring, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Archive Node / Explorer Operator + +**Eve** runs an archive node, a Rosetta API instance, and a block explorer. She needs to upgrade both the daemon and the database. + +**Weeks before the fork** — Eve installs the stop-slot release and decides on her upgrade method: + +```bash +sudo apt-get update +sudo apt-get install mina-mainnet=3.x.x +``` + +She chooses the **trustless** path — running the upgrade script early, while her archive is still online: + +```bash +# Back up the database first +pg_dump -U archive_db > berkeley-archive-backup.sql + +# Download and run the upgrade script (completes in under 1 minute) +curl -O https://raw.githubusercontent.com/MinaProtocol/mina/refs/heads/mesa/src/app/archive/upgrade_to_mesa.sql +psql -U -d archive_db -f upgrade_to_mesa.sql + +# Verify +psql -U -d archive_db -c "SELECT * FROM version;" +``` + +The script is backward-compatible — her existing Berkeley archive node keeps working normally after the upgrade. + +> For the full archive upgrade guide, see [Archive Upgrade](/network-upgrades/mesa/archive-upgrade). + +**Hours before the fork (State Finalization)** — Eve keeps her archive node running to capture all finalized blocks. + +> For details on this phase, see [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization). + +**Fork day (Upgrade)** — The network halts. Eve installs the Mesa archive release and points it at her already-upgraded database: + +```bash +sudo systemctl stop mina-archive +sudo systemctl stop mina + +sudo apt-get update +sudo apt-get install mina-archive-mainnet=4.x.x mina-mainnet=4.x.x + +# Start archive process pointing to the upgraded DB +mina-archive run \ + --postgres-uri postgres://:@localhost:5432/archive_db \ + --server-port 3086 \ + --log-json --log-level DEBUG + +# Start the non-block-producing daemon connected to the archive +mina daemon \ + --archive-address localhost:3086 \ + --config-directory ~/.mina-config \ + --libp2p-keypair ~/keys/libp2p-key \ + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt \ + --file-log-rotations 500 \ + --log-json +``` + +She also restarts Rosetta: + +```bash +docker run --name rosetta --rm -d \ + -p 3088:3088 \ + --entrypoint '' \ + minaprotocol/mina-rosetta:-bullseye-mainnet \ + /usr/local/bin/mina-rosetta \ + --archive-uri "postgres://:@localhost:5432/archive_db" \ + --graphql-uri "http://localhost:3085/graphql" \ + --log-json --port 3088 +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork (Post-Upgrade)** — Block production resumes. Eve verifies data integrity: + +```bash +# Check the archive is in sync +mina client status + +# Run the verification toolbox +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri postgres://:@localhost:5432/archive_db \ + --protocol-version \ + --migration-version +``` + +She checks her explorer UI to confirm new Mesa blocks are appearing and the historical data is intact. + +> For the full validation workflow, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) and the [Archive Hardfork Toolbox](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox). + +
+ +
+Example: zkApp Developer + +**Frank** maintains a zkApp deployed on mainnet. His contract uses on-chain state and he wants to take advantage of Mesa's expanded 32-field state. + +**Weeks before the fork** — Frank updates his o1js dependency to the Mesa-compatible version and tests his zkApp on the [preflight network](/network-upgrades/mesa/preflight-network): + +```bash +npm install o1js@3.0.0-mesa.698ca +``` + +This is the first o1js release that targets the Mesa transaction protocol. `o1js@latest` currently resolves to a pre-Mesa version and will produce transactions the post-fork preflight network rejects. See [Preflight Network](/network-upgrades/mesa/preflight-network) for the matching daemon/archive/rosetta release. + +He verifies that: +- His contract recompiles cleanly against Mesa-compatible o1js and deploys on the preflight Mesa network +- Transactions execute end-to-end against the redeployed contract +- If he plans to use the expanded state fields (indexes `8–31`), his updated contract version compiles and deploys on preflight + +> For details on testing with the preflight network, see [Preflight Network](/network-upgrades/mesa/preflight-network). + +**Hours before the fork (State Finalization)** — Frank does **nothing**. His deployed zkApp keeps running on the Berkeley chain. No transactions can be submitted during this phase anyway. + +**Fork day (Upgrade)** — Frank does **nothing during the network halt**. His zkApp account and on-chain state values carry over to the Mesa chain automatically (including state fields at indexes `0–7`). However, the verification key generated for Berkeley is no longer valid under the Mesa protocol — the contract cannot process transactions on Mesa until it is redeployed. + +**After the fork (Post-Upgrade)** — Block production resumes on Mesa. Frank **must redeploy his zkApp** before it can accept transactions, regardless of whether he uses the new state fields: + +```bash +# Required for every zkApp — the Mesa protocol version bump invalidates the Berkeley verification key +zkapp deploy --network mainnet +``` + +If he is also adopting the expanded `8–31` state slots, this same deployment step ships the updated contract version that declares the new fields. + +> For post-upgrade details, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +
+Example: Exchange Upgrading to Mesa + +**Bob** is an exchange operator. His main concern is avoiding lost deposits. + +**Weeks before the fork** — Bob tests his integration (Rosetta API, mina-signer) on the [preflight network](/network-upgrades/mesa/preflight-network). He reviews [schema changes](/network-upgrades/mesa/appendix/archive-node-schema-changes) and installs the stop-slot release: + +```bash +sudo apt-get install mina-mainnet=3.x.x +``` + +> For the full exchange preparation checklist, see [Requirements](/network-upgrades/mesa/requirements). + +**Hours before the fork** — Before the _stop-transaction-slot_ arrives, Bob **disables MINA deposits and withdrawals** on his platform and notifies customers about the maintenance window. + +:::danger +Any transactions submitted after the stop-transaction-slot **will not exist on the Mesa chain**. This is the most critical step for exchanges. +::: + +**Fork day** — The network halts. Bob waits for the Mesa release announcement, then upgrades: + +```bash +sudo systemctl stop mina +sudo apt-get update +sudo apt-get install mina-mainnet=4.x.x +sudo systemctl start mina +``` + +> For detailed upgrade instructions, see [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade). + +**After the fork** — Block production resumes. Bob verifies his node is on the Mesa chain, confirms Rosetta API is working, then **re-enables MINA deposits and withdrawals**. + +```bash +mina client status # verify Mesa chain ID +# test a small internal transfer before opening to customers +``` + +> For post-upgrade flags and configurations, see [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade). + +
+ +--- +url: /network-upgrades/mesa/upgrade-steps +--- + +# Upgrade Steps + +The Mesa upgrade proceeds through four sequential phases. Each phase has specific actions for different node operator types. + +| Phase | Description | +|---|---| +| [Pre-Upgrade](/network-upgrades/mesa/requirements) | Prepare infrastructure, upgrade to the stop-slot release, run archive upgrade scripts | +| [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization) | 100-slot stabilization period — no new transactions accepted, block production continues | +| [Upgrade](/network-upgrades/mesa/upgrade-steps/upgrade) | Network halts, state is exported, Mesa build is published | +| [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) | Block production resumes on Mesa, flags and configurations for the new network | + +**Please note:** A simplified Node Status service will be part of the upgrade tooling and enabled by default in the Pre-Upgrade release with stop-slots ([3.x.x](https://github.com/MinaProtocol/mina/releases)). This feature allows for a safe upgrade by monitoring the amount of upgraded active stake. Only non-sensitive data is reported. + +Before proceeding, make sure you have reviewed the [Requirements](/network-upgrades/mesa/requirements) and chosen your [Upgrade Mode](/network-upgrades/mesa/upgrade-modes). + +## Walkthroughs by role + +The per-phase pages above describe what every role does at each step. For end-to-end narrative walkthroughs with concrete commands by role and deployment style (block producer automode/manual, archive node, zkApp developer, exchange), see the [Examples](/network-upgrades/mesa/upgrade-steps/examples) page. + +Role-specific quick references: + +- **Block Producers & SNARK Coordinators** — pick a mode on [Upgrade Modes](/network-upgrades/mesa/upgrade-modes). Install commands for automode are in [Upgrade Modes — Installing automode](/network-upgrades/mesa/upgrade-modes#installing-automode). +- **SNARK Workers** — coordinator-spawned workers inherit the coordinator's binary; standalone `mina internal snark-worker` deployments must be redeployed with the Mesa binary after the fork. +- **Archive Node Operators** — archive nodes do **not** support automode. See [Archive Upgrade](/network-upgrades/mesa/archive-upgrade) for the schema upgrade and trustless/trustful paths. +- **Exchanges** — disable MINA deposits and withdrawals before the _stop-transaction-slot_ and keep them disabled until Mesa block production resumes. See [Exchanges](/network-upgrades/mesa/upgrade-steps/post-upgrade) on the Post-Upgrade page for the re-enable checklist. + +:::danger For exchanges +Any transactions submitted after the _stop-transaction-slot_ **will not exist on the Mesa chain**. Disable deposits and withdrawals **before** [State Finalization](/network-upgrades/mesa/upgrade-steps/state-finalization) begins and keep them disabled until block production resumes on Mesa. +::: + +--- +url: /network-upgrades/mesa/upgrade-steps/post-upgrade +--- + +# Post-Upgrade + +Exactly 3 hours after the network reaches the _stop-network-slot_, at the predefined Mesa genesis timestamp, block production starts and the network is successfully upgraded. + +This page helps you **verify your node is healthy** and running on the Mesa chain. + +The checks are split into separate tabs for **Rosetta API** and **Exchange** because the two roles overlap but are not the same. _Rosetta API_ here means an operator running the `mina-rosetta` integration layer (typically alongside an archive node). _Exchange_ means an operator running the customer-facing platform that ingests Mina balances and submits payments. Most exchanges use Rosetta, so they will work through both tabs; a handful operate without Rosetta (custom GraphQL integrations) and can skip the Rosetta tab. + +## Verify Your Node + +First, confirm your node is on the Mesa chain — this check is identical for every role: + +```bash +# If using automode, check the activation file exists +ls ~/.mina-config/auto-fork-mesa-mainnet/activated + +# Check node status +mina client status +``` + +You should see: +- **Sync status**: `Synced` +- **Chain ID** matching the Mesa chain ID for your network. The chain ID changes at the fork and is network-specific (mainnet, devnet, and preflight each get their own) — the exact value is published in the Mesa release announcement alongside the first Mesa packages. +- **Git SHA-1** matching the published Mesa daemon commit +- **Genesis timestamp** matching the published Mesa genesis timestamp +- **Block height** advancing + +Then run the role-specific checks below. + + + + +#### Verify block production + +Verify block production the same way you would on Berkeley — see [Block Producer Node](/node-operators/block-producer-node) for the relevant `mina client status` fields and checks. If you are a Delegation Program participant, your uptime continues to be tracked on the Mesa chain. + +#### Check logs for errors + +```bash +# Look for any errors in recent logs +journalctl -u mina --since "1 hour ago" --no-pager | grep -i error + +# For Docker +docker logs mina --since 1h 2>&1 | grep -i error +``` + + + + +#### Verify SNARK workers are connected + +```bash +# Check that workers are producing proofs +mina client status | grep -i snark +``` + +SNARK workers are **not compatible across the fork** — the transaction SNARK changed in Mesa, so a Berkeley-era worker cannot produce valid work for a Mesa coordinator. After the fork, every worker must run a Mesa-compatible release. Workers spawned by the coordinator pick up the new binary automatically; standalone workers must be redeployed. + +To restart a standalone worker against the coordinator: + +```bash +/usr/lib/mina/mesa/mina internal snark-worker \ + --proof-level full \ + --shutdown-on-disconnect false \ + --daemon-address : +``` + +The automode dispatcher does not route `mina internal` subcommands, and `mina-mesa` is not on `PATH` in the Docker image — invoke the Mesa worker binary by its full install path (`/usr/lib/mina/mesa/mina`) as shown above. + + + + +#### 1. Verify archive data integrity + +```bash +# Check the schema-version table +psql -U -d -c "SELECT * FROM migration_history;" + +# Run the hardfork toolbox verification +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri postgres://:@localhost:5432/ \ + --protocol-version \ + --migration-version +``` + +#### 2. Check for missing blocks + +Query the archive database to see if blocks are being captured: + +```bash +psql -U -d -c "SELECT height, state_hash FROM blocks ORDER BY height DESC LIMIT 5;" +``` + +If blocks are missing, use the archive tooling to backfill. See [In-Depth Validation](#in-depth-validation) below. + + + + +#### 1. Complete the Archive Node checks above first + +Rosetta depends on a healthy archive database. + +#### 2. Verify Rosetta is responding + +```bash +curl -s http://localhost:3088/network/list \ + -H 'Content-Type: application/json' \ + -d '{"metadata":{}}' | jq . +``` + +You should see `mainnet` in the response. + +#### 3. Test a balance lookup + +```bash +curl -s http://localhost:3088/account/balance \ + -H 'Content-Type: application/json' \ + -d '{ + "network_identifier": {"blockchain":"mina","network":"mainnet"}, + "account_identifier": {"address":""} + }' | jq . +``` + + + + +#### 1. Verify your integration stack + +- Confirm Rosetta API is responding (if used) +- Confirm archive database is up to date (if used directly) +- Test a small internal MINA transfer before re-enabling customer-facing operations + +#### 2. Re-enable deposits and withdrawals + +Only re-enable MINA deposits and withdrawals after: +- Block production is confirmed (blocks are advancing) +- Your integration is verified end to end +- You have confirmed balances match expectations + + + + +## In-Depth Validation + +The checks in the tabs above cover basic health. This section provides deeper validation procedures for operators who want thorough verification. + + + + +1. **Verify signature kind** — query the GraphQL endpoint to confirm the correct signature kind: + ```graphql + query { + signatureKind + } + ``` + For mainnet, this should return `mainnet`. + +2. **Verify connectivity** — ensure your node has peers and is connected to the network. Check that the node is receiving and gossiping blocks. + + + + +Use the `mina-archive-hardfork-toolbox` to verify the upgrade. All commands require `--postgres-uri postgresql://:@:/`. See the [Archive Upgrade](/network-upgrades/mesa/archive-upgrade#verification-with-the-archive-hardfork-toolbox) page for full toolbox documentation. + +**Verify schema upgrade:** +```bash +mina-archive-hardfork-toolbox verify-upgrade \ + --postgres-uri \ + --protocol-version \ + --migration-version +``` + +You can also verify manually: +```sql +SELECT * FROM migration_history; +``` + +**Validate fork block integrity** (after the fork activates): +```bash +mina-archive-hardfork-toolbox validate-fork \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` + +**Verify no commands after fork point:** +```bash +mina-archive-hardfork-toolbox fork-candidate no-commands-after \ + --postgres-uri \ + --fork-state-hash \ + --fork-slot +``` + +**Verify extended zkApp state columns exist:** +```sql +SELECT column_name FROM information_schema.columns +WHERE table_name = 'zkapp_states_nullable' +AND column_name LIKE 'element%' +ORDER BY column_name; +``` +Confirm columns `element0` through `element31` exist. + +**Check for missing blocks:** +```bash +mina-missing-blocks-auditor --archive-uri postgres://:@
:/ +``` + +**Compare block heights** — the archive height should match or be close to the daemon's reported height: +```sql +SELECT MAX(height) FROM blocks; +``` + + + + +Use `rosetta-cli` to verify the API conforms to the Rosetta specification: + +```bash +# Spec check +rosetta-cli check:spec --configuration-file config.json + +# Data check — aim for reconciliation coverage above 60% +rosetta-cli check:data --configuration-file config.json + +# Construction check (after block production starts) +rosetta-cli check:construction --configuration-file config.json --start-block 2 +``` + + + + +1. **Verify seed connectivity** — confirm all seeds listed in `https://bootnodes.minaprotocol.com/networks/mainnet.txt` are connectable. + +2. **Verify block production** — monitor that blocks are being produced at the expected cadence via block explorers and your node's logs. + +3. **Verify empty blocks during finalization** — confirm that all blocks between the _stop-transaction-slot_ and _stop-network-slot_ were empty (no user transactions, no coinbase, no fee transfers): + ```sql + SELECT b.height, b.state_hash + FROM blocks b + WHERE b.global_slot_since_genesis > + AND b.global_slot_since_genesis <= + AND ( + EXISTS (SELECT 1 FROM blocks_user_commands buc WHERE buc.block_id = b.id) + OR EXISTS (SELECT 1 FROM blocks_internal_commands bic WHERE bic.block_id = b.id) + ); + ``` + This query should return zero rows. + +:::note +In rare cases a non-empty block can appear after the _stop-transaction-slot_ — for example if a block producer and an archive operator both ran a build without the [stop-slot release](/network-upgrades/mesa/glossary#stop-slot-release). As long as the majority of stake upgraded, such a block lands on a short fork and does not affect the hard fork block, so it is not a problem for the upgrade. +::: + + + + +--- + +## Help Monitor the Network + +The Node Status reporting service is **enabled by default** on the Mesa daemon. If you want to keep it on and point it at the o1Labs collection endpoints, pass: + +```bash +--simplified-node-stats +--node-status-url https://nodestats.minaprotocol.com/submit/stats +--node-error-url https://nodestats.minaprotocol.com/submit/stats +``` + +- `--simplified-node-stats` (no value) makes the reported payload a minimal, non-sensitive subset (version, sync status, peer count). +- `--node-status-url` is the endpoint that receives status reports. +- `--node-error-url` is the endpoint that receives crash reports the daemon emits just before terminating. + +To opt out of the Node Status service entirely, pass `--disable-node-status` (no value). Note: the previously-documented `--node-stats-type full|none` argument was never accepted by the daemon — `--simplified-node-stats` and `--disable-node-status` are the actual flag names in the Mesa release. + +## Report Issues + +If you encounter any problems after the upgrade: + +- Report bugs on [GitHub](https://github.com/MinaProtocol/mina/issues) with the label `mesa` +- Reach out on [Discord](https://discord.gg/minaprotocol) in the appropriate channel + +--- + +## Flag and Configuration Reference + +The flags below are **unchanged from Berkeley** — if your node was running correctly before the fork, the same flags will work on Mesa. This section is provided as a reference for operators setting up fresh nodes or verifying their configuration. + +For full details, see the [Mesa release notes](https://github.com/MinaProtocol/mina/releases?q=mesa). {/* TODO(PR #1133): Replace with the specific Mesa release tag URL once published (e.g. `/releases/tag/4.0.0`). */} + +:::info What changed in Mesa +- **Only if you manually manage your genesis config or genesis ledgers, repoint `--genesis-ledger-dir` and `-config-file` at the Mesa-specific files.** Operators who rely on the package-installed genesis config — including all automode users — do not need to change anything. If you do override them, both ship inside the Mesa Debian/Docker package: `--genesis-ledger-dir` should be `/var/lib/coda/mesa` (or the equivalent path your package installed), and `-config-file` should be the Mesa runtime config (typically `/etc/mina/mesa/daemon.json`). +- **All other flags carry over unchanged** — `--block-producer-key`, `--libp2p-keypair`, `--peer-list-url`, `--external-ip/port`, log flags, etc. behave the same on Mesa as on Berkeley. +::: + +
+Block Producer flags + +``` +mina daemon + --block-producer-key + --config-directory + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt + +ENVIRONMENT VARIABLES + RAYON_NUM_THREADS=6 + MINA_LIBP2P_PASS + MINA_PRIVKEY_PASS +``` + +
+ +
+SNARK Coordinator flags + +``` +mina daemon + --config-directory + --enable-peer-exchange true + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt + --run-snark-coordinator + --snark-worker-fee 0.001 + --work-selection [seq|rand|roffset] + +ENVIRONMENT VARIABLES + MINA_LIBP2P_PASS +``` + +
+ +
+SNARK Worker flags + +``` +mina internal snark-worker + --proof-level full + --shutdown-on-disconnect false + --daemon-address + +ENVIRONMENT VARIABLES + RAYON_NUM_THREADS=8 +``` -```bash -# Step 1: Install dependencies -sudo apt-get install -y lsb-release ca-certificates wget gnupg postgresql postgresql-contrib +
-# Step 2: Import the GPG key -wget -q https://unstable.apt.packages.minaprotocol.com/repo-signing-key.gpg \ - -O /etc/apt/trusted.gpg.d/minaprotocol.gpg +
+Archive Node flags -# Step 3: Add the repository -echo "deb https://unstable.apt.packages.minaprotocol.com $(lsb_release -cs) preflight" | \ - sudo tee /etc/apt/sources.list.d/mina-preflight.list +Running an Archive Node involves a non-block-producing daemon connected to the archive process and a PostgreSQL database. For more information, see [Archive Node](/node-operators/archive-node). -# Step 4: Install mina-archive-mesa -sudo apt-get update -sudo apt-get install -y mina-archive-mesa=4.0.0-preflight-3f038cb +**Daemon (non-block-producing):** -# Step 5: Create PostgreSQL database -sudo -u postgres createdb archive -sudo -u postgres createuser archive_user -sudo -u postgres psql -c "ALTER USER archive_user WITH PASSWORD 'your-secure-password';" -sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE archive TO archive_user;" +``` +mina daemon + --archive-address :3086 + --config-directory + --enable-peer-exchange true + --file-log-rotations 500 + --libp2p-keypair + --log-json + --peer-list-url https://bootnodes.minaprotocol.com/networks/mainnet.txt -# Step 6: Start the archive node -mina-archive run \ - --postgres-uri postgresql://archive_user:your-secure-password@localhost:5432/archive \ - --server-port 3086 +ENVIRONMENT VARIABLES + MINA_LIBP2P_PASS ``` -**Using Docker:** +**Archive process:** -```bash -# Start the archive node (assumes you have a PostgreSQL server running) -docker run --name mina-archive-mesa -d \ - -p 3086:3086 \ - --restart=always \ - gcr.io/o1labs-192920/mina-archive:4.0.0-preflight-3f038cb-bookworm-mesa \ - mina-archive run \ - --postgres-uri postgresql://archive_user:your-secure-password@postgres-host:5432/archive \ +``` +mina-archive run + --metrics-port + --postgres-uri postgres://:@
:/ --server-port 3086 + --log-json + --log-level DEBUG ``` -**Connecting Mina Daemon to Archive:** +
-Configure your mina daemon to send blocks to the archive node: +
+Rosetta API + +Once your Archive Node stack is running: ```bash -mina daemon \ - --peer-list-url https://storage.googleapis.com/o1labs-gitops-infrastructure/mina-mesa-network/mina-mesa-network-seeds.txt \ - --archive-address localhost:3086 +docker run \ + --name rosetta --rm \ + -p 3088:3088 \ + --entrypoint '' \ + minaprotocol/mina-rosetta:-bullseye-mainnet \ + /usr/local/bin/mina-rosetta \ + --archive-uri "${PG_CONNECTION_STRING}" \ + --graphql-uri "${GRAPHQL_URL}" \ + --log-json \ + --log-level ${LOG_LEVEL} \ + --port 3088 ``` -### Installing and Running Rosetta - -Rosetta is a standardized API for blockchain integration. The Rosetta API requires both a mina daemon and an archive node. - -**Installation with Debian:** - -```bash -# Step 1: Install dependencies (if not already done) -sudo apt-get install -y lsb-release ca-certificates wget gnupg +
-# Step 2: Import the GPG key (if not already done) -wget -q https://unstable.apt.packages.minaprotocol.com/repo-signing-key.gpg \ - -O /etc/apt/trusted.gpg.d/minaprotocol.gpg +--- +url: /network-upgrades/mesa/upgrade-steps/state-finalization +--- -# Step 3: Add the repository (if not already done) -echo "deb https://unstable.apt.packages.minaprotocol.com $(lsb_release -cs) preflight" | \ - sudo tee /etc/apt/sources.list.d/mina-preflight.list +# State Finalization -# Step 4: Install mina-rosetta-mesa -sudo apt-get update -sudo apt-get install -y mina-rosetta-mesa=4.0.0-preflight-3f038cb +Between the predefined _stop-transaction-slot_ and _stop-network-slot_, a stabilization period of 100 slots will occur. During this phase, the network consensus will not accept new blocks with transactions in them, including coinbase transactions. This means all blocks produced during this period will be completely empty — no user commands, no coinbase rewards, and no fee transfers. The state finalization period ensures all nodes reach a consensus on the latest network state before the upgrade. -# Step 5: Start Rosetta -# Rosetta connects to both the mina daemon and archive node -mina-rosetta \ - --port 3087 \ - --archive-uri http://localhost:3086/graphql \ - --graphql-uri http://localhost:3085/graphql -``` +During the state finalization slots, it is crucial to maintain a high block density. Therefore, block producers and SNARK workers shall continue running their nodes to support the network's stability and security. -**Using Docker:** +Archive nodes should also continue to execute to ensure finalized blocks are in the database and can be carried over to the upgraded schema, preserving the integrity and accessibility of the network's history. -```bash -docker run --name mina-rosetta-mesa -d \ - -p 3087:3087 \ - --restart=always \ - gcr.io/o1labs-192920/mina-rosetta:4.0.0-preflight-3f038cb-bookworm-mesa \ - --port 3087 \ - --archive-uri http://archive-host:3086/graphql \ - --graphql-uri http://mina-daemon-host:3085/graphql -``` +## Block Producers and SNARK Workers -**Note:** Rosetta requires: -- A running mina daemon (GraphQL endpoint, typically port 3085) -- A running archive node (GraphQL endpoint, typically port 3086) -- Both must be fully synced for Rosetta to function properly +1. It is crucial for the network's successful upgrade that all block producers and SNARK workers maintain their block-producing nodes up and running throughout the state finalization phase. +2. If you are running multiple daemons like is common with many operators, you can run one single node at this stage. +3. If you are a Delegation Program operator, remember that your uptime data will continue to be tracked during the state finalization phase and will be considered for the delegation grant in the following epoch. -## Nightly Builds +:::info +During this phase, both Automode and Manual mode operators simply keep their nodes running. No special action is needed regardless of your upgrade mode. +::: -In addition to the preflight release builds, nightly builds are also available for testing the latest changes. +## Archive Node Operators and Rosetta Operators -**Repository:** `nightly.apt.packages.minaprotocol.com` +**If you plan to do the _trustful_ upgrade, you can skip this step.** -Nightly builds are available for both Debian packages and Docker images. These builds represent the latest development state and may be even more unstable than preflight releases. +If you are doing the trustless upgrade, then: -For more information about nightly builds and repository configuration, see the [Debian Repository documentation](https://nightly.apt.packages.minaprotocol.com/). +1. Continue to execute the archive node to ensure finalized blocks are in the database. +2. Execute the archive node upgrade script, which is backward compatible. +3. Continue to run archive node until after the network stops at the stop-network slot. +4. For more information on the archive node upgrade process, please refer to the [Archive Upgrade](/network-upgrades/mesa/archive-upgrade) section. -## Verification +## Exchanges -After starting your node, verify connectivity to the preflight network: +Exchanges shall disable MINA deposits and withdrawals during the state finalization period (the period between _stop-transaction-slot_ and _stop-network-slot_) since any transactions after the _stop-transaction-slot_ will not be part of the upgraded chain. -### Check Node Status +Note that this assumes the majority of block producers are running the [stop-slot release](/network-upgrades/mesa/glossary#stop-slot-release), which is what enforces the transaction cutoff. If your own node is still on a pre-stop-slot build, you might technically be able to submit transactions, but the block producers running the stop-slot release will discard any blocks containing them. -```bash -# For Docker -docker exec -it mina-mesa-preflight mina client status +--- +url: /network-upgrades/mesa/upgrade-steps/upgrade +--- -# For Debian installation -mina client status -``` +# Upgrade -### Monitor Logs +Starting at the _stop-network-slot_ the network will not produce nor accept new blocks, resulting in halting the network. During the upgrade period, o1Labs will use automated tooling to export the network state based on the block at the slot just before the _stop-transaction-slot_. The exported state will then be baked into the new Mesa build, which will be used to initiate the upgraded network. It is during the upgrade window that the Mesa network infrastructure will be bootstrapped, and seed nodes will become available. o1Labs will also finalize the archive node upgrade and publish the PostgreSQL database dumps for import by the archive node operators who wish to bootstrap their archives in a trustful manner. -```bash -# For Docker -docker logs -f mina-mesa-preflight +There is a tool available to validate that the Mesa node was built from the pre-upgrade network state. See [In-Depth Validation](/network-upgrades/mesa/upgrade-steps/post-upgrade#in-depth-validation) for the `mina-archive-hardfork-toolbox` commands (`validate-fork`, `fork-candidate no-commands-after`) used to verify fork-block integrity. {/* TODO(PR #1133): If/when an authoritative `upgrading-to-mesa.md` lands in MinaProtocol/mina, link to it here. The previous link to `mina/blob/mesa/docs/upgrading-to-mesa.md` returned 404 — the `mesa` branch does not exist on the main repo. */} -# For systemd service (Debian) -journalctl -u mina -f -``` +## Block Producers and SNARK Workers -## Support and Feedback +:::info +If you are using [Automode](/network-upgrades/mesa/upgrade-modes), your node handles this phase automatically. It will transition to the Mesa network without manual intervention. Skip to [Post-Upgrade](/network-upgrades/mesa/upgrade-steps/post-upgrade) for monitoring guidance. +::: -If you encounter issues or have feedback about the Mesa preflight network: +If you are using **Manual mode**: -1. Check the [Mina Protocol Discord](https://discord.gg/minaprotocol) for community support -2. Report issues on the [Mina GitHub repository](https://github.com/MinaProtocol/mina/issues) -3. Join the Mesa upgrade discussions in the community channels +1. During the upgrade phase (between _stop-network-slot_ and the publishing of the Mesa release), block producers can shut down their nodes. +2. After the publication of the Mesa node release, block producers and SNARK workers should upgrade their nodes and be prepared for block production at the genesis timestamp, which is the slot when the first Mesa block will be produced. +3. It is possible to continue using the same libp2p key after the upgrade. Pass it to the Mesa daemon with the [`--libp2p-keypair`](/network-upgrades/mesa/upgrade-steps/post-upgrade#flag-and-configuration-reference) flag (see the Flag and Configuration Reference on the Post-Upgrade page). -## Next Steps +## Archive Node Operators and Rosetta Operators -- Explore additional Mesa upgrade documentation -- Test your applications and infrastructure against the preflight network -- Provide feedback to help improve the Mesa upgrade process +1. Upon publishing the archive node Mesa release, archive node operators and Rosetta operators should upgrade their systems. There will be both Docker images and archive node releases available to choose from. +2. Depending on the chosen upgrade method: + - _Trustless_ + - Operators should direct their Mesa archive process to the previously upgraded database. + - _Trustful_ + - Operators shall import the SQL dump file provided by o1Labs to a freshly created database. + - Operators should direct their Mesa archive process to the newly created database. --- url: /node-developers/bip44 @@ -7389,6 +9292,271 @@ This section describes how to set up and run an Archive node within the Mina pro - [Getting Started with Archive Nodes](archive-node/getting-started) - A beginner's guide to launching an Archive Node. - [Ensuring Archive Redundancy](archive-node/archive-redundancy) - Strategies to mitigate the risk of losing archived data. +--- +url: /node-operators/archive-node/replayer +--- + +# Archive Replayer + +`mina-replayer` replays all transactions from a Mina archive database, applying them sequentially to reconstruct the ledger state. It is an **ongoing verification tool**: run it at any time to confirm that your archive database faithfully represents the canonical chain. It also plays a role at hard-fork time — see [Hard Fork Replay](#hard-fork-replay) at the end of this page. + +## What the Replayer Does + +The replayer reads transactions from the archive database in order (respecting global slot and sequence number) and applies them to a starting ledger. At each step, it verifies that the computed Merkle root matches what the archive recorded. This catches data corruption, missing transactions, or schema issues in your archive. + +## Prerequisites + +- A PostgreSQL database with Mina archive data +- The `mina-replayer` binary (ships with the Mina archive Debian package and the archive/daemon Docker images) +- An input JSON file specifying the starting ledger + +### Getting the binary + +`mina-replayer` ships with the Mina archive Debian package and the archive/daemon Docker images — there is no need to build it from source: + +- **Debian:** install the archive package, which puts `mina-replayer` on your `PATH`. +- **Docker:** run it straight from the archive image, for example: + ```bash + docker run --rm minaprotocol/mina-archive:-bullseye-mainnet mina-replayer --help + ``` + +## Basic Usage + +The replayer requires two arguments: an input file and a database connection. + +```bash +mina-replayer \ + --archive-uri postgres://:@:/ \ + --input-file input.json +``` + +### Required Flags + +| Flag | Description | +|---|---| +| `--archive-uri` | PostgreSQL connection string for the archive database | +| `--input-file` | JSON file specifying the starting ledger and target state | + +### Optional Flags + +| Flag | Description | +|---|---| +| `--output-file` | Write the final ledger state to this file | +| `--continue-on-error` | Don't stop on transaction application errors | +| `--checkpoint-interval` | Create intermediate checkpoint files every N blocks | +| `--checkpoint-output-folder` | Directory for checkpoint files | +| `--checkpoint-file-prefix` | Filename prefix for checkpoint files | +| `--genesis-ledger-dir` | Directory containing the genesis ledger | +| `--log-json` | Output logs in JSON format | +| `--log-level` | Console log level (e.g., `info`, `debug`, `spam`) | +| `--log-file` | Write logs to a file | + +## Input File Format + +The input file tells the replayer where to start. For a full replay from genesis: + +```json +{ + "genesis_ledger": { + "add_genesis_winner": false, + "s3_data_hash": "", + "hash": "" + } +} +``` + +If resuming from a previous checkpoint, include `start_slot_since_genesis` and any prior epoch data. + +## Replay Modes + +### Full Replay (from genesis) + +This is the most thorough verification. It replays every transaction from genesis, catching any inconsistency in the entire archive history. + +```bash +mina-replayer \ + --archive-uri \ + --input-file genesis-input.json \ + --output-file output.json +``` + +**Use this mode when:** you want full confidence that your archive is correct. + +### Replay from Checkpoint + +If you already have a checkpoint, you can replay just the portion after it instead of re-replaying the entire history. + +```bash +mina-replayer \ + --archive-uri \ + --input-file checkpoint.json +``` + +**Use this mode when:** you want to verify recent blocks without re-replaying the full history. + +## Using Checkpoints for Long Replays + +For mainnet, a full replay from genesis can take a long time. Use checkpoints to break it into resumable segments: + +```bash +mina-replayer \ + --archive-uri \ + --input-file input.json \ + --checkpoint-interval 10000 \ + --checkpoint-output-folder ./checkpoints \ + --checkpoint-file-prefix mainnet-replay +``` + +This creates a checkpoint file every 10,000 blocks. If the replay is interrupted, restart from the latest checkpoint by using it as the `--input-file`. + +## Troubleshooting + +### Merkle root mismatch + +The replayer verifies the computed ledger Merkle root against the archive at each block. A mismatch means your archive has missing or incorrect data. Check for: + +- Missing blocks (`mina-missing-blocks-auditor`) +- Database corruption +- Incomplete schema upgrade + +### Continue on error + +If you want to see all errors rather than stopping at the first one: + +```bash +mina-replayer --continue-on-error --archive-uri --input-file input.json +``` + +--- + +## Hard Fork Replay + +:::info Extra — only relevant during a hard fork +This section applies only while a hard fork is in progress; on a normal day you do not need it. +::: + +At a hard fork, the replayer does additional work: it replays the archive up to the fork point and exports the final ledger state as the genesis ledger for the new chain. Specifically, it: + +1. Replays all transactions from genesis (or a checkpoint) up to the fork point +2. Stops at the `slot_chain_end` (the stop-network-slot where the old chain halts) +3. Exports the final ledger state as a JSON checkpoint — the genesis ledger for the new chain +4. Lets you compare that checkpoint against the official fork config to verify your archive matches the canonical state + +### Hard fork flags + +| Flag | Description | +|---|---| +| `--hard-fork-target ` | The target hard fork (e.g. `mesa`) | +| `--stop-slot-config-file` | JSON file with the fork parameters (stop slots, epoch data) | +| `--hard-fork-output-file` | Output file for the post-fork genesis ledger checkpoint | + +### Stop-slot configuration file + +The stop-slot config defines the fork parameters. This file uses the same format as the daemon runtime config: + +```json +{ + "genesis": { + "genesis_state_timestamp": "2026-02-24T19:30:00Z" + }, + "ledger": { + "add_genesis_winner": false, + "s3_data_hash": "", + "hash": "" + }, + "daemon": { + "slot_tx_end": 1900, + "slot_chain_end": 1920, + "hard_fork_genesis_slot_delta": 40 + }, + "epoch_data": { + "staking": { + "seed": "", + "s3_data_hash": "", + "hash": "" + }, + "next": { + "seed": "", + "s3_data_hash": "", + "hash": "" + } + } +} +``` + +Key fields in `daemon`: + +- **`slot_tx_end`** — the stop-transaction-slot (no more transactions accepted after this slot) +- **`slot_chain_end`** — the stop-network-slot (chain halts here, this is the fork point) +- **`hard_fork_genesis_slot_delta`** — slot offset for the new genesis relative to the fork point + +### Running the hard fork replay + +```bash +mina-replayer \ + --archive-uri postgres://:@:/ \ + --input-file input.json \ + --hard-fork-target \ + --stop-slot-config-file stop-slot-config.json \ + --hard-fork-output-file output.json \ + --log-json \ + --log-level info +``` + +The replayer will: + +1. Start from the genesis ledger (or checkpoint) specified in `input.json` +2. Replay all transactions from the archive (user commands, internal commands, and zkApp transactions) +3. Stop at the `slot_chain_end` — blocks at or beyond this slot are excluded +4. Apply the hard fork migration to produce the post-fork ledger +5. Write the result to `output.json` + +### Output format + +The output file contains the genesis configuration for the new chain: + +```json +{ + "start_slot_since_genesis": 1960, + "genesis_ledger": { + "hash": "", + "s3_data_hash": "", + "add_genesis_winner": false + } +} +``` + +The `start_slot_since_genesis` is the new genesis slot, computed from the fork point plus `hard_fork_genesis_slot_delta`. + +### Verifying against the official fork config + +After producing the replayer output, compare it to the official fork configuration published with the release: + +```bash +# Compare ledger hashes (ignoring s3_data_hash, which may differ due to non-deterministic RocksDB metadata) +diff \ + <(jq -S 'del(.genesis_ledger.s3_data_hash)' output.json) \ + <(jq -S 'del(.genesis_ledger.s3_data_hash)' official-fork-config.json) +``` + +If the diff is empty, your archive database correctly represents the canonical chain state at the fork point. + +:::tip Why ignore s3_data_hash? +The `s3_data_hash` is a SHA3-256 hash of the gzipped RocksDB ledger directory. RocksDB includes non-deterministic metadata (timestamps, sequence numbers, compaction state) and gzip headers may also differ across runs. The logical ledger contents are identical even when this hash differs — the `hash` field (the Merkle root) is the authoritative check. +::: + +### Hard fork troubleshooting + +**"No blocks found before slot_chain_end"** — the replayer could not find any blocks in the archive before the fork point. Verify: + +- Your archive database contains blocks up to the fork slot +- The `slot_chain_end` in your stop-slot config is correct + +## Further Reading + +- [Archive Node](/node-operators/archive-node) — running and maintaining an archive node +- [Replayer source code](https://github.com/MinaProtocol/mina/tree/compatible/src/app/replayer) + --- url: /node-operators/block-producer-node/docker-compose ---