Merge branch 'unstable' into fix/rdb-skip-expired-fields

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>

Author: hanxizh9910

.github/benchmark_configs/benchmark-config-arm.json

@@ -0,0 +1,20 @@
+[
+  {
+    "duration": 180,
+    "keyspacelen": [3000000],
+    "data_sizes": [16,96],
+    "pipelines": [1, 10],
+    "clients": [1600],
+    "commands": [
+      "SET",
+      "GET"
+    ],
+    "cluster_mode": false,
+    "tls_mode": false,
+    "warmup": 30,
+    "io-threads": [1,9],
+    "benchmark-threads": 90,
+    "server_cpu_range": "0-8",
+    "client_cpu_range": "96-191"
+  }
+]

.github/benchmark_configs/benchmark-config-x86.json

@@ -0,0 +1,20 @@
+[
+  {
+    "duration": 180,
+    "keyspacelen": [3000000],
+    "data_sizes": [16,96],
+    "pipelines": [1, 10],
+    "clients": [1600],
+    "commands": [
+      "SET",
+      "GET"
+    ],
+    "cluster_mode": false,
+    "tls_mode": false,
+    "warmup": 30,
+    "io-threads": [1,9],
+    "benchmark-threads": 90,
+    "server_cpu_range": "0-8",
+    "client_cpu_range": "144-191,48-95"
+  }
+]

.github/benchmark_configs/pr_benchmark.json

@@ -0,0 +1,30 @@
+# Valkey Project Instructions
+
+You are an expert code reviewer for the Valkey project. Provide helpful, constructive feedback on code quality, safety, and adherence to project standards.
+
+## 1. Review Tone & Focus
+- **Tone:** Be professional, direct, constructive, and empathetic.
+- **Focus:** Critique the *code*, never the *person*.
+- **Constructive:** Suggest improvements, explain *why*, provide examples.
+
+## 2. Critical Checks
+- **DCO:** **Flag missing** `Signed-off-by: Name <email>` in commits. Every commit needs it.
+- **Security:** If PR fixes a security vulnerability, flag it: "Security fixes should be reported privately to security@lists.valkey.io, not via public PRs."
+
+## 3. Major Decision Detection
+Flag PRs that appear to be "Technical Major Decisions" requiring TSC consensus:
+- Fundamental changes to core datastructures
+- New data structures or APIs
+- Backward compatibility breaks
+- New user-visible fields requiring long-term maintenance
+- New external libraries affecting runtime behavior
+
+**Action:** Comment mentioning **@core-team** that this appears to require TSC review and ask if consensus was reached in a linked Issue.
+
+## 4. Documentation Reminder
+If PR changes user-facing behavior (new commands, changed semantics, new config):
+- **Remind** author that docs at [valkey-doc](https://github.com/valkey-io/valkey-doc) may need updating.
+- **Suggest** linking PR to related Issue with "Fixes #xyz" pattern if applicable.
+
+## 5. Governance Changes
+**ANY change to `GOVERNANCE.md`** requires special attention - comment mentioning **@core-team** for review.

.github/copilot-instructions.md

@@ -0,0 +1,48 @@
+---
+applyTo:
+  - "src/**/*.{c,h}"
+  - "valkey.conf"
+---
+
+# Valkey Core Engine Review Standards
+
+Apply these standards to core engine C code. Do NOT apply to `deps/` (vendored dependencies).
+
+## 1. Code Style (from DEVELOPMENT_GUIDE.md)
+- **Formatting:** Follow clang-format (4-space indent, no tabs, braces attached).
+- **Comments:**
+  - C-style `/* ... */` for single or multi-line.
+  - C++ `//` only for single-line.
+  - Multi-line: align leading `*`, final `*/` on last text line.
+  - Document *why* code exists, not just *what*. Document all functions.
+- **Line Length:** Keep below 90 characters when reasonable.
+- **Types:** Use the boolean type for true/false values.
+- **Static:** Use `static` for file-local functions.
+
+## 2. Naming Conventions
+- **Variables:** `snake_case` or lowercase (e.g. `cached_reply`, `keylen`).
+- **Functions:** `camelCase` or `namespace_camelCase` (e.g. `createStringObject`, `IOJobQueue_isFull`).
+- **Macros:** `UPPER_CASE` (e.g. `MAKE_CMD`).
+- **Structures:** `camelCase` (e.g. `user`).
+
+## 3. Safety & Correctness
+- **Memory:** Strict check for buffer overflows and leaks.
+- **Strings:** Validate `sds` string handling.
+- **Concurrency:** Verify thread safety in threaded I/O paths.
+
+## 4. Design Guidelines
+- **Configuration:** Avoid new configs if heuristics suffice. Only add for explicit trade-offs (CPU vs memory).
+- **Metrics:** No new metrics on hot paths without zero-overhead proof.
+- **PR Scope:** Separate refactoring from functional changes for easier backporting.
+
+## 5. Testing & Documentation
+- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.c` naming.
+- **Integration Tests:** Required for commands in `tests/`.
+- **Command Changes:** New/modified commands need corresponding updates in `src/commands/*.json`.
+- **New C Files:** Remind to update `CMakeLists.txt` when adding new `.c` source files.
+- **License:** New files need BSD-3-Clause header. Material changes (>100 lines) also require it.
+- **Documentation:** User-facing changes need docs at [valkey-doc](https://github.com/valkey-io/valkey-doc).
+
+## 6. Critical Escalation
+- **Trigger:** Changes to `cluster*.c`, `replication.c`, `rdb.c`, `aof.c`.
+- **Action:** Comment mentioning **@core-team** to request architectural review.

.github/instructions/core-engine.instructions.md

@@ -0,0 +1,23 @@
+---
+applyTo:
+  - "tests/**/*.tcl"
+---
+
+# Valkey Integration Test Review Standards
+
+Apply these standards to Tcl-based integration tests (from DEVELOPMENT_GUIDE.md).
+
+## 1. Test Organization
+- **Cluster:** Use `tests/unit/cluster/` (NOT legacy `tests/cluster/` which is deprecated).
+- **Coverage:** All contributions should include tests. New commands require integration tests.
+- **Naming:** Use descriptive test names that explain what is being tested.
+
+## 2. Test Quality
+- **Isolation:** Tests should not depend on execution order.
+- **Cleanup:** Ensure proper cleanup of resources and temporary files.
+- **Assertions:** Use clear assertions with meaningful error messages.
+
+## 3. Best Practices
+- **Readability:** Keep tests simple and focused on one scenario.
+- **Reliability:** Avoid timing-dependent tests; use proper synchronization.
+- **Documentation:** Comment complex test scenarios to explain intent.

.github/instructions/integration-tests.instructions.md

@@ -0,0 +1,24 @@
+---
+applyTo:
+  - "utils/**/*"
+---
+
+# Valkey Utilities Review Standards
+
+Apply these standards to utility scripts and tools.
+
+## 1. Script Quality
+- **Portability:** Scripts should work across common platforms (Linux, macOS).
+- **Error Handling:** Check for errors and provide clear error messages.
+- **Documentation:** Include usage instructions in comments or help text.
+
+## 2. Code Standards
+- **Python:** Follow PEP 8 style guidelines.
+- **Ruby:** Follow standard Ruby conventions.
+- **Shell:** Use shellcheck-compatible patterns.
+- **C Tools:** Follow LLVM style (4-space indent, no tabs).
+
+## 3. Best Practices
+- **Dependencies:** Minimize external dependencies.
+- **Safety:** Validate inputs and avoid destructive operations without confirmation.
+- **Maintainability:** Keep utilities simple and well-commented.

.github/instructions/utils.instructions.md

@@ -1,11 +1,11 @@
-name: On-demand Benchmark
+name: On-Demand Labeled Benchmark
 
 on:
   pull_request_target:
     types: [labeled]
 
 concurrency:
-  group: valkey-benchmark
+  group: ec2-al-2023-pr-benchmarking-arm64
   cancel-in-progress: false
 
 defaults:
@@ -19,9 +19,12 @@ permissions:
 
 jobs:
   benchmark:
-    runs-on: ${{ github.repository == 'valkey-io/valkey' && fromJSON('["self-hosted","ec2-ubuntu-24.04-benchmarking"]') || 'ubuntu-latest' }}
     if: |
-      github.event.action == 'labeled' && github.event.label.name == 'run-benchmark'
+      github.event.action == 'labeled' && github.event.label.name == 'run-benchmark' &&
+      github.repository == 'valkey-io/valkey'
+
+    runs-on: ["self-hosted", "ec2-al-2023-pr-benchmarking-arm64"]
+    timeout-minutes: 7200
     steps:
       - name: Checkout valkey
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -39,36 +42,65 @@ jobs:
           fetch-depth: 1
           persist-credentials: false
 
+      - name: Checkout valkey for latest benchmark.
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+        with:
+          repository: valkey-io/valkey
+          ref: "unstable"
+          path: valkey_latest
+          fetch-depth: 0
+          persist-credentials: false
+
       - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        uses: kishaningithub/setup-python-amazon-linux@63dc7bd642c6a564d0a93dd4b1cbe3e448ce3621 # v1.0.18
         with:
           python-version: "3.10"
           cache: "pip"
 
       - name: Install dependencies
         working-directory: valkey-perf-benchmark
         run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential
+          sudo dnf groupinstall "Development Tools" -y
+          sudo dnf install -y gcc gcc-c++ make \
+              python3-devel \
+              openssl-devel \
+              bzip2-devel \
+              libffi-devel
           pip install -r requirements.txt
 
+      - name: Build latest valkey_latest
+        working-directory: valkey_latest
+        run: |
+          echo "Building latest valkey-benchmark for latest benchmark executable..."
+          make distclean || true
+          make -j
+          if [[ -f "src/valkey-benchmark" ]]; then
+            echo "Successfully built latest valkey-benchmark"
+            ls -la src/valkey-benchmark
+            ./src/valkey-benchmark --version || echo "Version check completed"
+          else
+            echo "Failed to build valkey-benchmark"
+            exit 1
+          fi
+          VALKEY_BENCHMARK_PATH="$(pwd)/src/valkey-benchmark"
+          echo "VALKEY_BENCHMARK_PATH=$VALKEY_BENCHMARK_PATH" >> $GITHUB_ENV
+          echo "Latest valkey-benchmark path: $VALKEY_BENCHMARK_PATH"
+
       - name: Run benchmarks
         working-directory: valkey-perf-benchmark
         run: |
-          CONFIG_FILE="../valkey/.github/benchmark_configs/pr_benchmark.json"
-
-          # Removes server/client CPU range args if running in private repo
-          if [[ "${{ github.repository }}" != "valkey-io/valkey" ]]; then
-            jq 'map(del(.server_cpu_range, .client_cpu_range))' "$CONFIG_FILE" > tmp.json && mv tmp.json "$CONFIG_FILE"
-          fi
+          CONFIG_FILE="../valkey/.github/benchmark_configs/benchmark-config-arm.json"
 
           # Base benchmark arguments
           BENCHMARK_ARGS=(
+            --config "$CONFIG_FILE"
             --commits "${{ github.event.pull_request.merge_commit_sha }}"
             --baseline "${{ github.event.pull_request.base.ref }}"
-            --results-dir "results"
+            --valkey-benchmark-path "$VALKEY_BENCHMARK_PATH"
+            --target-ip ${{ secrets.EC2_ARM64_IP }}
             --valkey-path "../valkey"
-            --config "$CONFIG_FILE"
+            --results-dir "results"
+            --runs 5
           )
 
           # Run benchmark
@@ -78,9 +110,10 @@ jobs:
         working-directory: valkey-perf-benchmark
         run: |
           python ./utils/compare_benchmark_results.py \
-            ./results/${{ github.event.pull_request.base.ref }}/metrics.json \
-            ./results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json \
-            ../comparison.md
+            --baseline ./results/${{ github.event.pull_request.base.ref }}/metrics.json \
+            --new ./results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json \
+            --output ../comparison.md \
+            --metrics rps
 
       - name: Upload artifacts
         if: always()
@@ -115,10 +148,11 @@ jobs:
         if: always()
         continue-on-error: true
         run: |
+          rm -rf comparison.md valkey*
           pkill -f valkey
           exit_code=$?
           if [ $exit_code -eq 0 ]; then
-            echo "✓ Killed running valkey processes"
+            echo "Killed running valkey processes"
           elif [ $exit_code -eq 1 ]; then
             echo "No valkey processes found to kill"
           else