Skip to content

perf(make_idempotent): introduce row lock to improve 2PC for idempotent atomic writes#2214

Merged
empiredan merged 24 commits intoapache:masterfrom
empiredan:idempotent-rowlock
Apr 14, 2025
Merged

perf(make_idempotent): introduce row lock to improve 2PC for idempotent atomic writes#2214
empiredan merged 24 commits intoapache:masterfrom
empiredan:idempotent-rowlock

Conversation

@empiredan
Copy link
Copy Markdown
Contributor

@empiredan empiredan commented Mar 19, 2025

#2197

Previously in #2198, we implemented
idempotence for each atomic write by blocking the entire mutation queue until the 2PC
pipeline was drained. However, this significantly affects performance since the pipeline
is stalled and all write requests get stuck in the mutation queue.

To address this performance issue, we introduce a row-lock mechanism: each hash key
and the highest decree currently in the 2PC phase are recorded in a hash table. For each
atomic write request, if the maximum decree associated with its hash key has not yet been
applied to the storage engine, the request is blocked in the mutation queue. Otherwise,
the hash key is considered unlocked and the request can proceed into the 2PC phase at
any time.

To avoid the performance overhead of deserialization, we directly use the partition_hash
(an unsigned 64-bit integer) from the client instead of the hash key. This also makes memory
usage more predictable, as the partition_hash has a fixed size.

Additionally, to mitigate the performance impact caused by frequent insertions and deletions
in the row-lock hash table, we introduce an LRU strategy: keys are only evicted when the hash
table exceeds a certain size threshold, and only the least recently used keys with no active
usage are removed.

Give a concrete example to illustrate how atomic write requests are handled after introducing
row locks. Suppose a client issues an incr request to a primary replica. If the primary replica
has been configured to make all atomic write requests idempotent, then:

  1. A mutation will be created as a blocking candidate to hold this atomic write request and then
    appended to the mutation queue.
  2. This mutation will be blocked and cannot get popped once the hash key contained in it is
    locked(i.e. the maximum decree associated with the hash key has not been applied to the
    storage engine).
  3. This mutation can get popped only after its hash key becomes unlocked.
  4. Popped from the mutation queue, the current base value 100 is read from the storage
    engine, and create a single put request to store the final value 101.
  5. Another mutation is then created to hold this idempotent single put request.
  6. Subsequently the new mutation enters 2PC phase, appended to plog and broadcast to
    secondary replicas.

@github-actions github-actions Bot removed the github label Mar 20, 2025
@github-actions github-actions Bot added the github label Apr 2, 2025
@empiredan empiredan marked this pull request as ready for review April 9, 2025 03:06
@empiredan empiredan merged commit cebbfae into apache:master Apr 14, 2025
82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants