You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In large Valkey deployments, it is sometimes need to extract a subset of keys based on a custom condition (for example, a hash applied to the key name), while minimizing impact on the running instance.
Using SCAN (or similar keyspace iteration) doesn't seem to be ideal in this case, as it still requires iterating over the entire keyspace and may take a long time or introduce overhead when the number of keys is very large.
Commands like BGSAVE and PSYNC are attractive because they use background processes and allow dumping data efficiently without significantly slowing down Valkey’s main operations. However, they currently dump all keys, which then requires additional post-processing to filter the keys of interest.
Question
I’m looking for a way to extract only a subset of keys that satisfy a specific condition from a snapshot taken at a point in time, ideally without scanning the entire keyspace by SCAN command in the main Valkey process.
For example, consider a scenario like:
Extract only keys where hash_function(key) == 1000
The goal is to complete this within a few seconds
Avoid putting significant CPU or latency pressure on the primary Valkey process
At the moment, the options seem to be:
Use SCAN and filter client-side → Requires SCAN commands iteration and can be slow at scale.
Use BGSAVE or PSYNC and post-process the full dump offline → Efficient for the server, but still requires dumping all keys.
My questions are:
Are there recommended methods or extensions (modules, hooks, etc.) for achieving conditional key extraction more efficiently?
If this is intentionally unsupported, is the main reason architectural (e.g., design trade-offs)?
Thank you for the reading. Any guidance or confirmation would be greatly appreciated.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Background
In large Valkey deployments, it is sometimes need to extract a subset of keys based on a custom condition (for example, a hash applied to the key name), while minimizing impact on the running instance.
Using
SCAN(or similar keyspace iteration) doesn't seem to be ideal in this case, as it still requires iterating over the entire keyspace and may take a long time or introduce overhead when the number of keys is very large.Commands like
BGSAVEandPSYNCare attractive because they use background processes and allow dumping data efficiently without significantly slowing down Valkey’s main operations. However, they currently dump all keys, which then requires additional post-processing to filter the keys of interest.Question
I’m looking for a way to extract only a subset of keys that satisfy a specific condition from a snapshot taken at a point in time, ideally without scanning the entire keyspace by
SCANcommand in the main Valkey process.For example, consider a scenario like:
At the moment, the options seem to be:
My questions are:
Thank you for the reading. Any guidance or confirmation would be greatly appreciated.
Beta Was this translation helpful? Give feedback.
All reactions