Add RuleSetCustomizer SPI for multi-stage planner Calcite rules#18387
Add RuleSetCustomizer SPI for multi-stage planner Calcite rules#18387gortiz wants to merge 1 commit intoapache:masterfrom
Conversation
Introduces a ServiceLoader-discovered SPI that lets plugins add, remove, reorder, or replace the Calcite HEP rules used by the multi-stage query engine. The OSS defaults are themselves contributed by DefaultRuleSetCustomizer (registered via META-INF/services), and plugin customizers run on top. - New `Phase` enum covers every HEP phase QueryEnvironment runs: BASIC, FILTER_PUSHDOWN, PROJECT_PUSHDOWN, PRUNE, POST_LOGICAL, POST_LOGICAL_V2, POST_LOGICAL_ENRICHED_JOIN. Append-only contract. - New `RuleSetCustomizer` interface — plugins implement `customize(Phase, List<RelOptRule>)` and modify the per-phase list in place. - New `PinotRuleSet` owns the per-phase, immutable rule lists and is the single source of truth read by `QueryEnvironment`. The default instance is built lazily from `ServiceLoader` discovery. - `DefaultRuleSetCustomizer` replaces the static `PinotQueryRuleSets` lists; the deleted class is the rename source. - `QueryEnvironment` reads every phase's rules through `PinotRuleSet` via a new `Config#getRuleSet()` `@Value.Default`. The per-query `sortExchangeCopyLimit` override stays inside `getTraitProgram` (per-query copy of POST_LOGICAL with all `PinotSortExchangeCopyRule` instances replaced by the override). Per-query `usePlannerRules` / `skipPlannerRules` filtering is unchanged (still applied on top of the rule lists in `getOptProgram`).
gortiz
added
extension-point
xiangfu0
left a comment
xiangfu0
left a comment
There was a problem hiding this comment.
Found a few high-signal SPI/plugin issues; see inline comments.
| import org.apache.calcite.plan.RelOptRule; | ||
|
|
||
|
|
||
| /// Plugin SPI for customizing the multi-stage planner's Calcite rule sets. |
There was a problem hiding this comment.
This is documented as a plugin SPI, but Pinot plugin realms only import org.apache.pinot.spi by default. A standard plugin packaged with pinot-plugin.properties cannot even link org.apache.pinot.query.planner.rules.RuleSetCustomizer without extra realm-import wiring, so the advertised extension point is broken for normal plugins. Please move this SPI into pinot-spi or import this package by default before landing.
| /// Discovers every [RuleSetCustomizer] via [ServiceLoader] and builds a | ||
| /// rule set from them. Used by [#defaultInstance()] and by callers that | ||
| /// don't have an externally-managed customizer list. | ||
| public static PinotRuleSet loadFromServiceLoader() { |
There was a problem hiding this comment.
Even if a plugin adds the extra class-realm import, this discovery path only does ServiceLoader.load(RuleSetCustomizer.class). Broker startup does not enumerate Pinot plugin classloaders here, and PluginClassLoader keeps plugin jars isolated, so third-party customizers will never be discovered and only DefaultRuleSetCustomizer will run. Please load providers from PluginManager/plugin realms explicitly instead of relying on the default ServiceLoader lookup.
| /// applied later by `QueryEnvironment.getTraitProgram`, which swaps the | ||
| /// configured `PinotSortExchangeCopyRule` on a per-query copy of the | ||
| /// `POST_LOGICAL` list. | ||
| public final class DefaultRuleSetCustomizer implements RuleSetCustomizer { |
There was a problem hiding this comment.
This rename removes the existing public org.apache.pinot.calcite.rel.rules.PinotQueryRuleSets type entirely. Any out-of-tree planner extension or test compiled against current releases will fail to load or compile after upgrade. Please keep a deprecated bridge in the old package instead of replacing the class outright.
2 participants
Introduces a ServiceLoader-discovered SPI that lets plugins add, remove, reorder, or replace the Calcite HEP rules used by the multi-stage query engine. The OSS defaults are themselves contributed by DefaultRuleSetCustomizer (registered via META-INF/services), and plugin customizers run on top.
Phaseenum covers every HEP phase QueryEnvironment runs: BASIC, FILTER_PUSHDOWN, PROJECT_PUSHDOWN, PRUNE, POST_LOGICAL, POST_LOGICAL_V2, POST_LOGICAL_ENRICHED_JOIN. Append-only contract.RuleSetCustomizerinterface — plugins implementcustomize(Phase, List<RelOptRule>)and modify the per-phase list in place.PinotRuleSetowns the per-phase, immutable rule lists and is the single source of truth read byQueryEnvironment. The default instance is built lazily fromServiceLoaderdiscovery.DefaultRuleSetCustomizerreplaces the staticPinotQueryRuleSetslists; the deleted class is the rename source.QueryEnvironmentreads every phase's rules throughPinotRuleSetvia a newConfig#getRuleSet()@Value.Default. The per-querysortExchangeCopyLimitoverride stays insidegetTraitProgram(per-query copy of POST_LOGICAL with allPinotSortExchangeCopyRuleinstances replaced by the override).Per-query
usePlannerRules/skipPlannerRulesfiltering is unchanged (still applied on top of the rule lists ingetOptProgram).