Add FieldConfig.consumingOverride for richer in-memory shape on realtime consuming segments

[xiangfu0]

Summary

Adds an opt-in per-column override (FieldConfig.consumingOverride) that lets a realtime table column have a richer in-memory shape on the mutable consuming segment than what is persisted on disk.

Typical use case: keep the table in raw encoding (smaller storage, no dictionary on disk) while giving the consuming segment a dictionary + inverted index for fast filtering during the consumption window. The committed/immutable segment uses the un-overridden table config — the override never leaks to disk.

Example

{
  \"fieldConfigList\": [{
    \"name\": \"userId\",
    \"encodingType\": \"RAW\",
    \"consumingOverride\": {
      \"encodingType\": \"DICTIONARY\",
      \"indexes\": { \"inverted\": { \"enabled\": true } }
    }
  }],
  \"tableIndexConfig\": {
    \"noDictionaryColumns\": [\"userId\"]
  }
}

Scope (intentionally narrow)

The allowlist is just encoding + indexing: encodingType and indexes (the typed JSON-tree per-index config — going forward this is the canonical API; the legacy flat indexTypes list is deprecated for new features).

Everything else is rejected at validate time with an explicit error:

Override key	Status	Why
encodingType	✅ Allowed	High-level RAW vs DICTIONARY lever
indexes	✅ Allowed	Modern typed per-index JSON config
indexTypes, indexType	❌ Rejected	Legacy flat list — superseded by indexes
compressionCodec	❌ Rejected	Only meaningful for raw on-disk forward indexes
properties, timestampConfig	❌ Rejected	Influence cross-subsystem state (TransformPipeline, aggregation, timestamp index) that does NOT consult the consuming-override view
Unknown keys / typos	❌ Rejected	Surfaces misconfiguration up front

Validation also rejects override on non-realtime tables, sorted columns, and partition columns.

Architecture

• SPI: FieldConfig.consumingOverride is a @Nullable JsonNode. Backward-compatible — older clients without this field deserialize cleanly via @JsonIgnoreProperties(ignoreUnknown=true) on BaseJsonConfig. The previous 9-arg @JsonCreator constructor is preserved as a @Deprecated overload for source/binary compatibility.

• Single dispatch point: TableConfigUtils.buildConsumingSegmentConfigBuilder is used by both RealtimeSegmentDataManager and StatelessRealtimeSegmentWriter.

  • No override (default): returns the existing IndexLoadingConfig-driven Builder unchanged — zero behavior change for tables that don't opt in.
  • With override: routes the override-applied TableConfig back through a new IndexLoadingConfig (with a defensive schema clone) so instance config, tier overlay, and schema-derived timestamp index materialization all flow through unchanged.
  • On RuntimeException (a misconfiguration that slipped past validate): logs and falls back to the persisted shape so consumption keeps making forward progress.

• Merge utility: TableConfigUtils.applyConsumingOverrides deep-copies the JSON tree before mutating to avoid aliasing into the cached TableConfig (other server threads may be reading it concurrently).

• Validation: shared enforceConsumingOverrideInvariants helper used by both validate-time and apply-time paths so the rules can't drift. Merged shape is re-validated with diagnostics that point back to the originating consumingOverride.

• Scrub policy: deny-list — any per-column string-array in tableIndexConfig (invertedIndexColumns, noDictionaryColumns, etc.) is scrubbed for overridden columns. Object-keyed maps and structurally-tied collections (sortedColumn, starTreeIndexConfigs, tierOverwrites) are intentionally left alone to avoid over-scrubbing feature configs whose keys could collide with column names.

Mixed-version safety

• Older server reading a controller-written consumingOverride it doesn't understand: silently drops the unknown field (Jackson default) and behaves as if no override exists. The override only takes effect once the server is upgraded.
• Older controller writing a TableConfig: cannot produce the new field, so newer servers see no override. Default behavior preserved.

Plugin teams: the SPI change is additive — please re-test against the new pinot-spi.jar if you build FieldConfig objects directly via positional constructor calls (the previous 9-arg signature is preserved as @Deprecated).

Test plan

• 23 unit tests in TableConfigConsumingOverrideTest:
  • merge semantics (encoding replacement, scrub of legacy per-column lists)
  • input-mutation guard (cached TableConfig byte-for-byte unchanged after merge)
  • idempotence
  • validation rules (REALTIME-only, sorted-column rejection, partition-column rejection, allowlist enforcement at both validate-time and apply-time)
  • dispatch helper (no-override → IndexLoadingConfig path, valid override → merged path, fallback on failure)
  • JSON round-trip preservation
  • legacy FieldConfig JSON without consumingOverride still deserializes
• End-to-end converter test (RealtimeSegmentConverterTest.testConsumingOverrideKeepsRawShapeOnImmutableSegment):
  • asserts the consuming MutableSegmentImpl has dictionary + inverted index in memory
  • asserts the committed immutable segment retains the persisted RAW shape with no inverted index
• Integration test (ConsumingOverrideRealtimeTest) — full Kafka → consuming → forceCommit → immutable lifecycle:
  • Pre-commit: walks each server's TableDataManager, every consuming segment is MutableSegmentImpl, STRING_COLUMN has both dictionary and inverted index in memory, control column keeps default DICTIONARY shape.
  • Force-commit + wait: triggers forceCommit via admin REST and waits for ExternalView ONLINE.
  • Post-commit: every ImmutableSegmentImpl on every server has STRING_COLUMN with null dictionary AND null inverted index in memory, AND hasDictionary=false on the on-disk ColumnMetadata. Cross-checks both views so a loader can't silently re-create the index. Control column still dictionary-encoded.
  • Query parity: same COUNT(*) filter returns identical results before/after commit.
• No regression in TableConfigUtilsTest (56), IndexLoadingConfigTest (3), BaseTableDataManagerTest (25), RealtimeSegmentConverterTest (161 total).
• Spotless / Checkstyle / License all pass.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 72.48677% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.67%. Comparing base (80fb0e8) to head (393bc62).
⚠️ Report is 27 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/pinot/segment/local/utils/TableConfigUtils.java	71.76%	25 Missing and 23 partials ⚠️
...ealtime/writer/StatelessRealtimeSegmentWriter.java	0.00%	3 Missing ⚠️
...a/manager/realtime/RealtimeSegmentDataManager.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18406      +/-   ##
============================================
+ Coverage     63.46%   63.67%   +0.20%     
- Complexity     1701     1735      +34     
============================================
  Files          3254     3254              
  Lines        199104   199627     +523     
  Branches      30830    31017     +187     
============================================
+ Hits         126354   127105     +751     
+ Misses        62665    62362     -303     
- Partials      10085    10160      +75     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	63.67% <72.48%> (+0.20%)	⬆️
temurin	63.67% <72.48%> (+0.20%)	⬆️
unittests	63.66% <72.48%> (+0.20%)	⬆️
unittests1	55.65% <13.75%> (+0.23%)	⬆️
unittests2	35.00% <72.48%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds an opt-in FieldConfig.consumingOverride for realtime tables so mutable consuming segments can use a different in-memory encoding/index shape than the persisted immutable segment shape. It fits into Pinot’s realtime ingestion path by letting servers build richer consuming-segment indexes without changing what gets committed to disk.

Changes:

• Adds FieldConfig.consumingOverride support in SPI/config classes and exposes a getter/builder path for the new JSON field.
• Introduces consuming-override merge/validation logic in TableConfigUtils and routes realtime mutable-segment construction through that logic from both consuming writers.
• Adds unit, converter, and integration coverage for consuming-segment-only dictionary/inverted-index overrides.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java	Adds a JSON key constant used by consuming-override merge logic.
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/FieldConfig.java	Extends FieldConfig with the new consumingOverride field, constructor arg, getter, and builder support.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java	Implements override detection, validation, merge/scrub behavior, and the new consuming-segment builder dispatch.
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java	Switches realtime consuming-segment creation to the new override-aware builder helper.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/writer/StatelessRealtimeSegmentWriter.java	Applies the same override-aware builder path for stateless realtime segment writing.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigConsumingOverrideTest.java	Adds focused unit tests for merge behavior, validation rules, serde, and fallback behavior.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverterTest.java	Verifies the override affects the mutable consuming segment but not the committed immutable segment.
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/ConsumingOverrideRealtimeTest.java	Adds end-to-end realtime coverage across consume, force-commit, immutable load, and query parity.

[xiangfu0]

Found a couple of high-signal config-safety issues; see inline comments.