Auto-create dictionary when secondary indexes need it (inverted, FST, range)

[xiangfu0]

Summary

When a column is configured with encodingType: RAW (or appears in noDictionaryColumns) but a secondary index that needs a dictionary is also configured (inverted, fst, ifst, dict-id range), Pinot now auto-creates a standalone dictionary for that column so the secondary index can function. The forward index stays RAW; the dictionary lives alongside it.

Previously, such configs either silently produced wrong results, fell back to a deprecated raw-value bitmap inverted index, or required the user to manually add a "dictionary": {} block to the FieldConfig.indexes map. This PR makes the common case work without explicit configuration and removes the legacy raw-value bitmap inverted index path entirely.

Builds on the SPI surfaces introduced by #18364 (encoding) and #18365 (requiresDictionary / shouldInvalidateOnDictionaryChange).

What's covered

Auto-create dictionary

• Config deserialization (DictionaryIndexType.fromFieldConfigs): when a FieldConfig declares encodingType=RAW AND any enabled index returns IndexType.requiresDictionary()=true (FST, IFST, INVERTED, etc.), the dictionary config is left at its default-enabled state instead of being marked DISABLED. Validation and the rest of the pipeline see a dict-enabled column from the start.
• Segment creation (BaseSegmentCreator): DictionaryIndexConfig.requiresDictionary() drives auto-dict creation when a dict-requiring index is configured on a RAW column. If the user explicitly disabled the dictionary, a warning is logged and the dictionary is still created (alternative is a hard segment-build failure or silent index loss).
• Realtime mutable segments (MutableSegmentImpl): respects the encoding plumbing.
• Reload (ForwardIndexHandler.ENABLE_DICTIONARY op): if an existing RAW segment is loaded under a config that now requests a dict-requiring index, ForwardIndexHandler materializes a standalone dictionary; subsequent handlers (InvertedIndexHandler, RangeIndexHandler) build their dict-id-based indexes from it.
• Default columns (BaseDefaultColumnHandler): same logic for newly-added columns at reload.
• Reconciliation (FieldIndexConfigsUtil): fail-fast if the user-supplied ForwardIndexConfig.encodingType disagrees with the column-level FieldConfig.encodingType / noDictionaryColumns.

ForwardIndexHandler.computeOperations simplification

The reload-time decision logic in ForwardIndexHandler.computeOperations was rewritten to four orthogonal questions per column, with the dictionary state expressed as a single rule that drives every dict-toggle decision:

desiredDict = newIsDict || any-enabled-index-requires-dict

State is tracked as FieldConfig.EncodingType (DICTIONARY, RAW, or null for "forward index disabled / not on disk"), making the state space explicit. The transition steps run in this order:

1. Forward-index transition — based on existingFwdEncoding vs newFwdEncoding. Three sub-cases: forward disabled, forward re-enabled (rebuild from dict + inverted), or encoding flips with forward staying on.
2. Dictionary transition — based on existingHasDict vs desiredDict. Auto-enables the dict whenever a secondary index requires it; refuses to remove the dict if any enabled index in the new config still needs it.
3. Compression-type change — only when no encoding change happened.
4. Cross-cutting guards — sorted-column toggles ignored, range-index format compatibility check, enable-forward needs dict + inverted on disk, enable-dict needs forward to be on.

The ENABLE_FORWARD_INDEX operation is split into ENABLE_DICT_FORWARD_INDEX and ENABLE_RAW_FORWARD_INDEX so the intent is explicit. Both variants cover two on-disk shapes:

• Forward absent → rebuild from dict + inverted as the requested encoding.
• Forward present but encoding flips DICT⇄RAW → rewrite in place. The new helper convertDictForwardToRawKeepingDictionary handles the DICT→RAW + dict-kept case (e.g. when an inverted index is added to a dict-encoded column without explicitly enabling dict).

Range index correctness gates (RAW + dict combination)

Shared-dict + RAW exposes a subtle correctness hazard around range indexes: when a dictionary exists at index build time, both RangeIndexCreator (v1) and BitSlicedRangeIndexCreator (v2) build over dict IDs. v1 is non-exact — its query-time partial matches fall back to ScanBasedFilterOperator, which on a RAW forward column would have to apply a dict-based evaluator against raw values, silently producing wrong results.

• RangeIndexType.validate: rejects RAW forward + dictionary + RangeIndexCreator.VERSION (v1). Forces the BitSliced v2 (exact) range index for shared-dict + RAW columns so the failure mode is config validation rather than wrong query answers.
• RangeIndexHandler.needUpdateIndices / updateIndices: read the on-disk range index version (first int of the buffer) and rebuild when it differs from the configured version. v1 and v2 have incompatible on-disk layouts and different exact/non-exact semantics, so a version change requires a full rebuild.
• RangeIndexBasedFilterOperator.canEvaluate: returns false when dataSource.getDictionary() != null and the predicate evaluator isn't dict-based — falls through to ScanBasedFilterOperator, which correctly applies raw values against the raw forward index. Defends against any other path that produces a raw-value evaluator on a column whose range index was built over dict IDs.
• PredicateEvaluatorProvider.getDictionaryUsableForFiltering: per-predicate-type decision. For RANGE on a RAW forward column, the dictionary is kept only when the range index is exact (isExact() == true) so non-exact range readers can't be paired with a dict-based evaluator that scan-fallback can't apply.

Remove legacy raw-value bitmap inverted index

The pre-shared-dict format embedded its own dictionary inline inside the <col>.bitmap.inv.idx file (written by the now-deleted RawValueBitmapInvertedIndexCreator). Reads went through RawValueBitmapInvertedIndexReader and RawValueInvertedIndexFilterOperator. All three are deleted in favor of the standard BitmapInvertedIndexReader over a real standalone dictionary.

• SegmentPreProcessor.removeLegacyRawValueInvertedIndexes pre-pass detects the legacy 44-byte big-endian header (version=1 + cardinality + offsets) on segment load, deletes the file, and lets the new ForwardIndexHandler + InvertedIndexHandler chain rebuild the index in dict-id format.
• InvertedIndexType.ReaderFactory drops the hasDictionary branch; asserts a dictionary exists.

Dict-id-based rebuild path

• DictionaryBasedIndexBuilder (new, ~200 lines): shared helper extracted from InvertedIndexHandler and RangeIndexHandler — reads raw forward values, looks each up in the dictionary, feeds (value, dictId) pairs into a DictionaryBasedInvertedIndexCreator. Single per-data-type dispatch handles SV, MV, INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING/BYTES.
• InvertedIndexAndDictionaryBasedForwardIndexCreator: split _dictionaryEnabled into two flags — _dictionaryPresent (a standalone dict file exists) and _dictionaryBasedForwardIndex (the forward index stores dict IDs). The two are now independent (RAW forward + standalone dict is the new third state).
• BaseIndexHandler: wires the two-flag model.
• Handler ordering contract: InvertedIndexHandler's class-level Javadoc documents that it requires a dictionary by the time updateIndices runs, and that SegmentPreProcessor enforces this by always running ForwardIndexHandler first.

Query path: predicate evaluator selection on shared-dict columns

With shared-dict columns, predicates on the same column may need different evaluators: an EQ can use the dict-id-based inverted index, while a LIKE '%x%' on the same column must scan the raw forward.

• PredicateEvaluatorProvider.getDictionaryUsableForFiltering: per-predicate-type decision returning the dictionary only when a dict-consuming filter operator (sorted, inverted, exact range) is actually available and enabled for that specific predicate type. Inverted-only is dropped for RANGE; non-exact range is dropped for RANGE/EQ. RAW forward + scan reads raw values directly.
• FilterMvTransformFunction: per-value evaluation via transformToDictIdsMV requires forward.getDictIdMV() to actually serve dict ids cheaply. RAW forward indexes throw UnsupportedOperationException from that method, so when the inner Identifier wraps a RAW column (with or without a shared dictionary), the dictionary is dropped here and filterMv falls back to per-value raw matching.

DataFetcher

DataFetcher.addDataSource drops the (shared) dictionary when the forward index is RAW — the existing per-method branches in ColumnValueReader then take the raw-value paths uniformly without touching the read methods. Callers that genuinely need dict ids on a RAW + shared-dict column read raw values and consult the dictionary directly. DefaultGroupByExecutor similarly gates on forwardIndex.isDictionaryEncoded() so shared-dict + RAW columns route to the no-dict GROUP BY path.

addRaw(Object) SPI extraction

• ForwardIndexCreator.addRaw(Object) (new default method): extracted from the add(Object, dictId) body so handlers can write raw values without going through the dict-id routing branch. Required by the rebuild path that converts dict-id forward back to raw forward when dropping a dictionary.
• 7 concrete creators (CLPV1, CLPV2, MV/SV Fixed/VarByte, SameValue) implement add() → addRaw().

Tests

• IndexCombinationValidationTest (~640 lines, new) — exhaustive matrix over (encoding, dictionary, secondary index, column type, compression codec) combinations.
• LegacyRawValueInvertedIndexMigrationIntegrationTest (new, in pinot-integration-tests) — full-cluster integration test with a synthetically-constructed legacy segment (raw forward + legacy embedded-dict inverted file written by a resurrected LegacyRawValueBitmapInvertedIndexCreator). Tars it, uploads to a real cluster, verifies the server preprocessor migrates the segment and that EQ / IN / NOT_EQ queries return correct counts.
• RawForwardIndexWithDictionaryTest (~580 lines) — 33 cases covering SV/MV × EQ/RANGE/IN/REGEXP_LIKE on RAW forward + dict, plus comprehensive mixed-predicate matrix combining inverted-index dict path and raw-scan path on the same column with explicit non-zero expected counts.
• RawForwardIndexInvertedIndexTest — query-result equivalence vs DICTIONARY-encoded baseline.
• SegmentPreProcessorTest.testRangeIndexRebuiltOnDictionaryToggle (new) — asserts the range index is rebuilt with the right format every time the dictionary state toggles, covering both the auto path (driven by inverted-index add/remove) and the explicit toggle path.
• SegmentPreProcessorTest.testFlipDictForwardToRawForwardKeepingDictionaryForInvertedIndex (new) — encoding-flip end-to-end test asserting that a dict-encoded INT column with an inverted index, when reloaded under a config that puts the column in noDictionaryColumns, ends up as dict + inverted + raw forward (dict kept, encoding flipped, inverted intact).
• InvertedIndexHandlerTest — isLegacyRawValueInvertedIndexFormat detection + SegmentPreProcessor rebuild path.
• ForwardIndexHandlerTest (+477 lines) — reload-time auto-create-dict coverage.
• PredicateEvaluatorProviderTest (new) — per-predicate-type dict-drop decisions, including non-exact range and dict-required-by-inverted scenarios.
• FilterMvTransformFunctionTest — RAW + dict + inverted column filterMv parity with dict-encoded baseline; per-value path drops dict for RAW forward.
• SegmentPreProcessorTest, ColumnMetadataImplTest (new), SegmentGeneratorConfigTest, TableConfigUtilsTest, DictionaryIndexTypeTest, LazyRowTest, ForwardIndexHandlerReloadQueriesTest, TableIndexingTest + CSV — combination coverage and reload regressions.
• RealtimeSegmentConverterTest + CrcUtilsTest — CRC values updated for the new metadata key footprint.

Backward compatibility

• Old segments without FORWARD_INDEX_ENCODING in metadata.properties: ColumnMetadataImpl.fromPropertiesConfiguration falls back to inferring encoding from HAS_DICTIONARY (the field added by Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks #18364, which this PR depends on).
• Old segments with the legacy raw-value bitmap inverted format on disk: detected by the 44-byte header signature and migrated by SegmentPreProcessor on first load under the new code. Covered by LegacyRawValueInvertedIndexMigrationIntegrationTest.
• Old segments with v1 (non-exact) range index on a column that now needs a shared dictionary: RangeIndexHandler detects the on-disk version mismatch and rebuilds in v2 format on reload.
• Existing valid configs: noDictionaryColumns + invertedIndexColumns overlap, previously implicit, now produces a real dict + standard inverted via auto-create. No config change required.

Example FieldConfig

Implicit (auto-create dictionary because inverted needs it):

{
  "fieldConfigList": [
    {
      "name": "myColumn",
      "encodingType": "RAW",
      "indexes": { "inverted": {} }
    }
  ]
}

Explicit (also valid; both dictionary and inverted listed):

{
  "fieldConfigList": [
    {
      "name": "myColumn",
      "encodingType": "RAW",
      "compressionCodec": "LZ4",
      "indexes": { "dictionary": {}, "inverted": {} }
    }
  ]
}

Out of scope (follow-up PR)

The full operator-routing migration to gate on BlockValSet.isDictionaryEncoded() (instead of BlockValSet.getDictionary() != null) is split into a separate branch blockvalset-isDictionaryEncoded-routing-for-shared-dict-raw. That branch wires the gate through DistinctExecutorFactory, NoDictionaryMultiColumnGroupKeyGenerator, and all DistinctCount* aggregation functions. This PR includes only the DefaultGroupByExecutor, BinaryOperatorTransformFunction, and FilterMvTransformFunction gate updates needed to keep the new shared-dict + RAW tests green.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enables dictionary inference for indexes that require dictionaries (inverted, FST, range) and enhances raw forward index handling throughout the codebase. It introduces automatic dictionary creation when required by certain index types, plumbs raw-encoding awareness through forward index creator/reader factories, and extends the forward index handler to support raw-forward columns in various operations.

Key changes:

• Automatic dictionary inference when indexes requiring dictionaries are configured
• Raw-encoding flag in ForwardIndexConfig to distinguish raw vs. dictionary-encoded forward indexes
• Enhanced ForwardIndexHandler to support raw forward indexes with dictionary add/remove operations
• New integration test verifying raw forward index + inverted index functionality

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
githubComplexTypeEvents_offline_table_config.json	Adds raw-encoded forward index with inverted index configuration example
ForwardIndexCreator.java	Refactors raw value handling into separate addRaw method
IndexType.java	Adds requiresDictionary method to indicate dictionary requirement
ForwardIndexUtils.java	New utility to identify raw forward index columns from table config
ForwardIndexConfig.java	Adds rawEncoding flag and getter/builder methods
FieldIndexConfigsUtil.java	Implements automatic dictionary enablement for required indexes
DictionaryIndexConfig.java	Adds utility methods to check if dictionary is required by any index
ForwardIndexTypeTest.java	Updates tests to include rawEncoding flag in expected configs
ColumnMinMaxValueGenerator.java	Passes ForwardIndexConfig when creating readers
InvertedIndexAndDictionaryBasedForwardIndexCreator.java	Respects rawEncoding flag when determining dictionary usage
ForwardIndexHandler.java	Adds ENABLE_DICTIONARY_FOR_RAW_FORWARD_INDEX operation and handling logic
InvertedIndexType.java	Implements requiresDictionary to return true
IFSTIndexType.java	Implements requiresDictionary to return true
FstIndexType.java	Implements requiresDictionary to return true
ForwardIndexType.java	Updates getFileExtension to return multiple possible extensions
ForwardIndexReaderFactory.java	Uses rawEncoding flag to determine reader type
ForwardIndexCreatorFactory.java	Uses rawEncoding flag to determine creator type
SingleValueVarByteRawIndexCreator.java	Overrides add method to delegate to addRaw
SingleValueFixedByteRawIndexCreator.java	Overrides add method to delegate to addRaw
MultiValueVarByteRawIndexCreator.java	Overrides add method to delegate to addRaw
MultiValueFixedByteRawIndexCreator.java	Overrides add method to delegate to addRaw
CLPForwardIndexCreatorV2.java	Overrides add method for CLP encoding
CLPForwardIndexCreatorV1.java	Overrides add method for CLP encoding
SegmentIndexCreationDriverImpl.java	Fixes inverted logic bug (isDisabled → isEnabled)
SegmentColumnarIndexCreator.java	Adds warning when dictionary required but explicitly disabled
SameValueForwardIndexCreator.java	Overrides add method to delegate to addRaw
RawForwardIndexInvertedIndexTest.java	New integration test for raw forward index with inverted index
ForwardIndexHandlerReloadQueriesTest.java	Updates test expectations and adds raw forward index test coverage
DataFetcher.java	Adds _useDictionary field for proper dictionary usage check

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 64.31298% with 187 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.65%. Comparing base (5ccd383) to head (0e11efc).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...der/invertedindex/DictionaryBasedIndexBuilder.java	12.19%	69 Missing and 3 partials ⚠️
...ocal/segment/index/loader/ForwardIndexHandler.java	79.12%	14 Missing and 24 partials ⚠️
...ator/transform/function/ItemTransformFunction.java	0.00%	19 Missing ⚠️
...segment/spi/index/creator/ForwardIndexCreator.java	0.00%	15 Missing ⚠️
...dex/loader/invertedindex/InvertedIndexHandler.java	62.96%	7 Missing and 3 partials ⚠️
.../segment/index/dictionary/DictionaryIndexType.java	86.53%	1 Missing and 6 partials ⚠️
...ertedindex/LegacyRawValueInvertedIndexCleanup.java	75.00%	3 Missing and 3 partials ⚠️
...r/filter/predicate/PredicateEvaluatorProvider.java	80.95%	1 Missing and 3 partials ⚠️
.../index/loader/invertedindex/RangeIndexHandler.java	82.60%	2 Missing and 2 partials ⚠️
...operator/filter/RangeIndexBasedFilterOperator.java	50.00%	1 Missing and 2 partials ⚠️
... and 5 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17269      +/-   ##
============================================
+ Coverage     63.57%   63.65%   +0.08%     
- Complexity     1717     1723       +6     
============================================
  Files          3252     3254       +2     
  Lines        199132   199459     +327     
  Branches      30875    30977     +102     
============================================
+ Hits         126596   126966     +370     
+ Misses        62454    62361      -93     
- Partials      10082    10132      +50     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	63.65% <64.31%> (+0.08%)	⬆️
temurin	63.65% <64.31%> (+0.08%)	⬆️
unittests	63.65% <64.31%> (+0.08%)	⬆️
unittests1	55.72% <39.50%> (+0.06%)	⬆️
unittests2	34.95% <55.34%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 7 comments.

[Jackie-Jiang]

For future proof, how do you plan to support both dictionary encoded and raw forward index on the same column? Specifically, how does the FieldConfig look like?

[Copilot]

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 6 comments.

[xiangfu0]

Opened a follow-up docs PR for this change: pinot-contrib/pinot-docs#804