Nested KNN vs inner hits scoring consistency

[tteofili]

This attempts at fixing child visibility issue at #144207 and approximate vs rescore score consistency issues (#138011 and #138496) for nested KNN.

For nested vectors, we retrieve vectors in child (nested) space but ultimately return parents, so in the diversifying kNN step we gather enough nearest nested hits and then fold them up under parents. When that step only considers too few children per parent, the parent score can be computed from an incomplete set of nested candidates. Meanwhile inner_hits may be driven by a different evaluation path (e.g. focusing on children with exact kNN during inner-hit rewrite).

This adds an internal flag useMaxNestedKnnCandidatesInNested such that when the query runs in a nested-parent context (parentBitSet != null), the implementation raises the effective k and num_candidates to at least the (new) MAX_NESTED_KNN_CANDIDATES_SETTING index setting (defaults to 100).
Higher index setting values will diversify over more nested docs per parent (more CPU/memory, usually stabler parent vs inner-hit agreement when many children compete). Lower values will mean cheaper execution, but easier to miss the nested child combination that dominates the true max score for that parent.

[elasticsearchmachine]

Hi @tteofili, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[benwtrent]

I disagree with this approach. I don't think it fixes two of the main issues and its a dangerous performance hit.

If we want to force the index exploration to gather MORE children per parent node, we need to do something directly with the graph/index exploration (example: apache/lucene#16034)

However, that said, I am not sure that this approach actually fixes the issue. Its significantly dependent on the distribution of the number of children vs. parent. I don't think a new magic number is what we need.

This inconsistency seems to be caused by multiple places:

• Quantized vs. raw scoring
• Potentially seeing NEW children that weren't considered before

First one is scorer focused
Second one is approximation focused