Feat: new column access mechanism

.changeset/calm-ravens-deny.md

@@ -0,0 +1,12 @@
+---
+"@milaboratories/columns-collection-driver": minor
+"@milaboratories/milaboratories.ui-examples.ui": minor
+"@milaboratories/pl-middle-layer": minor
+"@milaboratories/pf-driver": minor
+"@milaboratories/pl-model-common": minor
+"@milaboratories/pl-tree": minor
+"@platforma-sdk/ui-vue": minor
+"@platforma-sdk/model": minor
+---
+
+ColumnLazy / ColumnsCollection API that bring new way to work with columns in model part

docs/column-identity.md

@@ -0,0 +1,141 @@
+# Column identity: physical vs logical
+
+This document defines the conventions for column identifiers across the SDK
+and the host (`pl-middle-layer`). Two distinct identities are involved in a
+table-rendering pipeline; mixing them up causes engine-level "column not
+found" errors and silent variant deduplication.
+
+## The two identities
+
+### Physical identity — `PObjectId`
+
+A bare `PObjectId` (`LocalPObjectId | GlobalPObjectId`) names exactly one
+**stored** column in the result pool. It is the key the host column
+registry resolves against.
+
+Use it (and only it) at these boundaries:
+
+- Column registry lookup on the host (`resolvePColumnById` in
+  `lib/node/pl-middle-layer/src/js_render/column_registry.ts`).
+- The id list passed to `ctx.createPFrame([...ids])` — PFrame is the
+  physical column registry, one entry per bare id, so duplicates must be
+  collapsed (use `extractPObjectId` + dedupe).
+- Spec frame keys passed to `pframeSpec.createSpecFrame({...})` when
+  evaluating queries over already-resolved physical specs.
+
+### Logical identity — `ColumnUniversalId`
+
+`ColumnUniversalId` is the union of bare ids and rich wrapper ids
+serialized as canonical JSON:
+
+- `LocalPObjectId` / `GlobalPObjectId` — bare physical id.
+- `ColumnFilteredId` — `{source, axisFilters}`.
+- `ColumnDiscoveredId` — `{column, path, qualifications}`.
+- `ColumnOverridedId` — `{source, specOverrides}`.
+
+Definitions: `lib/model/common/src/drivers/pframe/spec/ids.ts`.
+
+A logical id describes a contextualised view of a physical column: which
+linker chain you reached it through, which axes are sliced, which spec
+overrides are applied. Two `ColumnDiscoveredId`s sharing the same
+`column` but with different `path` are *different* logical columns even
+though they point to the same stored data.
+
+Use it (and only it) everywhere else:
+
+- `ColumnRecipe.id` (`sdk/model/src/columns/column_recipes/types.ts`).
+- The `column` field of every `SpecQuery` leaf (`{type: "column", column: <id>}`).
+- `PTableColumnId.id` in filter and sorting references — UI gets these
+  from the recipe and ships them back unchanged.
+- `derivedLabels` / `hiddenColumnIds` keys: variants of the same physical
+  column have independent labels and independent hide/show state.
+
+### Crossing the boundary: `extractPObjectId`
+
+`extractPObjectId(id: ColumnUniversalId): PObjectId`
+(`lib/model/common/src/drivers/pframe/spec/ids.ts:97`) is the **only**
+sanctioned way to drop from a logical id to its physical counterpart. It
+walks wrapper layers (`isColumnFilteredKey` → `source`,
+`isColumnOverridedKey` → `source`, `isColumnDiscoveredKey` → `column`)
+until it reaches a bare `PObjectId`.
+
+Call it only at the boundary points listed above. Recipe internals and
+SpecQuery construction must never strip down to bare ids early.
+
+## Recipe contract
+
+Every concrete `ColumnRecipe` must satisfy:
+
+1. `recipe.id` is its full logical id (`ColumnUniversalId`).
+2. `recipe.getQuery()` returns a `SpecQuery` whose terminal column leaf
+   carries `recipe.id` — not the inner recipe's id, not the bare id.
+
+For leaf recipes (`ColumnLazyImpl`) this is trivial: `id` is the bare
+`PObjectId` and the query is `{type: "column", column: id}`.
+
+For wrapper recipes (Overrided / Filtered / Discovered) the inner recipe
+already produced a query whose leaf carries the **inner's** id. The
+wrapper must lift its own id onto that leaf while preserving everything
+else (linker chains, sliceAxes, specOverride wrappers). Use the shared
+helper:
+
+```ts
+import { rebrandLeafId } from "./leaf_rebrand";
+
+// inside Wrapper.getQuery():
+const wrapped = wrapWith(rebrandLeafId(this.inner.getQuery(), this.inner.id, this.id));
+```
+
+`rebrandLeafId(node, fromId, toId)`
+(`sdk/model/src/columns/column_recipes/leaf_rebrand.ts`) walks the full
+`SpecQuery` tree via `mapSpecQueryColumns` and renames only those column
+leaves whose id equals `fromId`. Linker leaves and unrelated refs are
+left untouched. This is the only correct way to compose wrapper queries.
+
+## SpecQuery generics
+
+The default leaf type of `SpecQuery<C>` was widened from `PObjectId` to
+`ColumnUniversalId`
+(`lib/model/common/src/drivers/pframe/query/query_spec.ts`). The same
+default propagates to `SpecQueryColumn`, `SpecQueryJoinEntry`,
+`SpecQueryLinkerJoin`, `SpecExprColumnRef`, `PTableColumnSpecColumn.id`,
+`PTableColumnIdColumn.id`, `QueryColumnIdColumn.id`. Existing call sites
+that pin `<PObjectId>` keep their narrow contract; everything else
+widens to accept rich ids.
+
+This makes the type system match the runtime: leaves and table-column
+ids genuinely carry logical ids, and we no longer need to cast.
+
+## Why per-variant uniqueness matters
+
+`pframe-engine` deduplicates output columns by their leaf id. If two
+`ColumnDiscoveredRecipe` instances reach the same physical hit through
+different linker chains and both emit `{column: <bare hit id>}` at the
+leaf, the engine sees one column and the second variant disappears
+silently. With every recipe lifting its own id to the leaf, the engine
+sees two distinct leaves and produces two output columns — which is what
+the table is asking for.
+
+The same logic applies to wrappers used as filter / sort targets: the
+resolver returns `recipe.id` (logical), so the engine must find a column
+with that logical id in the integrated table. Lifting wrapper ids onto
+the leaves is what makes that lookup succeed.
+
+## Where these rules live in code
+
+- Type definitions: `lib/model/common/src/drivers/pframe/spec/ids.ts`,
+  `lib/model/common/src/drivers/pframe/query/query_spec.ts`,
+  `lib/model/common/src/drivers/pframe/table_common.ts`.
+- Recipe implementations:
+  `sdk/model/src/columns/column_lazy.ts`,
+  `sdk/model/src/columns/column_recipes/column_overrided_recipe.ts`,
+  `sdk/model/src/columns/column_recipes/column_filtered_recipe.ts`,
+  `sdk/model/src/columns/column_recipes/column_discovered_recipe.ts`,
+  `sdk/model/src/columns/column_recipes/leaf_rebrand.ts`.
+- Linker collection (logical→physical at the boundary):
+  `sdk/model/src/columns/utils.ts` (`collectLinkerIds`,
+  `collectLinkerColumns`).
+- Resolver (filters/sorting): `sdk/model/src/components/PlDataTable/columnResolver.ts`.
+- Table assembly: `sdk/model/src/components/PlDataTable/createPlDataTable/createPlDataTableV3.ts`
+  (notice `createPFrame(uniq([...].map(extractPObjectId)))`).
+- Host-side physical resolver: `lib/node/pl-middle-layer/src/js_render/column_registry.ts`.

etc/blocks/filter-column-test/test/src/wf.test.ts

@@ -1,10 +1,10 @@
 import type { BlockData } from "@milaboratories/milaboratories.test-filter-column.model";
 import type { platforma } from "@milaboratories/milaboratories.test-filter-column.model";
-import { blockSpec as tableTestBlockSpec } from "@milaboratories/milaboratories.test-block-table";
 import type { InferBlockState, Platforma } from "@platforma-sdk/model";
 import { createDatasetSelection, createPrimaryRef, wrapOutputs } from "@platforma-sdk/model";
 import type { ML, RawHelpers } from "@platforma-sdk/test";
 import { awaitStableState, blockTest } from "@platforma-sdk/test";
+import { blockSpec as tableTestBlockSpec } from "@milaboratories/milaboratories.test-block-table";
 import { blockSpec } from "this-block";
 import { assert } from "vitest";
 

etc/blocks/table-test/model/src/index.ts

@@ -1,10 +1,15 @@
 import {
   BlockModelV3,
+  ColumnUniversalId,
+  ColumnsCollection,
   DataModelBuilder,
   PObjectId,
   PlDataTableFilters,
   createPlDataTableStateV2,
   createPlDataTableV3,
+  deriveAxisValuesLabels,
+  expandByPartition,
+  isColumnLazy,
   type InferHrefType,
   type InferOutputsType,
   type PlDataTableStateV2,
@@ -13,11 +18,13 @@ import {
 export type BlockData = {
   label: string;
   tableState: PlDataTableStateV2;
+  tableSplitState: PlDataTableStateV2;
 };
 
 const blockDataModel = new DataModelBuilder().from<BlockData>("v1").init(() => ({
   label: "Table Test",
   tableState: createPlDataTableStateV2(),
+  tableSplitState: createPlDataTableStateV2(),
 }));
 
 export type BlockArgs = BlockData;
@@ -32,6 +39,11 @@ export const platforma = BlockModelV3.create(blockDataModel)
         href: "/",
         label: "Table V3",
       },
+      {
+        type: "link",
+        href: "/split",
+        label: "Table Split",
+      },
     ];
   })
 
@@ -69,8 +81,10 @@ export const platforma = BlockModelV3.create(blockDataModel)
         filters: [
           {
             type: "greaterThan",
-            column:
-              '{"type":"column","id":"{\\"name\\":\\"value\\",\\"resolvePath\\":[\\"main\\",\\"tableFrame\\"]}"}',
+            column: {
+              type: "column",
+              id: '{"name":"value","resolvePath":["main","tableFrame"]}' as PObjectId,
+            },
             x: 11,
           },
         ],
@@ -84,21 +98,66 @@ export const platforma = BlockModelV3.create(blockDataModel)
       displayOptions: {
         ordering: [
           // "category" leftmost (highest priority)
-          { match: (spec) => spec.name === "category", priority: 20 },
+          { match: { name: "^category$" }, priority: 20 },
           // Then "value"
-          { match: (spec) => spec.name === "value", priority: 10 },
+          { match: { name: "^value$" }, priority: 10 },
           // Unmatched columns (score, note) keep their original order
         ],
         visibility: [
           // "note" hidden by default (user can re-enable in UI)
-          { match: (spec) => spec.name === "note", visibility: "hidden" },
+          { match: { name: "^note$" }, visibility: "hidden" },
           // "score" optional — hidden by default but toggleable
-          { match: (spec) => spec.name === "score", visibility: "optional" },
+          { match: { name: "^score$" }, visibility: "optional" },
         ],
       },
     });
   })
 
+  // Same `value`/`name` anchor as tableV3, but with extra columns produced by
+  // splitting the `count` (group, name) column on its `group` axis: each group
+  // value becomes its own ColumnOverridedRecipe in `columns`, joining onto the
+  // primary on the shared `name` axis.
+  .outputWithStatus("tableSplitV3", (ctx) => {
+    const valueAnchor = { name: "value", axes: [{ name: "name" }] };
+
+    // Use only leaf (ColumnLazy) hits as primary. `discover` with anchors can
+    // also surface multi-axis Discovered variants (e.g. count [group, name]
+    // reached via linker_name_group_alt); those belong in secondary, not in
+    // the join's primary side — mirrors what `discoverTableColumns` does
+    // internally for the selector-form path.
+    const primary = ColumnsCollection()
+      .discover({ anchors: { main: valueAnchor }, mode: "exact" })
+      .getColumns()
+      .filter(isColumnLazy);
+    if (primary.length === 0) return undefined;
+
+    const countLeaves = ColumnsCollection()
+      .filter({ include: { name: [{ type: "exact", value: "count" }] } })
+      .getColumns()
+      .filter(isColumnLazy);
+
+    const splitRecipes = expandByPartition(countLeaves, [{ idx: 0 }], {
+      axisValuesLabels: deriveAxisValuesLabels(),
+    });
+    if (splitRecipes === undefined) return undefined;
+
+    const primaryIds = new Set<ColumnUniversalId>(primary.map((c) => c.id));
+    const secondary = ColumnsCollection()
+      .discover({ anchors: { main: valueAnchor }, mode: "enrichment", maxHops: 4 })
+      .getColumns()
+      .filter((c) => !primaryIds.has(c.id));
+
+    return createPlDataTableV3(ctx, {
+      primaryColumns: primary,
+      columns: [...secondary, ...splitRecipes],
+
+      tableState: ctx.data.tableSplitState,
+      labelsOptions: {
+        formatters: { linker: (linkerLabels) => `[${linkerLabels.join(" > ")}]` },
+      },
+    });
+  })
+
   .done();
 
 export type BlockOutputs = InferOutputsType<typeof platforma>;

etc/blocks/table-test/test/src/wf.test.ts

@@ -2,10 +2,9 @@ import type { PTableHandle } from "@platforma-sdk/model";
 import { blockTest } from "@platforma-sdk/test";
 import { blockSpec } from "this-block";
 
-type TableOutput = {
-  ok: boolean;
-  value: { visibleTableHandle: string; fullTableHandle: string } | undefined;
-};
+type TableOutput =
+  | { ok: true; value: { visibleTableHandle: string; fullTableHandle: string }; stable: boolean }
+  | { ok: false; errors: { message?: string }[]; moreErrors: boolean };
 
 blockTest(
   "V3 table resolves linked columns through multi-hop linker chain",
@@ -17,8 +16,11 @@ blockTest(
     const state = await helpers.awaitBlockDoneAndGetStableBlockState(blockId, 50000);
 
     const tableOutput = state.outputs!["tableV3"] as TableOutput;
-    expect(tableOutput.ok).toBe(true);
-    expect(tableOutput.value).toBeDefined();
+    if (!tableOutput.ok) {
+      throw new Error(
+        `tableV3 failed:\n${tableOutput.errors.map((e) => e.message ?? JSON.stringify(e)).join("\n")}`,
+      );
+    }
 
     const pFrameDriver = ml.driverKit.pFrameDriver;
 
@@ -28,7 +30,7 @@ blockTest(
     //   2-hop linked via name→group→region chain: region axis + regionName + population + linker_group_region
     // Total: 5 + 4 + 4 = 13 columns (at least)
     // Block applies filter `value > 11` → 4 rows (Alpha with value=10 excluded).
-    const fullHandle = tableOutput.value!.fullTableHandle as PTableHandle;
+    const fullHandle = tableOutput.value.fullTableHandle as PTableHandle;
     const fullShape = await pFrameDriver.getShape(fullHandle);
     expect(fullShape.rows).toBe(4);
     expect(fullShape.columns).toBeGreaterThan(9); // well above direct-only count
@@ -59,7 +61,7 @@ blockTest(
     // Visible table should also resolve without "axis not present in join result" error.
     // columnsDisplayOptions hides "note" and marks "score" optional (hidden by default),
     // so visible table has fewer columns than full but still contains the remaining ones.
-    const visibleHandle = tableOutput.value!.visibleTableHandle as PTableHandle;
+    const visibleHandle = tableOutput.value.visibleTableHandle as PTableHandle;
     const visibleShape = await pFrameDriver.getShape(visibleHandle);
     expect(visibleShape.rows).toBe(4);
     expect(visibleShape.columns).toBeGreaterThan(3);
@@ -72,3 +74,46 @@ blockTest(
     expect(visibleValues).toContain(JSON.stringify(["B", "A", "B", "A"]));
   },
 );
+
+blockTest(
+  "V3 split table fans out `count` over the `group` partition axis",
+  { timeout: 60000 },
+  async ({ rawPrj: project, helpers, expect, ml }) => {
+    const blockId = await project.addBlock("Block", blockSpec);
+
+    await project.runBlock(blockId);
+    const state = await helpers.awaitBlockDoneAndGetStableBlockState(blockId, 50000);
+
+    const splitOutput = state.outputs!["tableSplitV3"] as TableOutput;
+    if (!splitOutput.ok) {
+      throw new Error(
+        `tableSplitV3 failed:\n${splitOutput.errors.map((e) => e.message ?? JSON.stringify(e)).join("\n")}`,
+      );
+    }
+
+    const pFrameDriver = ml.driverKit.pFrameDriver;
+
+    // tableSplitV3 has no filter and no sort override — primary is `value`
+    // discovered via {main: value@name}; all 5 names participate.
+    const fullHandle = splitOutput.value.fullTableHandle as PTableHandle;
+    const fullShape = await pFrameDriver.getShape(fullHandle);
+    expect(fullShape.rows).toBe(5);
+
+    const fullIndices = Array.from({ length: fullShape.columns }, (_, i) => i);
+    const fullData = await pFrameDriver.getData(fullHandle, fullIndices);
+
+    // `count` (group, name) is partitioned by `group`. expandByPartition fans
+    // it out into one column per group value — G1 and G2. Each split column
+    // has `group` axis sliced away and its `domain.group = "<value>"` patched
+    // via specOverride. The values come from count.tsv:
+    //   G1: A=100, B=200, C=10,  D=25, E=5
+    //   G2: A=50,  B=150, C=300, D=75, E=1000
+    const findInt = (expected: number[]) =>
+      fullData.find(
+        (c) => c.type === "Int" && JSON.stringify([...c.data]) === JSON.stringify(expected),
+      );
+
+    expect(findInt([100, 200, 10, 25, 5])).toBeDefined();
+    expect(findInt([50, 150, 300, 75, 1000])).toBeDefined();
+  },
+);