Skip to content

Commit 043b104

Browse files
committed
docs: actualize benchmark, architecture, CI and install docs to 5.4.0 state
Sweep by an opus documentation pass, closing the last gaps after the MoonSharp removal + R6 + F-12/F-22: - benchmark guide: package-graph version 5.3.0→5.4.0; embed the benchmark_v2_frontier.svg comparison chart. - INSTALL: benchmark groups G1-G7 → G1-G8. - ARCHITECTURE + DEVELOPER_GUIDE: document the TimeoutLlmClientDecorator + RetryingStreamingLlmClientDecorator in the LLM decorator chain (Timeout → Logging → RetryingStreaming → Routing) and the portable-core timeout. - CONTRIBUTING: document the new CI jobs (standalone CoreAI.Core.Tests, fork-safe package-graph lockstep, merge-queue-gate that fails without UNITY_LICENSE).
1 parent b31febc commit 043b104

5 files changed

Lines changed: 18 additions & 7 deletions

File tree

Assets/CoreAIBenchmark/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CoreAI Game-Creation Benchmark
22

3-
Package graph: `com.neoxider.coreaibenchmark` 5.3.0 depends on `com.neoxider.coreai`,
3+
Package graph: `com.neoxider.coreaibenchmark` 5.4.0 depends on `com.neoxider.coreai`,
44
`com.neoxider.coreaiunity`, and `com.neoxider.coreaimods` at the same version. The Mods dependency is
55
required because the scenarios instantiate and execute the real Lua tool/runtime.
66

@@ -223,6 +223,8 @@ Frontier hosted models, run through the [`cli-agents`](https://github.com/) `ope
223223
OpenAI-compatible shim over the Claude Code / Codex CLIs. This is a **separate leaderboard** from the local
224224
LM Studio models above (different suite version *and* different backend class). Ranked by suite base score.
225225

226+
![CoreAI Game-Creation Benchmark v2 — frontier-model comparison](../../Docs/Images/benchmark_v2_frontier.svg)
227+
226228
| # | Model | Score | Pass-rate | P/PA/F | G1 | G2 | G3 | G4 | G5 | G6 | G7 | G8 | Tokens |
227229
|---:|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
228230
| 1 | `codex/gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 96.3 | 100 | 100 | 100 | 100 | 63.9 | 100 | 96.3 | 16379 |

Assets/CoreAiUnity/Docs/ARCHITECTURE.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,10 @@ Tool calls execute in **parallel** on both the batch and the streamed path, boun
2020
flowchart TD
2121
Game["Game or UI"] --> Orchestrator["IAiOrchestrationService"]
2222
Orchestrator --> LlmClient["ILlmClient"]
23-
LlmClient --> Logging["LoggingLlmClientDecorator"]
24-
Logging --> Routing["RoutingLlmClient"]
23+
LlmClient --> Timeout["TimeoutLlmClientDecorator"]
24+
Timeout --> Logging["LoggingLlmClientDecorator"]
25+
Logging --> RetryStream["RetryingStreamingLlmClientDecorator"]
26+
RetryStream --> Routing["RoutingLlmClient"]
2527
Routing --> Registry["LlmClientRegistry"]
2628
Registry --> LocalModel["LocalModel"]
2729
Registry --> ClientOwnedApi["ClientOwnedApi"]
@@ -84,9 +86,9 @@ If the backend reports **`LlmErrorCode.ContextLengthExceeded`** (`MeaiOpenAiChat
8486

8587
## Timeout & Retry Rule (v1.5.1)
8688

87-
**Timeout:** enforced exclusively by `CoreAiChatService` via UniTask `CancelAfterSlim` (PlayerLoop-based, WebGL-compatible). The portable layer (`AiOrchestrator`, `LoggingLlmClientDecorator`) passes `CancellationToken` through without wrapping. See [`STREAMING_ARCHITECTURE.md`](STREAMING_ARCHITECTURE.md) §8.
89+
**Timeout:** the interactive UI timeout is enforced by `CoreAiChatService` via UniTask `CancelAfterSlim` (PlayerLoop-based, WebGL-compatible). Since **5.4.0** the portable pipeline also wraps every client in `TimeoutLlmClientDecorator`, which applies a request timeout off `ICoreAISettings.LlmRequestTimeoutSeconds` on both the streaming and non-streaming paths (belt-and-braces for hosts that call the pipeline outside the chat service). See [`STREAMING_ARCHITECTURE.md`](STREAMING_ARCHITECTURE.md) §8.
8890

89-
**Retries:** network-level retries (HTTP 429, 5xx, exponential backoff) are handled exclusively by `LoggingLlmClientDecorator`. The orchestrator does not multiply those retries with its own counters.
91+
**Retries:** network-level retries (HTTP 429, 5xx, exponential backoff) are handled exclusively by `LoggingLlmClientDecorator`. The orchestrator does not multiply those retries with its own counters. Since **5.4.0** `RetryingStreamingLlmClientDecorator` additionally retries a *streaming* call, but only before the stream commits any content, so a mid-stream failure is never silently restarted. `LlmPipelineInstaller` composes the chain as `Timeout( Logging( RetryingStreaming( routed ) ) )`.
9092

9193
**Context-length retry:** in addition to network retries above, **`AiOrchestrator.RunTaskAsync`** may issue bounded additional LLM calls when the completion result carries **`ContextLengthExceeded`**, after rebuilding prompts with progressively tighter history compaction.
9294

Assets/CoreAiUnity/Docs/DEVELOPER_GUIDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ flowchart LR
9797

9898
1. The **game** calls **`IAiOrchestrationService.RunTaskAsync(AiTaskRequest)`** (role, hint, **`Priority`**, **`CancellationScope`**, optional Lua repair fields, **`TraceId`**).
9999
2. The default implementation is **`QueuedAiOrchestrator`** (concurrency limit, priority, canceling the previous task with the same **`CancellationScope`**) around **`AiOrchestrator`**. **`AiOrchestrator`** assigns **`TraceId`**, assembles prompts, asks **`IConversationContextManager`** to prepare long chat history, then obtains a completion — **streaming by default** (drives **`ILlmClient.CompleteStreamingAsync`** and collapses the stream to a result when **`ICoreAISettings.EnableStreaming`** is on, the same execute-as-you-stream tool path as chat), falling back to **`ILlmClient.CompleteAsync`** only when streaming is off; with **`IRoleStructuredResponsePolicy`** for a role, **one** retry is allowed with a **`structured_retry:`** hint in user/hint. Then **`ApplyAiGameCommand`** is published (**`AiEnvelope`**, **`TraceId`**, …). Metrics — **`IAiOrchestrationMetrics`** (log under **`GameLogFeature.Metrics`**).
100-
3. In DI, **`ILlmClient`** is **`LoggingLlmClientDecorator`** around **`RoutingLlmClient`** (or a legacy single client): inside — **`OpenAiChatLlmClient`** / **`MeaiLlmUnityClient`** / **`StubLlmClient`** per **`LlmRoutingManifest`** and role. Log **`GameLogFeature.Llm`** (`LLM ▶` / `LLM ◀` / `LLM ⏱`), backend line **`RoutingLlmClient→OpenAiHttp`**, etc. For “is this stub?” — **`LoggingLlmClientDecorator.Unwrap(client)`**.
100+
3. In DI (composed by **`LlmPipelineInstaller`** as `Timeout( Logging( RetryingStreaming( routed ) ) )`), **`ILlmClient`** is **`TimeoutLlmClientDecorator`****`LoggingLlmClientDecorator`****`RetryingStreamingLlmClientDecorator`** around **`RoutingLlmClient`** (or a legacy single client): inside — **`OpenAiChatLlmClient`** / **`MeaiLlmUnityClient`** / **`StubLlmClient`** per **`LlmRoutingManifest`** and role. Log **`GameLogFeature.Llm`** (`LLM ▶` / `LLM ◀` / `LLM ⏱`), backend line **`RoutingLlmClient→OpenAiHttp`**, etc. For “is this stub?” — **`LoggingLlmClientDecorator.Unwrap(client)`**.
101101
4. Subscriber **`AiGameCommandRouter`** receives **`ApplyAiGameCommand`** from MessagePipe and **marshals handling to the Unity main thread** (`UniTask.SwitchToMainThread`), then calls **`LuaAiEnvelopeProcessor.Process`**: Lua is extracted from text, executed in the sandbox with API from **`IGameLuaRuntimeBindings`**; **`[MessagePipe]`** logs include the same task **`traceId`**.
102102
5. On success / failure, **`LuaExecutionSucceeded`** / **`LuaExecutionFailed`** are published (**`TraceId`** preserved). For the **Programmer** role on error, the orchestrator is invoked again with repair context and the same **`TraceId`** (up to **3 attempts** by default, configurable via **`CoreAISettings.MaxLuaRepairRetries`**).
103103

CONTRIBUTING.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ If the hook blocks your commit, unstage the listed files with `git restore --sta
2525
Symbols, compiling all Lua features out). Both must stay green.
2626
- **Sandbox coverage gate** — the `lua` job fails if the `SecureLuaSandboxEditModeTests`
2727
escape-test fixture did not actually execute, so Lua isolation coverage cannot silently drop out.
28+
- **Standalone core tests** — a package-local `CoreAI.Core.Tests` EditMode assembly proves the
29+
`com.neoxider.coreai` package compiles and tests on its own, without the Unity layer.
30+
- **`package-graph` job** — fork-safe (no Unity licence needed): checks that all five packages carry
31+
the same lockstep version and that their internal dependency pins agree.
32+
- **`merge-queue-gate` job** — on the `merge_group` trigger the workflow **fails** (does not skip)
33+
when `UNITY_LICENSE` is absent, so a PR can never be merged through the queue without the licensed
34+
test run having actually executed.
2835

2936
The workflow requires the standard [GameCI](https://game.ci/docs/github/getting-started) repository
3037
secrets: `UNITY_LICENSE`, `UNITY_EMAIL`, `UNITY_PASSWORD`.

INSTALL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ No further setup — the Hub window picks up any pages already registered by the
172172

173173
## 5. Benchmark module (dev-only)
174174

175-
`com.neoxider.coreaibenchmark` is the LLM game-creation benchmark harness (scenario groups G1-G7,
175+
`com.neoxider.coreaibenchmark` is the LLM game-creation benchmark harness (scenario groups G1-G8,
176176
scoring, role-fitness reports) used to evaluate model quality — it is a development/test-time tool,
177177
not something a shipped game depends on.
178178

0 commit comments

Comments
 (0)