Merge pull request #59 from Database-Tycoon/v0.1.9

Release v0.1.9 — Google Sheets, managed .venv, Quack, notify & schedule

Author: db-tycoon-stephen

CHANGELOG.md

@@ -1,5 +1,34 @@
 All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.1.9] - 2026-06-10
+
+_Headline: Google Sheets source ([#52][]). Plus managed project-local `.venv` + `doctor --fix` ([#57][]), DuckDB Quack live warehouse ([#42][]), pipeline notifications ([#46][]), and scheduled runs ([#48][]). Layer-granular backup ([#31][]) is deferred — no backup track exists yet to make layer-aware (it depended on the deferred DuckLake path). Also closes two High-severity findings from a security review ([#60][], [#61][]). Cycle plan in [`docs/proposals/v0.1.9-scope.md`](docs/proposals/v0.1.9-scope.md)._
+
+### Added
+
+- **`tycoon schedule` — local cron-less scheduled runs** ([#48][]). Run any tycoon command on a timer via the platform-native scheduler — macOS launchd LaunchAgents, Linux `systemd --user` timers — with no daemon and no cloud. `tycoon schedule add <name> --command "data run-all" --at 04:00 [--cadence daily|hourly|weekly] [--notify]`, plus `list` / `remove` / `status` (which tails the run log). Output lands in `~/.local/share/tycoon/schedule/<name>/run.log`; scheduled runs execute from the project root so `tycoon.yml` resolves. `tycoon doctor` reports the count of installed schedules. Windows prints a Task Scheduler pointer (unsupported in v1). Pairs with `--notify` ([#46][]) so unattended runs report in.
+- **Pipeline notifications — `tycoon notify` + `run-all --notify`** ([#46][]). A small local-first notification surface so unattended runs don't fail silently. `tycoon notify <success|error|info> <message> [-f key=val ...]` posts to a webhook read from `$TYCOON_NOTIFY_WEBHOOK_URL` (Slack incoming webhooks get a colour-coded attachment; any other URL gets a generic JSON envelope). `tycoon data run-all --notify` emits `success` on completion and `error` (with the failing stage) on failure — best-effort, so a notification problem never fails the pipeline. Non-secret prefs (which severities to emit, a source label) live in an optional `notify:` block in `tycoon.yml`; the URL never does. Webhook + Slack only for v1.
+- **DuckDB Quack — live multi-client warehouse, folded into `tycoon start`** ([#42][]). When the [Quack](https://duckdb.org/quack/) extension is available (currently `core_nightly`), `tycoon start` also serves the warehouse over Quack's local RPC protocol on `:9494`, turning the single-writer DuckDB file into a multi-client server. No new commands or flags: a per-project token is generated once into `.tycoon/secrets.yml` (gitignored), and `tycoon data query` attaches over Quack automatically when the server is up (labelled `warehouse (Quack)`), falling back to direct file access otherwise. Skipped silently where the extension can't load. Opt-in by nature (core_nightly-only) and local-only. Coordinating standalone dbt writes against a served warehouse is a planned follow-up.
+- **`tycoon setup` builds a managed project-local `.venv`** ([#57][]). The second piece of the managed-environment work: `tycoon setup` creates a `.venv` beside `tycoon.yml` on a supported interpreter via `uv venv --python` (uv auto-downloads a python-build-standalone CPython if the machine only has an unsupported one — zero manual installs), pins it with `.python-version`, and installs tycoon + its dbt/dlt/duckdb stack into it. Flags: `--python` (default 3.13), `--from` (install spec; `-e .` for dev checkouts), `--no-install`, `--force`, `--no-prompt`. If `uv` is absent it points at the standalone installer rather than running it. **`tycoon doctor --fix`** runs the same build flow when the interpreter check fails. Together with the v0.1.9 interpreter check, this closes the Python-version trap that produced [#55][].
+- **Google Sheets data source** ([#52][]). `tycoon data sources add google_sheets` registers a spreadsheet as a first-class source — pulling tabs/ranges into DuckDB with the header row becoming typed columns. Auth is a Google service-account JSON key (headless/cron-friendly), defaulting to the standard `${GOOGLE_APPLICATION_CREDENTIALS}` env var; leaving the key blank falls back to dlt's own resolution (the OAuth / ADC path). Config is `spreadsheet_url_or_id` (required) and an optional comma-separated `range_names` (blank loads every sheet). Works under `--no-prompt` for scripted bootstrap. Per project convention the dlt `google_sheets` verified source isn't bundled — it's pulled on demand via `dlt init`. First cut is full-refresh replace per run (Sheets has no native incremental key).
+- **`tycoon doctor` checks the Python interpreter version** ([#57][]). A new first check verifies the running interpreter is within tycoon's supported range (`>=3.12,<3.14`) and fails with an actionable hint otherwise — calling out that 3.14 has no dbt wheels yet and pointing at `uv venv --python 3.13`. tycoon runs dbt out of the same interpreter it lives in, so an out-of-range interpreter previously failed far from its cause at `data transform run` (this is how [#55][] stayed invisible). First sub-piece of the managed-`.venv` onboarding work; runs even without a `tycoon.yml`.
+
+### Security
+
+- **Fivetran credentials can no longer leak into a committed `tycoon.yml`** ([#60][]). `stack.ingestion_metadata.api_key` / `api_secret` are now `pydantic.SecretStr`, so they mask to `**********` in any `repr`/traceback, and their field docs steer you to `${FIVETRAN_API_SECRET}`-style env-var indirection (the loader already expands `${VAR}` before validation, keeping the literal secret out of the file). `save_project` now preserves the hand-authored `ingestion_metadata` block verbatim on round-trip — previously a `sources add` / `register` write would re-dump the *expanded* secret straight back into `tycoon.yml`. The scaffolded `.gitignore` also now excludes `.env`, `.dlt/secrets.toml`, and `**/profiles.yml` (the files that actually hold credentials); `tycoon.yml` stays committable as the shareable stack template.
+- **SQL identifier injection in Rill export and `data schema` is closed** ([#61][]). Schema/table/column names sourced from `tycoon.yml` or introspected from a DuckDB file are now quoted via a shared `quote_identifier` helper before interpolation into SQL. Previously an attacker-crafted `schema:` value in a shared project could break out of the `COPY … TO` statement that `tycoon data analyze --rill` runs — and DuckDB's `COPY`/`INSTALL` can write files and load extensions, so this was a path to arbitrary file write, not just data disclosure. The export's destination-path literal is escaped too. (The broader `tycoon.yml` identifier/path validation layer is tracked in [#65][].)
+
+[#31]: https://github.com/Database-Tycoon/tycoon-cli/issues/31
+[#42]: https://github.com/Database-Tycoon/tycoon-cli/issues/42
+[#46]: https://github.com/Database-Tycoon/tycoon-cli/issues/46
+[#48]: https://github.com/Database-Tycoon/tycoon-cli/issues/48
+[#52]: https://github.com/Database-Tycoon/tycoon-cli/issues/52
+[#55]: https://github.com/Database-Tycoon/tycoon-cli/issues/55
+[#57]: https://github.com/Database-Tycoon/tycoon-cli/issues/57
+[#60]: https://github.com/Database-Tycoon/tycoon-cli/issues/60
+[#61]: https://github.com/Database-Tycoon/tycoon-cli/issues/61
+[#65]: https://github.com/Database-Tycoon/tycoon-cli/issues/65
+
 ## [0.1.8] - 2026-05-30
 
 _A small, focused release: live Fivetran status in `tycoon data status` ([#50][]) plus a Python-version guardrail ([#55][]) so dbt-incompatible interpreters (3.14) aren't silently selected. The Quack headline and the backup / operate-in-prod tracks planned in [`docs/proposals/v0.1.8-scope.md`](docs/proposals/v0.1.8-scope.md) continue toward a later cut. See [`docs/releases/v0.1.8.md`](docs/releases/v0.1.8.md) for the full narrative._

docs/commands/data/query.md

@@ -42,6 +42,10 @@ tycoon data query --db .tycoon/metadata.duckdb \
 tycoon data query --db ./snap.duckdb "SHOW ALL TABLES"
 ```
 
+### Live warehouse via Quack (v0.1.9)
+
+When a [Quack](../start.md#quack-the-live-multi-client-warehouse-v019) server is holding the warehouse — i.e. `tycoon start` is running — a plain warehouse query (no `--db` / `--source` / `--raw`) **attaches over Quack** instead of opening the file. You'll see the result labelled `warehouse (Quack)`. This is what lets you query the live warehouse while the rest of the stack is up, rather than hitting DuckDB's single-writer file lock. When no server is running, the command opens the file directly as before. It's automatic — there's no flag.
+
 `--source` is a convenience for the common case of "show me the raw landed data". It walks tycoon's path resolution:
 
 1. `config.raw_db` if it contains the `raw_<name>` schema (single-DB mode — current default)

docs/commands/data/run-all.md

@@ -66,8 +66,19 @@ tycoon data run-all --skip-dbt
 
 Every source ingest and the final `dbt build` are captured into `.tycoon/metadata.duckdb` exactly as if you'd run them individually. `tycoon data history` will show one entry per source plus one for the dbt build.
 
+## Notifications (`--notify`)
+
+For unattended runs, `--notify` posts a webhook notification on completion: `success` when the run finishes, `error` (with the failing stage and a message tail) if ingestion or `dbt build` fails. Set `$TYCOON_NOTIFY_WEBHOOK_URL` first; which severities fire is governed by the `notify.severities` block in `tycoon.yml`. The webhook is best-effort — a notification failure warns but never fails the run. See [`tycoon notify`](../notify.md) for setup and payload shapes.
+
+```bash
+export TYCOON_NOTIFY_WEBHOOK_URL="https://hooks.slack.com/services/..."
+tycoon data run-all --notify
+```
+
 ## Related
 
 - [`tycoon data sources run`](sources.md#run-ingest) — single-source ingest
 - [`tycoon data transform build`](transform.md#build-run-test-together) — dbt build alone
 - [`tycoon data history`](history.md) — review what `run-all` did
+- [`tycoon notify`](../notify.md) — the notification surface `--notify` uses
+- [`tycoon schedule`](../schedule.md) — run `run-all --notify` on a timer

docs/commands/data/sources.md

@@ -28,6 +28,7 @@ Output:
 │ github     │ Developer Tools │ Issues, PRs, commits     │ issues, ...        │
 │ slack      │ Communication   │ Channels, users, ...     │ channels, ...      │
 │ stripe     │ Payments        │ Customers, ...           │ customers, ...     │
+│ google_she…│ Spreadsheets    │ Tabs/ranges from a sheet │ sheets             │
 │ rest_api   │ Generic         │ Any REST API             │ pokemon, ...       │
 │ filesystem │ Generic         │ Local CSV / Parquet      │ files              │
 └────────────┴─────────────────┴──────────────────────────┴────────────────────┘
@@ -54,6 +55,7 @@ Then type-specific config:
 
 - **github** — repo owner + name, optional access token env var name
 - **slack** — workspace + bot token env var name
+- **google_sheets** — service-account key path, spreadsheet URL/ID, optional tab/range list
 - **rest_api** — base URL, dataset list, optional auth header
 - **filesystem** — directory path + glob pattern
 - **sql_database** — connection string + table list
@@ -100,6 +102,34 @@ Flag reference:
 
 Catalog credentials default to `${ENV_VAR}` references in both modes — set the env var separately. Catalog-source files (`dlt init`-style downloads) are not auto-installed under `--no-prompt`; run `tycoon data sources catalog install <type>` separately if you need them.
 
+### Google Sheets
+
+Google Sheets pulls one or more tabs/ranges from a spreadsheet into DuckDB. The header row becomes typed columns. First cut is **full-refresh replace** per run — Sheets has no native incremental key.
+
+**1. Get a service-account key (headless/cron-friendly).** Create a service account at [console.cloud.google.com/iam-admin/serviceaccounts](https://console.cloud.google.com/iam-admin/serviceaccounts), enable the **Google Sheets API** for the project, and download a JSON key. Then **share the spreadsheet** with the service-account's email (`...@....iam.gserviceaccount.com`) — read access is enough.
+
+**2. Point tycoon at the key.** The credential defaults to the standard `${GOOGLE_APPLICATION_CREDENTIALS}` env var:
+
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS=~/keys/sheets-sa.json
+```
+
+**3. Register the source.** Interactively (`tycoon data sources add google_sheets`) you'll be prompted for the key path, the spreadsheet URL/ID, and an optional tab/range list. Non-interactively:
+
+```bash
+tycoon data sources add google_sheets \
+  --name marketing-sheet \
+  --config 'spreadsheet_url_or_id=https://docs.google.com/spreadsheets/d/<ID>/edit' \
+  --config 'range_names=Sheet1,Q1 2026!A1:F' \
+  --no-prompt
+```
+
+- **`spreadsheet_url_or_id`** — the full share URL or just the ID. Required.
+- **`range_names`** — comma-separated tab names and/or A1 ranges. Leave blank to load **every** sheet in the spreadsheet.
+- **`credentials_path`** — defaults to `${GOOGLE_APPLICATION_CREDENTIALS}`. Leave it blank to fall back to dlt's own credential resolution (env / `secrets.toml`), which is where the **OAuth / Application Default Credentials** path lives if you'd rather not use a service account.
+
+**4. Ingest:** `tycoon data sources run marketing-sheet`. The first run downloads the dlt `google_sheets` verified source to `~/.tycoon/sources/` (or `tycoon data sources catalog install google_sheets` under `--no-prompt`), then loads into `raw_<name>` and auto-scaffolds a staging dbt model.
+
 ## `list` — list registered sources
 
 ```bash

docs/commands/doctor.md

@@ -6,31 +6,47 @@ Health check for the tycoon environment — `tycoon.yml` validity, dbt / Rill /
 
 ```
 tycoon doctor [OPTIONS]
+
+Options:
+  --fix       Attempt to repair fixable problems. Currently: when the Python
+              interpreter is out of range, build a project-local `.venv` on a
+              supported interpreter (via uv) for you to activate.
+  -h, --help  Show this message and exit.
 ```
 
-No flags. Just runs a series of checks and prints one panel per check.
+Runs a series of checks and prints one panel per check. With `--fix`, doctor also attempts to repair the problems it knows how to fix (see [below](#fixing-problems-fix)).
 
 ## What it checks
 
 The output is a sequence of `Checking ...` panels:
 
-### 1. `tycoon.yml`
+### 1. Python interpreter (v0.1.9)
+
+Confirms the interpreter running tycoon is within the supported range, **`>=3.12,<3.14`** (mirrors `requires-python` in `pyproject.toml`).
+
+- `OK Python 3.13 is in the supported range (>=3.12,<3.14).`
+- `ERROR Python 3.11 is too old. ... run tycoon setup (or uv venv --python 3.12).`
+- `ERROR Python 3.14 is too new for tycoon's dbt stack (dbt-core / dbt-duckdb have no 3.14 wheels yet) ... run tycoon setup (or uv venv --python 3.13).`
+
+This is the first check because tycoon runs dbt out of the *same* interpreter it lives in (it resolves dbt at `Path(sys.executable).parent / "dbt"`). A too-new interpreter — notably 3.14, which has no dbt wheels — otherwise fails far from its cause, at `tycoon data transform run`, which is exactly how [#55](https://github.com/Database-Tycoon/tycoon-cli/issues/55) stayed invisible. Surfacing the mismatch here makes it the first thing you see. Environment-level, so it runs even without a `tycoon.yml`. When this check fails, [`tycoon doctor --fix`](#fixing-problems-fix) (or [`tycoon setup`](setup.md)) builds a corrected `.venv`.
+
+### 2. `tycoon.yml`
 
 Confirms `tycoon.yml` exists in the current directory or a parent. Errors if not — with a hint to run `tycoon init`.
 
-### 2. dbt project
+### 3. dbt project
 
 If `stack.transformation = dbt`: confirms `dbt_project_dir` exists and contains `dbt_project.yml`. Reports a clean skip ("dbt: skipped by choice (stack.transformation = none)") if the user opted out of dbt.
 
-### 3. Rill project
+### 4. Rill project
 
 If `stack.bi = rill`: confirms `rill_dir` exists. Reports a skip if `bi: none` or a different BI tool is configured.
 
-### 4. Dagster
+### 5. Dagster
 
 If the `[dagster]` extra is installed: confirms the `dagster` binary is on `$PATH`. The orchestrator is optional — most projects don't need it.
 
-### 5. Warehouse auth
+### 6. Warehouse auth
 
 For DuckDB warehouses: nothing to check (no auth).
 
@@ -43,7 +59,7 @@ Reports `OK token (env)`, `OK OAuth (cached session)`, or `ERROR not configured`
 
 For Snowflake / BigQuery / Redshift: warehouse auth lives in dbt's `profiles.yml` and isn't tycoon's concern. `doctor` skips these.
 
-### 6. Layer coverage (v0.1.7)
+### 7. Layer coverage (v0.1.7)
 
 When `stack.transformation = dbt` and a compiled dbt manifest exists, doctor verifies that every registered source in `tycoon.yml` has at least one staging model:
 
@@ -54,7 +70,7 @@ Silently skipped when `transformation: none` or when the manifest hasn't been co
 
 See [layered architecture](../recipes/layered-architecture.md) for the underlying classification rules.
 
-### 7. Observability
+### 8. Observability
 
 Reports the state of `.tycoon/metadata.duckdb`:
 
@@ -64,6 +80,16 @@ Reports the state of `.tycoon/metadata.duckdb`:
 
 Useful when "my dashboards are empty" — usually it means observability hasn't fired yet.
 
+## Fixing problems (`--fix`)
+
+`tycoon doctor --fix` runs all the checks, then attempts to repair the ones it knows how to fix. Today that's the **Python interpreter** check:
+
+- The running interpreter can't be swapped under tycoon's own feet, so the repair is to build a project-local `.venv` on a supported interpreter (via `uv venv --python 3.13`) and pin it with `.python-version` — the same flow as [`tycoon setup`](setup.md). You then `source .venv/bin/activate` and re-run.
+- If `uv` isn't installed, `--fix` can't proceed and prints the one-line installer instead of running it.
+- When the interpreter is already in range, `--fix` is a no-op.
+
+`--fix` never lowers the bar: an unfixable check still reports its error.
+
 ## Exit codes
 
 `tycoon doctor` always exits **0**, even when checks fail. The output is informational; nothing else in tycoon depends on doctor having a clean run.