Make notebooks standalone

Author: kan-fu

.gitignore

@@ -160,5 +160,4 @@ cython_debug/
 .idea/
 
 # Custom
-/output
-/data
+output

CATALOG.md

@@ -0,0 +1,41 @@
+# Notebook Catalog
+
+## Workflows
+
+### workflows/vertical-profiling/
+
+- **[bacvp_ctd_up_profiles.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/vertical-profiling/bacvp_ctd_up_profiles.ipynb)**  
+  Demonstrates vertical profile plotting for potential density from CTD data.  
+  *Keywords:* gsw
+
+- **[bacvp_videocam_opencv.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/vertical-profiling/bacvp_videocam_opencv.ipynb)**  
+  Illustrates video processing and frame extraction from videocam sources.  
+  *Keywords:* opencv-python, video
+
+### workflows/transit-analysis/
+
+- **[distance_from_terminals.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/transit-analysis/distance_from_terminals.ipynb)**  
+  Computes and visualizes time series for ferry-to-terminal distance.  
+  *Keywords:* geopy, xarray
+
+- **[grid_transit.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/transit-analysis/grid_transit.ipynb)**  
+  Analyzes ferry transit grid patterns and routing.  
+  *Keywords:* xarray
+
+- **[system_sampling_state.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/transit-analysis/system_sampling_state.ipynb)**  
+  Examines sampling state and temporal coverage of ferry measurements.
+
+- **[transit_identifier.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/transit-analysis/transit_identifier.ipynb)**  
+  Identifies and classifies distinct transit events.
+
+- **[twsb_tsg_clean_no_gaps.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/workflows/transit-analysis/twsb_tsg_clean_no_gaps.ipynb)**  
+  Performs TSG data quality and gap-filling workflows.  
+  *Keywords:* xarray
+
+## Case Studies
+
+### case-studies/multi-source-analysis/
+
+- **[fraser_river_plume.ipynb](https://github.com/OceanNetworksCanada/python-community-notebooks/blob/main/case-studies/multi-source-analysis/fraser_river_plume.ipynb)**  
+  Analyzes the Fraser River plume using multiple oceanographic datasets.  
+  *Keywords:* hypercoast, xarray, cartopy, gsw, plume

CONTRIBUTING.md

@@ -0,0 +1,124 @@
+# Contributing
+
+Thank you for contributing to Python Community Notebooks.
+
+## Where To Place Content
+
+Place your work under the best matching top-level category:
+
+- `tutorials`: Entry-level learning notebooks.
+- `workflows`: Reusable, method-focused notebooks.
+- `case-studies`: End-to-end, question-driven analyses.
+- `data-pipelines`: Script-first acquisition/processing workflows with optional notebooks.
+- `shared`: Common helper code, schemas, and templates used across notebooks and scripts.
+
+## Colab-compatible notebooks
+
+We encourage contributors to create notebooks that are compatible with Google Colab whenever possible.
+
+- Include an **"Open in Colab"** badge in the first cell of the notebook. The markdown link is `[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OceanNetworksCanada/python-community-notebooks/blob/main/AAA/BBB/CCC.ipynb)`. 
+- Handle dependencies in the notebook.
+- Add the bootstrap cell for token initialization and sys.path update before importing libraries.
+- You can open/test notebooks in Colab using any of the following:
+    - Open the notebook in GitHub, then replace `github.com` with [githubtocolab.com](https://githubtocolab.com/) in the URL.
+    - Open [Google Colab](https://colab.research.google.com/), choose **GitHub**, and paste the notebook GitHub URL.
+    - Build a direct Colab URL in this format: `https://colab.research.google.com/github/OceanNetworksCanada/python-community-notebooks/blob/<branch-or-main>/AAA/BBB/CCC.ipynb`.
+    - Click the **Open in Colab** badge in the first cell and update the branch name in the url for non-main branches.
+
+## Dependency Management
+
+- Prefer local dependency files inside related notebooks/scripts. Common libraries like `matplotlib`, `numpy`, and `pandas` do not need to be listed per notebook, as they are included in the root requirements.txt and Google Colab. 
+- Non-versioned dependencies are easier to maintain and usually better for simple notebooks that rely on stable, common packages.
+- Versioned dependencies are better when reproducibility matters or when a notebook depends on fragile scientific/geospatial stacks.
+- Recommended default: avoid strict pinning for simple community notebooks, but pin versions when a notebook is known to break across package releases or when results must be reproducible.
+
+Recommended dependency initialization cell (**always** install onc library because Google Colab does not have it by default):
+
+```python
+!uv pip install -q onc xxx yyy
+```
+
+## Bootstrap
+### Token Initialization
+
+- Use `ONC_TOKEN` from Colab secrets (the "key" icon in the left sidebar) or local `.env`.
+- Do not hardcode API tokens in notebooks or scripts.
+
+### Helper Files In Colab
+
+If a notebook depends on helper scripts or shared modules in this repository, put them in the `shared` folder, and use a bootstrap cell in the notebooks. This keeps local runs clean while enabling imports in Colab.
+
+### Recommended bootstrap cell
+
+```python
+# Bootstrap cell
+import sys
+from pathlib import Path
+import os
+
+def in_colab() -> bool:
+    try:
+        import google.colab
+        return True
+    except ImportError:
+        return False
+
+def setup_repo_for_shared_imports():
+    """
+    Locate (or clone, in Colab) the notebook repository and add it to ``sys.path``.
+    Assume that only the root directory has the ``shared`` directory.
+
+    Notes:
+        - In Colab, this ensures the repository exists at
+          ``/content/python-community-notebooks`` by cloning it if needed.
+        - In local environments, this searches upward from the current working
+          directory for ``shared`` to identify the repo root.
+        - If a repo root is found, it is prepended to ``sys.path`` to enable
+          imports from the repository (e.g., ``shared`` modules).
+    """
+    repo_dir: Path | None = None
+
+    if in_colab():
+        repo_dir = Path('/content/python-community-notebooks')
+        if not repo_dir.exists():
+            !git clone --depth 1 https://github.com/OceanNetworksCanada/python-community-notebooks.git /content/python-community-notebooks
+    else:
+        cwd = Path.cwd().resolve()
+        for candidate in [cwd, *cwd.parents]:
+            if (candidate / 'shared').exists():
+                repo_dir = candidate
+                break
+
+    if repo_dir is not None and str(repo_dir) not in sys.path:
+        sys.path.insert(0, str(repo_dir))
+
+
+
+def init_token():   
+    """
+    Initialize the ONC_TOKEN environment variable.
+
+    Notes:
+        - If in Colab, add your ONC_TOKEN secrets by clicking the key icon on the left sidebar
+        - If in local, create .env file in the root directory, and add ONC_TOKEN=XXX in your .env file
+    """
+    if in_colab():
+        from google.colab import userdata
+        os.environ['ONC_TOKEN'] = userdata.get('ONC_TOKEN')
+    else:
+        from dotenv import load_dotenv
+        load_dotenv()
+
+init_token()
+setup_repo_for_shared_imports()
+```
+
+If helper files are not needed, remove `setup_repo_for_shared_imports()` definition and usage from the cell.
+
+## Catalog Updates
+
+When adding or moving notebooks, update [CATALOG.md](CATALOG.md):
+
+- Add a short description.
+- Add keywords (optional but recommended).
+

README.md

@@ -2,166 +2,61 @@
 
 ## Overview
 
-The **Python Community Notebooks** repository contains a curated collection of Jupyter notebooks demonstrating practical applications of the [onc](https://github.com/OceanNetworksCanada/api-python-client) Python client library. These notebooks showcase practices for data manipulation, analysis, and visualization using Ocean Networks Canada (ONC) datasets.
+This repository contains community-maintained notebooks and scripts demonstrating practical use of the [onc](https://github.com/OceanNetworksCanada/api-python-client) Python client and ONC datasets.
 
-This repository serves as a learning resource for users at all levels seeking to deepen their understanding of oceanographic data processing and analysis techniques.
+The repository is organized by analysis purpose (not by contributor).
 
 ### Disclaimer
 
-Community notebooks are provided as-is and are not regulated, endorsed, developed, or maintained by Ocean Networks Canada. All feature requests and bug reports must be directed to the original notebook author.
+Community content is provided as-is and is not regulated, endorsed, developed, or maintained by Ocean Networks Canada. Feature requests and bug reports should be directed to the original notebook or script author.
+
+## Repository Structure
+
+Folders are organized by purpose:
+
+- `tutorials/`: Entry-level guided notebooks for learning core ONC and analysis concepts.
+- `workflows/`: Reusable method notebooks. They teach repeatable techniques that can be applied to many datasets.
+- `case-studies/`: End-to-end, question-driven analyses focused on a specific oceanographic story.
+- `data-pipelines/`: Script-first data acquisition and processing workflows, with optional companion notebooks.
+- `shared/`: Common helper code, schemas, and templates used across notebooks and scripts.
+
+Notebook listings are maintained in [CATALOG.md](CATALOG.md).
 
 ## Quick Start
 
 ### Prerequisites
 
 - Python 3.10+
-- An ONC API token (see [Token Configuration](#token-configuration))
-- Virtual environment (recommended)
+- [uv](https://docs.astral.sh/uv/)
+- ONC API token
+- Google account (optional, for running notebooks in Google Colab)
 
 ### Setup
 
+For direct Google Colab usage (optional):
+
+- Some notebooks support running in Colab before any local setup.
+- Use the notebook links in [CATALOG.md](CATALOG.md) to open the notebook on GitHub, then click the **Open in Colab** badge in the first cell (if present).
+- In Colab, set `ONC_TOKEN` using **Secrets** (key icon in the left sidebar): add a secret named `ONC_TOKEN`, then run the notebook token initialization cell. Adding a secret in Google Colab is a one-time effort, but you might need to toggle the notebook access for each new notebook to grant the access.
+
 1. **Clone the repository**
    ```bash
    git clone https://github.com/OceanNetworksCanada/python-community-notebooks.git
    cd python-community-notebooks
    ```
 
-2. **Create and activate a virtual environment**
-   ```bash
-   python -m venv .venv
-   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
-   ```
-
-3. **Install dependencies**
-   ```bash
-   pip install -r XXX/requirements.txt # Adjust the path as needed
-   ```
+2. **Install common dependencies**
+    ```bash
+    uv pip install -r requirements.txt
+    ```
 
-4. **Configure your ONC token** (see [Token Configuration](#token-configuration))
-
-5. **Launch Jupyter in the terminal or using your favorite IDE** 
+3. **Create a `.env` file for your token**
    ```bash
-   jupyter notebook
+   echo "ONC_TOKEN=your_token_here" > .env
    ```
 
-## Notebook Catalog
-
-### IanTBlack
-
-#### barkley_canyon_vertical_profiler/
-
-- **bacvp_ctd_up_profiles.ipynb**  
-  Demonstrates the creation of vertical profile plots for sea water potential density using CTD data.  
-  *Keywords:* gsw
-
-- **bacvp_videocam_opencv.ipynb**  
-  Illustrates video processing and frame extraction from videocam sources.  
-  *Keywords:* opencv-python, video
-
-
-#### british_columbia_ferries/
-
-- **distance_from_terminals.ipynb**  
-  Computes and visualizes time series data for ferry-to-terminal distances using geospatial calculations.  
-  *Keywords:* geopy, xarray
-
-- **grid_transit.ipynb**  
-  Analyzes transit grid patterns and routing through ferry data.  
-  *Keywords:* xarray
-
-- **system_sampling_state.ipynb**  
-  Examines sampling state and temporal coverage of ferry system measurements.  
-  *Keywords:* 
-
-- **transit_identifier.ipynb**  
-  Identifies and classifies distinct transit events within ferry operational data.  
-  *Keywords:* 
-
-- **twsb_tsg_clean_no_gaps.ipynb**  
-  Performs data quality assurance and gap-filling for TSG (Thermosalinograph) measurements.  
-  *Keywords:* xarray
-
-
-#### multiple_datasets/
-
-- **fraser_river_plume.ipynb**  
-  Analyzes and visualizes the transport and evolution of the Fraser River plume using multi-source oceanographic data.  
-  *Keywords:* hypercoast, xarray, cartopy, gsw, plume
-
-## Guidelines and Best Practices
-
-The following guidelines are recommended for consistency across contributed notebooks. Individual contributors may maintain their own conventions; consult their notebook documentation for specific implementation details.
-
-### Token Configuration
-
-Authentication to the ONC API requires a personal token. For instructions on obtaining a token, refer to the [api-python-client documentation](https://github.com/OceanNetworksCanada/api-python-client#obtaining-a-token).
-
-#### Recommended: Environment Variable Storage
-
-Store your token as an environment variable rather than hardcoding it in notebooks. The `onc` library (version 2.6.0+) automatically reads the `ONC_TOKEN` environment variable when instantiating the `ONC` class.
-
-**Configuration Steps:**
-
-1. Create a `.env` file in the repository root:
-   ```
-   ONC_TOKEN=your_token_here
-   ```
-
-2. Add the initialization code to your notebook:
-   ```python
-   from dotenv import load_dotenv
-   from onc import ONC
-   
-   load_dotenv()
-   onc = ONC()  # Automatically uses ONC_TOKEN from environment
-   ```
-
-The `load_dotenv()` function searches for the `.env` file in the current directory and parent directories, allowing all notebooks to share a single configuration file.
-
-### Dependency Management
-
-#### Virtual Environments
-
-Since different notebooks may require conflicting library versions, **always use virtual environments** to isolate dependencies. You can also use [uv](https://docs.astral.sh/uv/pip/environments/) or IDEs like [VS Code](https://code.visualstudio.com/docs/python/environments) and [PyCharm](https://www.jetbrains.com/help/pycharm/creating-virtual-environment.html) to manage virtual environments.
-
-You may create multiple virtual environments for different notebook sets if needed.
-
-#### Requirements Files
-
-Each contributor should maintain a `requirements.txt` file documenting all external library dependencies. You may maintain:
-- A single `requirements.txt` covering all your notebooks
-- Multiple `requirements.txt` files for notebook-specific dependencies
-
-#### Installation in Notebooks
-
-Optionally include an initialization cell to install dependencies within a notebook:
-```python
-!pip install -r ./requirements.txt  # Adjust path as needed
-```
+4. **Open any notebook** in Jupyter or VS Code, and run the cells in the notebook.
 
 ## Contributing
 
-Contributions are welcome! When adding new notebooks, please follow these standards:
-
-### Notebook Structure
-
-- Include a descriptive title and markdown cells explaining the analysis objective
-- Optionally add a brief overview of the data sources and processing steps
-- Document all external dependencies in a `requirements.txt` file
-- Use clear variable names and include inline comments for complex calculations
-- Add your notebook description to this README under the appropriate category. Common dependency libraries like `numpy`, `pandas`, `matplotlib` and `onc` can be omitted in the keywords.
-
-### Code Organization
-
-Helper functions and shared utilities could be organized in separate Python modules. To import modules from parent directories within notebooks, use the standard `sys` library, or the notebooks magic command `%cd`:
-
-```python
-import sys
-sys.path.append('..')
-import module_name
-```
-
-```python
-%cd ..
-import module_name
-```
+Contribution guidelines are in [CONTRIBUTING.md](CONTRIBUTING.md).