Skip to content

DOCS-DEPLOYING.md

Latest commit

 

History

History
442 lines (318 loc) · 13.7 KB

DOCS-DEPLOYING.md

File metadata and controls

442 lines (318 loc) · 13.7 KB
Raw

Deploying InfluxData Documentation

This guide covers deploying the docs-v2 site to staging and production environments, as well as LLM markdown generation.

Table of Contents

Staging Deployment

Staging deployments are manual and run locally with your AWS credentials.

Prerequisites

  1. AWS Credentials - Configure AWS CLI with appropriate permissions:

    aws configure

  2. s3deploy - Install the s3deploy binary:

    ./deploy/ci-install-s3deploy.sh

  3. Environment Variables - Set required variables:

    export STAGING_BUCKET="test2.docs.influxdata.com"
export AWS_REGION="us-east-1"
export STAGING_CF_DISTRIBUTION_ID="E1XXXXXXXXXX"  # Optional

Deploy to Staging

Use the staging deployment script:

yarn deploy:staging

Or run the script directly:

./scripts/deploy-staging.sh

What the Script Does

  1. Builds Hugo site with staging configuration (config/staging/hugo.yml)
  2. Generates LLM-friendly Markdown (yarn build:md)
  3. Uploads to S3 using s3deploy
  4. Invalidates CloudFront cache (if STAGING_CF_DISTRIBUTION_ID is set)

Optional Environment Variables

Skip specific steps for faster iteration:

# Skip Hugo build (use existing public/)
export SKIP_BUILD=true

# Skip markdown generation
export SKIP_MARKDOWN=true

# Build only (no S3 upload)
export SKIP_DEPLOY=true

Example: Test Markdown Generation Only

SKIP_DEPLOY=true ./scripts/deploy-staging.sh

Production Deployment

Production deployments are automatic via CircleCI when merging to master.

Workflow

  1. Build Job (.circleci/config.yml):

    • Installs dependencies
    • Builds Hugo site with production config
    • Generates LLM-friendly Markdown (yarn build:md)
    • Persists workspace for deploy job

  2. Deploy Job:

    • Attaches workspace
    • Uploads to S3 using s3deploy
    • Invalidates CloudFront cache
    • Posts success notification to Slack

Environment Variables (CircleCI)

Production deployment requires the following environment variables set in CircleCI:

  • BUCKET - Production S3 bucket name
  • REGION - AWS region
  • CF_DISTRIBUTION_ID - CloudFront distribution ID
  • SLACK_WEBHOOK_URL - Slack notification webhook

Trigger Production Deploy

git push origin master

CircleCI will automatically build and deploy.

LLM Markdown Generation

Both staging and production deployments generate LLM-friendly Markdown files at build time.

Output Files

The build generates two types of markdown files in public/:

  1. Single-page markdown (index.md)

    • Individual page content with frontmatter
    • Contains: title, description, URL, product, version, token estimate

  2. Section bundles (index.section.md)

    • Aggregated section with all child pages
    • Includes child page list in frontmatter
    • Optimized for LLM context windows

Generation Script

# Generate all markdown
yarn build:md

# Generate for specific path
node scripts/build-llm-markdown.js --path influxdb3/core/get-started

# Limit number of files (for testing)
node scripts/build-llm-markdown.js --limit 100

Configuration

Edit scripts/build-llm-markdown.js to adjust:

// Skip files smaller than this (Hugo alias redirects)
const MIN_HTML_SIZE_BYTES = 1024;

// Token estimation ratio
const CHARS_PER_TOKEN = 4;

// Concurrency (workers)
const CONCURRENCY = process.env.CI ? 10 : 20;

Performance

  • Speed: ~105 seconds for 5,000 pages + 500 sections
  • Memory: ~300MB peak (safe for 2GB CircleCI)
  • Rate: ~23 files/second with memory-bounded parallelism

Making Deployment Changes

During Initial Implementation

If making changes that affect yarn build commands or .circleci/config.yml:

  1. Read the surrounding context in the CI file
  2. Notice flags, such as --destination workspace/public on the Hugo build
  3. Ask: "Does the build script need to know about environment details--for example, do paths differ between production and staging?"

Recommended Prompt for Future Similar Work

"This script will run in CI. Let me read the CI configuration to understand the build environment and directory structure before finalizing the implementation."

Summary of Recommendations

Strategy Implementation Effort
Read CI config before implementing Process/habit change Low
Test on feature branch first Push and watch CI before merging Low
Add CI validation step Add file count check in config.yml Low

Testing and Validation

Local Testing

Test markdown generation locally before deploying:

# Prerequisites
yarn install
yarn build:ts
npx hugo --quiet

# Generate markdown for testing
yarn build:md

# Generate markdown for specific path
node scripts/build-llm-markdown.js --path influxdb3/core/get-started --limit 10

# Run validation tests
node cypress/support/run-e2e-specs.js \
  --spec "cypress/e2e/content/markdown-content-validation.cy.js"

Validation Checks

The Cypress tests validate:

  • ✅ No raw Hugo shortcodes ({{< >}} or {{% %}})
  • ✅ No HTML comments
  • ✅ Proper YAML frontmatter with required fields
  • ✅ UI elements removed (feedback forms, navigation)
  • ✅ GitHub-style callouts (Note, Warning, etc.)
  • ✅ Properly formatted tables, lists, and code blocks
  • ✅ Product context metadata
  • ✅ Clean link formatting

See DOCS-TESTING.md for comprehensive testing documentation.

PR Preview

Deploys a full build of docs-v2 pull requests to the staging bucket (https://test2.docs.influxdata.com) for manual validation, under pr-preview/pr-{number}/.

The preview builds with --environment production, the same Hugo environment production uses, so it faithfully represents what production will serve: fingerprinted/SRI JavaScript, minification, real GA4/GTM/Marketo tracking, and the same __tests__/test_only content exclusions. This matters for validating site-wide changes (analytics, layout, JS) against a close approximation of production, not a different build configuration.

CI checks that need __tests__ fixtures or test_only content (Cypress, render checks) build their own copy of the site in their own job with the testing/development environment -- they never consume this deployed preview, so excluding those fixtures from the preview build doesn't affect them.

Workflow Files

File Purpose
.github/scripts/parse-pr-urls.js Extract docs URLs from PR description
.github/scripts/detect-preview-pages.js Generate deep links for the sticky PR comment
.github/scripts/preview-comment.js Manage sticky PR comments
.github/workflows/pr-preview.yml Main preview workflow
.github/workflows/cleanup-stale-previews.yml Weekly orphan cleanup

Key Design Decisions

  1. Full-site deployment - Every push deploys the complete built site, not a hand-picked subset. Previously, previews deployed only pages selected by a change-detection script; that pruning step was the root cause of a recurring bug class (missing pages, broken shared-content links, blank pages when a page's dependencies weren't copied). Deploying the full build removes that failure mode structurally.
  2. Production-parity build - hugo-environment: production (not a dedicated pr-preview environment), so the preview matches what ships.
  3. S3 prefix, not GitHub Pages - Deploys to the existing staging bucket under pr-preview/pr-{number}/, not the gh-pages branch. A full build is too large to deploy to gh-pages repeatedly -- the branch retains every push's payload in git history forever, since cleanup only removes the working tree, not history. S3 prefixes are deleted for real.
  4. Isolated from manual staging deploys - Manual staging deploys (scripts/deploy-staging.sh) go to the bucket root and pass -ignore='^pr-preview/' so they can never delete a live preview. PR preview deploys are scoped to their own pr-preview/pr-{number}/ prefix (both by the -path flag and by IAM policy), so they can never touch the bucket root or another PR's preview.
  5. Security hardening - XSS protection, path traversal prevention, input validation in the PR-description URL parser.
  6. Comment deep links, not a deploy gate - detect-preview-pages.js still maps changed content files and PR-description URLs to a page list, but that list only becomes clickable links in the sticky comment. It no longer decides what gets deployed -- the whole site deploys regardless, so there's no "needs author input" state for layout-only changes.

AWS Setup (one-time, by an AWS admin)

The preview and cleanup workflows authenticate via GitHub OIDC (no long-lived AWS keys in Actions). This requires, before the workflow can run:

  1. A GitHub OIDC provider trusted by the AWS account (if not already present for this org).
  2. An IAM role assumable by this repo's pull_request and refs/heads/master workflows, scoped to the pr-preview/* object prefix in the staging bucket (s3:GetObject/PutObject/DeleteObject, s3:ListBucket conditioned on that prefix, s3:GetBucketLocation, and cloudfront:CreateInvalidation on the staging distribution only).
  3. Repo secrets/variables: AWS_PR_PREVIEW_ROLE_ARN, STAGING_BUCKET, STAGING_CF_DISTRIBUTION_ID, AWS_REGION.
  4. The updated deploy/edge.js (see its top-of-file comments) deployed to the staging distribution's Lambda@Edge association -- this fixes two bugs that would otherwise break previews: a trailing-slash redirect hardcoded to docs.influxdata.com, and unanchored archive/version redirects that would hijack /pr-preview/pr-{number}/... subpaths.
  5. Optionally, an S3 lifecycle rule on the pr-preview/ prefix (e.g. expire after 60 days) as a backstop if the close/cleanup workflows ever fail to remove a preview.

Known parity gaps

A few things production generates that the preview build still skips, same as before this change: AI discovery artifacts, llms-full.txt, and the Rust markdown converter (build-rust: false by default). The preview's HTML also advertises Markdown-twin alternates that 404, since .md twin generation isn't run for previews. These don't affect the fidelity that matters most (rendered HTML, JS, analytics), but are worth knowing about when comparing a preview pixel-for-pixel against production.

Troubleshooting

s3deploy Not Found

Install the s3deploy binary:

./deploy/ci-install-s3deploy.sh

Verify installation:

s3deploy -version

Missing Environment Variables

Check required variables are set:

echo $STAGING_BUCKET
echo $AWS_REGION

Set them if missing:

export STAGING_BUCKET="test2.docs.influxdata.com"
export AWS_REGION="us-east-1"

AWS Permission Errors

Ensure your AWS credentials have the required permissions:

  • s3:PutObject - Upload files to S3
  • s3:DeleteObject - Delete old files from S3
  • cloudfront:CreateInvalidation - Invalidate cache

Check your AWS profile:

aws sts get-caller-identity

Hugo Build Fails

Check for:

  • Missing dependencies (yarn install)
  • TypeScript compilation errors (yarn build:ts)
  • Invalid Hugo configuration

Build Hugo separately to isolate the issue:

yarn hugo --environment staging

Markdown Generation Fails

Check for:

  • Hugo build completed successfully
  • TypeScript compiled (yarn build:ts)
  • Sufficient memory available

Test markdown generation separately:

yarn build:md --limit 10

CloudFront Cache Not Invalidating

If you see stale content after deployment:

  1. Check STAGING_CF_DISTRIBUTION_ID is set correctly
  2. Verify AWS credentials have cloudfront:CreateInvalidation permission
  3. Manual invalidation: 
    aws cloudfront create-invalidation \
  --distribution-id E1XXXXXXXXXX \
  --paths "/*"

Deployment Timing Out

For large deployments:

  1. Skip markdown generation if unchanged:

    SKIP_MARKDOWN=true ./scripts/deploy-staging.sh

  2. Use s3deploy's incremental upload:

    • s3deploy only uploads changed files
    • First deploy is slower, subsequent deploys are faster

  3. Check network speed:

    • Large uploads require good bandwidth
    • Consider deploying from an AWS region closer to the S3 bucket

Deployment Checklist

Before Deploying to Staging

  • Run tests locally (yarn lint)
  • Build Hugo successfully (yarn hugo --environment staging)
  • Generate markdown successfully (yarn build:md)
  • Set staging environment variables
  • Have AWS credentials configured

Before Merging to Master (Production)

  • Test on staging first
  • Verify LLM markdown quality
  • Check for broken links (yarn test:links)
  • Run code block tests (yarn test:codeblocks:all)
  • Review CircleCI configuration changes
  • Ensure all tests pass

Related Documentation