Skip to content

s3.md

Latest commit

 

History

History
234 lines (186 loc) · 7.1 KB

s3.md

File metadata and controls

234 lines (186 loc) · 7.1 KB
Raw

S3 Object Storage

ScrapAI can automatically upload crawl results to S3-compatible object storage for backup, archiving, and sharing across systems.

Supported Providers

Works with any S3-compatible object storage:

  • Hetzner Object Storage ⭐ (Recommended) - European provider, excellent pricing, free egress
  • AWS S3 - Amazon's original object storage
  • DigitalOcean Spaces - Simple, developer-friendly
  • Wasabi - Hot cloud storage, cheaper than S3
  • Backblaze B2 - Cost-effective alternative
  • Cloudflare R2 - Zero egress fees
  • MinIO - Self-hosted S3-compatible storage
  • Any other S3-compatible provider

Setup

Add S3 credentials to .env:

# S3-Compatible Object Storage
# Uncomment and fill in your S3 details to enable automatic uploads
# S3_ACCESS_KEY=your_access_key_here
# S3_SECRET_KEY=your_secret_key_here
# S3_ENDPOINT=https://hel1.your-objectstorage.com
# S3_BUCKET=your-bucket-name

Uncomment the lines and fill in your provider credentials.

Provider-Specific Endpoints

Hetzner Object Storage ⭐ (Recommended):

# Get your endpoint from Hetzner Cloud Console → Object Storage
S3_ENDPOINT=https://fsn1.your-objectstorage.com  # Falkenstein, Germany
# or
S3_ENDPOINT=https://hel1.your-objectstorage.com  # Helsinki, Finland
# or
S3_ENDPOINT=https://nbg1.your-objectstorage.com  # Nuremberg, Germany

# Full example for Hetzner:
S3_ACCESS_KEY=your_hetzner_access_key
S3_SECRET_KEY=your_hetzner_secret_key
S3_ENDPOINT=https://fsn1.your-objectstorage.com
S3_BUCKET=scrapai-crawls

DigitalOcean Spaces:

S3_ENDPOINT=https://nyc3.digitaloceanspaces.com  # New York
# or
S3_ENDPOINT=https://sfo3.digitaloceanspaces.com  # San Francisco

Wasabi:

S3_ENDPOINT=https://s3.us-east-1.wasabisys.com   # US East
# or
S3_ENDPOINT=https://s3.eu-central-1.wasabisys.com # EU Central

AWS S3:

S3_ENDPOINT=https://s3.us-east-1.amazonaws.com   # US East
# Note: AWS S3 endpoints vary by region

Backblaze B2:

S3_ENDPOINT=https://s3.us-west-000.backblazeb2.com  # Check your B2 dashboard for correct endpoint

Cloudflare R2:

S3_ENDPOINT=https://<account_id>.r2.cloudflarestorage.com

How It Works

S3 upload is now supported in both regular crawls and Airflow workflows:

Regular Crawl Command

./scrapai crawl spider_name --project proj
  • ✅ Automatically uploads to S3 if credentials configured
  • ✅ Only for production crawls (no --limit flag)
  • ✅ Compresses files with gzip before upload (saves bandwidth & storage)
  • ✅ Keeps local copy after upload (S3 is backup, not replacement)
  • ✅ Gracefully skips if boto3 not installed

Airflow Workflows

  • Same S3 upload functionality
  • Runs in dedicated task after crawl completes
  • Does not block spider execution

Upload Process:

  1. Spider completes crawl successfully (exit code 0)
  2. Crawl results saved to DATA_DIR/<project>/<spider>/crawls/crawl_TIMESTAMP.jsonl
  3. If S3 configured and boto3 installed:
    • File compressed to .jsonl.gz (typically 70-90% smaller)
    • Uploaded to S3 bucket
    • Compressed file deleted, original kept locally
  4. S3 path: s3://<bucket>/<project>/<spider>/crawls/crawl_TIMESTAMP.jsonl.gz

Local Files: All crawl data remains in your local DATA_DIR even after S3 upload. S3 is for backup/archiving, not a replacement for local storage.

Configuration Check

Airflow DAGs log S3 status on startup:

S3 Upload: ENABLED

or

S3 Upload: DISABLED (credentials not found)

If disabled, check that all 4 environment variables are set in .env:

  • S3_ACCESS_KEY
  • S3_SECRET_KEY
  • S3_ENDPOINT
  • S3_BUCKET

When to Mention to Users

Recommend S3 setup when:

  • User asks about data backup or archiving
  • User needs to share crawl data across systems
  • User mentions cloud storage (S3, Hetzner, DigitalOcean Spaces, etc.)
  • User has large crawls and wants off-site storage
  • User is using Airflow workflows
  • User asks about data durability/redundancy

Use Cases

Backup & Disaster Recovery:

  • Automatic off-site backup of all crawl data
  • Protect against local disk failure
  • Easy restoration from S3 if needed

Data Sharing:

  • Share crawl results across multiple systems
  • Centralized data storage for team access
  • Export data from one system, import on another

Archiving:

  • Long-term storage of historical crawls
  • Cheaper than keeping on local SSD/NVMe
  • Compliance and audit requirements

Multi-Region Deployment:

  • Run crawlers in multiple regions
  • All data centralized in S3 bucket
  • Access from anywhere

Limitations

Current Limitations:

  • Upload happens after entire crawl completes (not streaming)
  • No selective upload (all production crawls uploaded if S3 enabled)
  • No automatic cleanup of old files in S3
  • Only uploads production crawls (crawls without --limit flag)

Cost Considerations

Storage Costs:

  • Hetzner: €0.005/GB/month ($0.005)
  • DigitalOcean Spaces: $0.02/GB/month (first 250GB free with droplet)
  • Wasabi: $0.0059/GB/month (minimum 1TB)
  • AWS S3: $0.023/GB/month (US East)
  • Backblaze B2: $0.005/GB/month
  • Cloudflare R2: $0.015/GB/month

Egress Costs (downloading from S3):

  • Hetzner: Free egress
  • DigitalOcean Spaces: Free up to 1TB/month
  • Wasabi: Free egress
  • AWS S3: $0.09/GB (expensive!)
  • Backblaze B2: $0.01/GB
  • Cloudflare R2: Zero egress fees (best for frequent downloads)

Tip: For frequent data downloads, use providers with free/cheap egress (Cloudflare R2, Hetzner, Wasabi).

Troubleshooting

S3 upload not happening

  1. Check .env has all 4 S3 variables set
  2. Verify credentials are correct (test with AWS CLI or s3cmd)
  3. Confirm bucket exists and is accessible
  4. Ensure crawl is production mode (no --limit flag)
  5. Check crawl completed successfully (exit code 0)
  6. For Airflow workflows, check DAG logs for errors

Upload fails with authentication error

  1. Double-check access key and secret key
  2. Verify endpoint URL is correct for your provider
  3. Ensure bucket exists in the same region as endpoint
  4. Check IAM permissions (AWS) or access policies (other providers)

Files not appearing in S3 bucket

  1. Check bucket name is correct
  2. Verify upload task completed successfully in Airflow logs
  3. Browse S3 bucket with web console or CLI
  4. Check S3 path: s3://<bucket>/<spider_name>/crawl_*.jsonl

Upload too slow

  1. Check network bandwidth to S3 provider
  2. Consider provider closer to your location
  3. Split large crawls into smaller batches
  4. Use provider with faster upload speeds

Future Enhancements

Planned features:

  • S3 upload for regular ./scrapai crawl ✅ Done
  • Compression before upload (gzip) ✅ Done
  • Streaming upload during crawl (not just at end)
  • Selective upload (upload only specific spiders)
  • Automatic cleanup of old S3 files
  • Upload progress reporting
  • S3 download command (restore from S3)
  • Multi-bucket support (different buckets per project)

See Also