Skip to content

Add DynamoDB connector support (define_dynamodb_connection + dynamodb datasource block) #18

@gnzjgo

Description

@gnzjgo

Summary

The Tinybird DynamoDB connector is generally available through the Tinybird CLI (tb connection create dynamodb, TYPE dynamodb in .connection files, IMPORT_TABLE_ARN / IMPORT_EXPORT_BUCKET in .datasource files), but the Python SDK currently only exposes helpers for Kafka, S3, and GCS.

This forces SDK users to keep a mixed project (Python definitions + raw .connection / .datasource files) just for DynamoDB.

Docs page: https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb

Proposal

Add a define_dynamodb_connection helper and a dynamodb block on define_datasource, mirroring the Kafka / S3 / GCS shape.

Connection

from tinybird_sdk import define_dynamodb_connection, secret

my_ddb = define_dynamodb_connection("my_ddb", {
    "arn": secret("dynamodb_role_arn_my_ddb"),  # DYNAMODB_ARN
    "region": "us-east-1",                       # DYNAMODB_REGION
})

Datasource

from tinybird_sdk import define_datasource, engine, t
from .connections import my_ddb

orders = define_datasource("orders", {
    "schema": {
        "pk": t.string(),
        "sk": t.string(),
        "_record": t.string(),
        "_old_record": t.string().nullable(),
        "_timestamp": t.date_time64(3),
        "_event_name": t.string().low_cardinality(),
        "_is_deleted": t.uint8(),
    },
    "engine": engine.replacing_merge_tree({
        "sorting_key": ["pk", "sk"],
        "ver": "_timestamp",
        "is_deleted": "_is_deleted",
    }),
    "dynamodb": {
        "connection": my_ddb,
        "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/orders",  # IMPORT_TABLE_ARN
        "export_bucket": "my-orders-exports",                                  # IMPORT_EXPORT_BUCKET
    },
})

Notes:

  • The DynamoDB datasource must use ReplacingMergeTree; consider validating this in the SDK.
  • The columns _record, _old_record, _timestamp, _event_name, _is_deleted are mandatory system columns. Consider exposing them as a helper / preset so users don't have to redeclare them every time.
  • Once engine.replacing_merge_tree and the dynamodb block exist, the docs at https://www.tinybird.co/docs/forward/ingest-data/connectors/dynamodb can add Python SDK tabs alongside the Tinybird CLI ones.

Mirror of the matching request on the TypeScript SDK: tinybirdco/tinybird-sdk-typescript#164

Why

Today docs need an explicit callout that the DynamoDB connector is CLI-datafile-only, which breaks the "define everything in Python" workflow the SDK is built around.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions