[#11936] fix(spark-connector): reject sub-microsecond timestamp precision#11941
Draft
nevzheng wants to merge 2 commits into
Draft
[#11936] fix(spark-connector): reject sub-microsecond timestamp precision#11941nevzheng wants to merge 2 commits into
nevzheng wants to merge 2 commits into
Conversation
Map Iceberg V3's timestamp_ns / timestamptz_ns to Gravitino's existing
timestamp[_tz](9) instead of ExternalType("TIMESTAMP_NANO").
- FromIcebergType: TIMESTAMP_NANO -> TimestampType.with[out]TimeZone(9),
honoring shouldAdjustToUTC().
- ToIcebergType: precision 9 -> TimestampNanoType.with[out]Zone(); precision
6 / unset still map to the microsecond TimestampType; any other precision
is rejected (Iceberg cannot represent it).
The conversion is backend-agnostic: it produces TimestampNanoType for any
backend. Whether that persists depends on the backend's own Iceberg glue -
the REST/JDBC path handles it; Iceberg's HiveSchemaUtil has no nanosecond arm
yet, so a Hive-backed create throws from within Iceberg (not from Gravitino),
and it will start working with no change here once Iceberg adds that arm.
Tests:
- Unit (TestConvertUtil): both directions, both zones, a precision-6
regression, and the reject case for an unsupported precision.
- Integration (CatalogIcebergBaseIT): a REST-backed round-trip creating a
format-version-3 table with timestamp_ns / timestamptz_ns columns and
asserting the native round-trip plus the underlying Iceberg schema.
The Iceberg<->Gravitino mapping is bijective (precision 9 + the time-zone flag
fully determine the nanosecond type), so no distinct timestamp_ns token is
needed. No changes to the core type model, JSON serde, Python client, or
OpenAPI: timestamp(9) is already a first-class Gravitino type. Discussed in
apache#11936.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… precision Spark timestamps are microsecond precision (the nanosecond types in SPARK-57454 are preview-only and flag-gated), so the connector's toSparkType now rejects a Gravitino timestamp[_tz] with precision > 6 instead of silently truncating it to microseconds. This matters now that Iceberg V3 tables load nanosecond columns as timestamp[_tz](9). - SparkTypeConverter / SparkTypeConverter34: validate precision <= 6 in the zoned and TIMESTAMP_NTZ branches before mapping (3.5 inherits via the 3.4 converters). - Tests: precision unset/0/6 map; precision 7/8/9 throw, for both the zoned and TIMESTAMP_NTZ paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The Spark connector maps every Gravitino
timestamp[_tz]to Spark's microsecond timestamp regardlessof precision, so a
timestamp(9)is silently truncated to microseconds. Spark itself has noreleased nanosecond timestamp type — the nanosecond types in
SPARK-57454 are preview-only and flag-gated
(
spark.sql.timestampNanosTypes.enabled), and bothTimestampTypeandTimestampNTZTypearemicrosecond precision.
So there is no lossless target; the connector should reject a precision it cannot represent rather
than drop digits silently.
This adds a single precision guard (
precision() > 6is rejected) and routes every Spark typeconverter through it via a shared
convertTimestampseam, so it cannot be bypassed:SparkTypeConverter/SparkTypeConverter34(iceberg, paimon).SparkJdbcTypeConverter/SparkJdbcTypeConverter34(jdbc).SparkHiveTypeConverter/SparkHiveTypeConverter34(hive).Precision
6and below (and unset) are unchanged.Why are the changes needed?
Once #11938 lands, reading an Iceberg V3
timestamptz_nstable through the Spark connector wouldotherwise present a microsecond column and silently discard the sub-microsecond digits. Failing
loudly is the correct behavior until Spark ships a stable nanosecond type (at which point a
spark-4.xconverter can map it natively).Does this PR introduce any user-facing change?
Yes. The Spark connector now throws
UnsupportedOperationExceptionwhen asked to represent atimestamp[_tz]with precision greater than microsecond (> 6), instead of silently truncating it.Precision
6and below is unchanged.How was this patch tested?
timestamp[_tz]at precision 7/8/9 throws and precision unset/0/6 maps, acrossthe base, jdbc, and hive converters (zoned and
TIMESTAMP_NTZpaths), for Spark 3.3/3.4/3.5.