Skip to content

fix: decode signed timestamp nanoseconds and truncate to the requested unit - #93

Open
ahed-jneed wants to merge 1 commit into
datafusion-contrib:mainfrom
ahed-jneed:upstream-signed-timestamp-nanos
Open

ahed-jneed wants to merge 1 commit into
datafusion-contrib:mainfrom
ahed-jneed:upstream-signed-timestamp-nanos

Conversation

@ahed-jneed

Copy link
Copy Markdown

Problem

The Apache ORC C++ writer (and so pyarrow) stores the negative fraction of a pre-1970 timestamp as is: 1969-07-20 20:17:40.5 is written as -14182939 seconds and -500000000 nanoseconds. decode cast the nanoseconds to u64, so undoing the trailing-zero encoding overflowed: debug builds panic, and release builds fail the whole file.

Separately, decode_timestamp refused any value with digits below the requested unit, so a single nanosecond-precision value failed an entire read with with_timestamp_precision(Microsecond).

Change

  • Decode the nanoseconds as signed, as the C++ reader does, and do the arithmetic in i128 so no input can overflow it.
  • Truncate toward zero to the requested unit, matching Arrow's cast between timestamp units. A value that overflows i64 still fails with DecodeTimestamp.

Adds unit tests for the negative-nanosecond case, truncation, and out-of-range values.

cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings (default, --no-default-features, --all-features) pass. cargo test --all-features --no-fail-fast passes except the 14 tests/bin CLI snapshot tests, which fail identically on main on Windows (path/line-ending differences in the expected output).

The Apache ORC C++ writer stores the negative fraction of a pre-1970
timestamp as is (pyarrow writes 1969-07-20 20:17:40.5 as -14182939 seconds
and -500000000 nanoseconds). decode cast the nanoseconds to u64, so undoing
the trailing-zero encoding overflowed: debug builds panicked and release
builds failed the whole file. Decode them as signed, as the C++ reader
does, and do the arithmetic in i128 so no input can overflow it.

decode_timestamp also refused any value with digits below the requested
unit, so a single nanosecond value failed a read at microsecond precision.
Truncate toward zero instead, matching Arrow's cast between timestamp
units; a value that overflows i64 still fails.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant