Skip to content

About

High-performance TypeScript gRPC ingester SDK for GreptimeDB

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

@greptime/ingester

npm CI License Node

Official TypeScript ingester SDK for GreptimeDB. gRPC row inserts, streaming inserts, and Arrow Flight bulk writes in one package.

Features

  • Three write modes on one Client: unary, streaming, and Arrow Flight bulk with LZ4 / ZSTD body compression
  • Stage-3 decorators (TS 5) for object mapping — no reflect-metadata
  • TLS (system / PEM / file), basic auth, gzip transport compression
  • Multi-endpoint failover: pluggable EndpointSelector (random / round-robin / health-aware outlier detection) with retry-time exclusion of failed peers
  • Configurable retry (aggressive / conservative) with full-jitter exponential backoff + AbortSignal
  • Dual ESM + CJS, strict TypeScript, Node.js ≥ 20

Install

pnpm add @greptime/ingester
# or: npm install @greptime/ingester
# or: yarn add @greptime/ingester

Quickstart

import { Client, DataType, Precision, Table } from '@greptime/ingester';

const client = new Client(Client.create('localhost:4001').withDatabase('public').build());

const table = Table.new('cpu_usage')
  .addTagColumn('host', DataType.String)
  .addFieldColumn('usage', DataType.Float64)
  .addTimestampColumn('ts', Precision.Millisecond)
  .addRow(['server-01', 75.3, Date.now()]);

const result = await client.write(table);
console.log(`inserted ${result.value} rows`);
await client.close();

Three write modes

Mode API When
Unary client.write(tables) <1k rows/s, mixed schemas, supports auto-create table
Streaming client.createStreamWriter() Sustained writes on a single connection
Bulk client.createBulkStreamWriter(schema) >10k rows/s, Arrow Flight DoPut with parallelism

Streaming

const stream = client.createStreamWriter();
for (const batch of batches) await stream.write(batch);
const { value } = await stream.finish();

The stream is not auto-retried — rebuild it on error.

Bulk

// Prerequisite: the table exists with this schema. One unary write auto-creates it.
await client.write(buildTable().addRow([...sampleRow]));
const schema = buildTable().schema();

const bulk = await client.createBulkStreamWriter(schema);
for (const batch of batches) {
  await bulk.writeRows({ kind: 'rows', rows: batch });
}
const { totalAffectedRows } = await bulk.finish();

For LZ4 / ZSTD body compression, pass { compression: BulkCompression.Lz4 }. See examples/05-bulk-compression-lz4.ts.

For fire-and-forget, use writeRowsAsync(batch) — returns the request id and lets you submit the next batch immediately. finish() waits for every in-flight call and throws BulkError if any rejected, or if any group settled as a failure and was never claimed via waitForResponse(id) — silent partial ingestion isn't possible. Unclaimed acks are capped at maxUnclaimedResponses (default 10_000, oldest-first eviction). Claim ids with waitForResponse(id) when you need per-batch affectedRows; otherwise writeRows(batch) does the claim for you.

Decorator API

Stage-3 decorators. Keep experimentalDecorators off in your tsconfig.

import { Client, DataType, Precision, field, tableName, tag, timestamp } from '@greptime/ingester';

@tableName('cpu_usage')
class CpuMetric {
  @tag(DataType.String) host!: string;
  @field(DataType.Float64) usage!: number;
  @timestamp({ precision: Precision.Millisecond }) ts!: number | Date;
}

await client.writeObject([
  Object.assign(new CpuMetric(), { host: 'a', usage: 1.5, ts: Date.now() }),
]);

JSON2 columns

DataType.Json2 writes GreptimeDB JSON2 columns (server 1.2.1+). It is supported on FIELD columns with unary and streaming writes; the bulk path throws ValueError.

const table = Table.new('json2_logs')
  .addFieldColumn('payload', DataType.Json2)
  .addTimestampColumn('ts', Precision.Millisecond)
  .addRow(['{"message":"hello","nested":{"items":[1,"two",null]}}', Date.now()])
  .addRow([{ message: 'from an object' }, Date.now() + 1])
  .addRow([null, Date.now() + 2]);

// JSON2 tables must be created with append_mode=true.
await client.write(table, { hints: { append_mode: 'true' } });
  • Strings are parsed as JSON text; other values are serialized with JSON.stringify first.
  • The top-level value must be an object or null. null, undefined, and the JSON text null write SQL NULL.
  • Numbers keep their JSON literal type: integers without . or an exponent are sent as exact int64/uint64 (including values beyond 2^53), everything else as float64.
  • Strings and keys with unpaired UTF-16 surrogates throw ValueError, because UTF-8 encoding cannot represent them.

Configuration

Client.create('host:port')
  .withDatabase('public')
  .withBasicAuth('user', 'pw')
  .withTls({ kind: 'system' })
  .withRetry({ mode: 'aggressive', maxAttempts: 3 })
  .build();

Full reference: docs/configuration.md.

Errors

All errors extend IngesterError. Non-retriable: ConfigError, SchemaError, ValueError, StateError, AbortedError. Retriable or case-by-case: TransportError (.grpcCode), ServerError (.statusCode), TimeoutError, BulkError.

Classify with isRetriable(err, 'aggressive' | 'conservative'). Default is aggressive: it retries runtime SDK errors broadly, while ServerError is classified by GreptimeDB status code. conservative narrows transport retry to transient gRPC codes.

Examples

File What
examples/01-simple-insert.ts Table builder → client.write
examples/02-insert-object-decorators.ts @tableName / @tag / @field / @timestamp + writeObject
examples/03-stream-insert.ts StreamWriter with 10k rows
examples/04-bulk-insert.ts Unary bootstrap → bulk 100k rows
examples/05-bulk-compression-lz4.ts LZ4 frame compression on the bulk path
examples/06-auth-and-tls.ts Basic auth + TLS config
examples/07-multi-endpoint-lb.ts Multiple endpoints, outlier detection + stream rebuild
examples/08-abort-and-retry.ts AbortSignal + conservative retry

Run any of them with pnpm example <name> after ./scripts/run-greptimedb.sh starts a local server.

Performance

Writing to GreptimeDB from Node.js? The bulk path is the fastest option by a wide margin. Median of 3 runs against a local GreptimeDB on an Apple M4 Max, 1M rows with the 4-tag / 5-field CPU schema, default SDK config, parallelism=8:

JS client batch=1000 batch=5000 Relative
@greptime/ingester (bulk) 789k r/s 758k r/s baseline
@opentelemetry/exporter-logs-otlp-proto 679k r/s 638k r/s 0.85×
@influxdata/influxdb-client 494k r/s 520k r/s 0.66×

Same schema, same data generator, same server, same Node.js runtime; each client is driven with its own default configuration. Arrow Flight ships the batch already-columnar so the server skips text/proto parsing and per-attribute column mapping.

On the 22-column log schema the bulk path reaches ~137k rows/s (2M rows, batch=5000). Unary and streaming numbers, the exact SDK-usage decisions behind each bench, and reproduction commands: docs/benchmarking.md.

Compatibility

  • CI-tested: Node.js 20.x and 22.x, full suite; integration tests against greptime/greptimedb:v1.2.1.
  • Node.js 20.x is the supported minimum.
  • Bun (latest) and Deno (2.x): CI-gated via a smoke-level integration test against a live GreptimeDB. Full unit suite runs on Node only.

See docs/divergences.md for where the TS SDK intentionally differs from the Rust / Go SDKs.

Endpoint selection & failover

With multiple endpoints, every unary call is routed through a pluggable EndpointSelector:

import { Client, roundRobinSelector, outlierDetectingSelector } from '@greptime/ingester';

const client = new Client(
  Client.create('host1:4001')
    .withEndpoints('host2:4001', 'host3:4001')
    // default is random; or roundRobinSelector(), or:
    .withEndpointSelector(outlierDetectingSelector({ consecutiveFailures: 5 }))
    .build(),
);
  • Random (default), round-robin, or outlier-detecting (ejects an endpoint after consecutive transport failures, re-admits it after an exponential back-off window).
  • Retry-time exclusion: within a single write()'s retry sequence, a peer that just failed is excluded so one dead endpoint can't burn the whole retry budget.
  • Only endpoint-level transport failures feed outlier detection — a server business error (e.g. RegionBusy, TableNotFound) never ejects a healthy frontend.
  • Streaming and bulk are not auto-retried. On a transport error the session is dead; "rebuild" is simply calling createStreamWriter() / createBulkStreamWriter() again — the selector re-picks a healthy peer. See examples/07-multi-endpoint-lb.ts.

Roadmap

  • Off-main-thread Arrow encoding (worker_threads pool) to close the TS↔Go throughput gap on wide-schema bulk — today rowsToArrowTable is ~99% of client CPU (see docs/benchmarking.md)
  • OpenTelemetry instrumentation of the SDK itself (write latency, retries, bulk/stream state as metrics + spans)
  • Browser build via gRPC-Web in a separate @greptime/ingester-web package (unary + streaming only; no bulk)

Links

License

Apache-2.0 — see LICENSE.

About

High-performance TypeScript gRPC ingester SDK for GreptimeDB

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages