Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ cli = ["anyhow", "clap", "serde", "serde_json"]
# Enable opendal support.
opendal = ["dep:opendal"]

[[example]]
name = "opendal"
required-features = ["async", "opendal"]

[[bench]]
name = "arrow_reader"
harness = false
Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ providing APIs to read data into [Apache Arrow](https://arrow.apache.org) in-mem

See the [documentation](https://docs.rs/orc-rust/latest/orc_rust/) for examples on how to use this crate.

The [OpenDAL example](examples/opendal.rs) writes an ORC file to an in-memory service
and reads it back asynchronously using `AsyncOpendalReader`:

```shell
cargo run --example opendal --features opendal
```

## Supported features

This crate supports reading ORC files into Arrow arrays and provides initial support for writing
Expand Down
53 changes: 53 additions & 0 deletions examples/opendal.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

//! Read and write ORC with Apache OpenDAL using an in-memory service.
//!
//! Run with `cargo run --example opendal --features opendal`.
//! Replace the Memory service and enable the corresponding OpenDAL service feature
//! to use remote storage. Reads use the built-in `AsyncOpendalReader` for range I/O.
//! The synchronous ORC writer buffers this small file in memory before uploading it.

use std::sync::Arc;

use arrow::array::{ArrayRef, Int64Array, RecordBatch};
use futures::TryStreamExt;
use opendal::{services::Memory, Operator};
use orc_rust::{reader::AsyncOpendalReader, ArrowReaderBuilder, ArrowWriterBuilder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let operator = Operator::new(Memory::default())?.finish();
let path = "example.orc";
let col = Arc::new(Int64Array::from(vec![Some(1), None, Some(3)])) as ArrayRef;
let batch = RecordBatch::try_from_iter([("col", col)])?;

let mut buffer = Vec::new();
let mut writer = ArrowWriterBuilder::new(&mut buffer, batch.schema()).try_build()?;
writer.write(&batch)?;
// Finish the ORC footer before uploading the complete file.
writer.close()?;
operator.write(path, buffer).await?;

let reader = AsyncOpendalReader::new(operator, path);
let builder = ArrowReaderBuilder::try_new_async(reader).await?;
let read: Vec<RecordBatch> = builder.build_async().try_collect().await?;
assert_eq!(read, vec![batch]);
println!("read {} rows", read[0].num_rows());

Ok(())
}
Loading