Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,14 @@ simdutf8 = "0.1.5"
zmij = "1.0"

[dev-dependencies]
criterion = "0.5.1"
criterion = "0.8.2"
goldenfile = "1.8"
json-deserializer = "0.4.4"
json-five = "0.3.0"
mockalloc = "0.1.2"
proptest = "1.7"
serde_json = "1.0"
simd-json = "0.15.0"
simd-json = "0.19.0"

[features]
default = ["databend", "preserve_order", "arbitrary_precision"]
Expand All @@ -72,3 +72,19 @@ harness = false
[[bench]]
name = "strip_nulls"
harness = false

[[bench]]
name = "read"
harness = false

[[bench]]
name = "write"
harness = false

[[bench]]
name = "array"
harness = false

[[bench]]
name = "json_path"
harness = false
105 changes: 105 additions & 0 deletions benches/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Benchmarks

The benches use [criterion](https://docs.rs/criterion). Run them from the
repository root.

| Bench | What it measures |
|---|---|
| `read` | Parsing JSON text into JSONB and reading JSONB: `parse_owned_jsonb`, `from_slice`, `to_value`, `to_string`, `from_raw_jsonb`, `object_each`, `get_by_name`, `select_by_path`, `contains`, `partial_cmp`, `convert_to_comparable`, `strip_nulls` |
| `write` | Building and modifying JSONB, and encoding other values as JSONB: `build_array`, `object_insert`, `array_insert`, `concat`, `delete_by_name`, `select_array_by_path($.**)`, `to_owned_jsonb(serde_json)`, `Value::to_vec`, `OwnedJsonb::try_from(jaq)` |
| `array` | `array_distinct`, `array_intersection`, `array_except` and `array_overlap`, each array taken against itself |
| `json_path` | `parse_json_path` on a simple path, filters with 1 to 256 `&&`/`||` operators, and a field name that contains `&&` |
| `parser` | Parsing JSON text with jsonb, serde_json, json-deserializer and simd-json |
| `get_path` | Looking up a value by path in jsonb and in `serde_json::Value` |
| `strip_nulls` | `strip_nulls` on JSONB, and the same through `Value` |

`read`, `write` and `array` run every operation on four documents:
`data/canada.json` (mostly numbers), `data/citm_catalog.json` (mostly
objects), `data/twitter.json` (mostly strings), and a 162-byte object defined
in `common/mod.rs`, which also holds the paths and keys each document is queried
with. `contains` skips canada: array containment is quadratic, and one call
takes about 36 s.

## Running

```sh
# Every bench. `read`, `write`, `array` and `json_path` alone take about
# 10 minutes together.
cargo bench

# One bench.
cargo bench --bench read

# Only the benchmarks whose ID matches a regular expression. IDs are
# `<document>/<operation>`, or `parse_json_path/<case>` for `json_path`.
cargo bench --bench read -- canada/to_string
cargo bench --bench read -- 'twitter/(to_value|to_string)'

# Run each benchmark once without measuring, to check that they all work.
cargo bench --bench read -- --test
```

Criterion prints each result and writes an HTML report to
`target/criterion/report/index.html`.

On the large documents an operation takes milliseconds, so those benchmarks
take 20 flat samples over 3 s after 1 s of warm-up. The small object and
`json_path` use criterion's defaults: 100 samples over 5 s after 3 s of
warm-up.

## Measuring with Databend's build settings

Databend builds jsonb with `opt-level = "s"`, `codegen-units = 1` and
`lto = "thin"`, which can rank changes differently from the default bench
profile (`opt-level = 3`). To measure what Databend runs, set the same values
for the bench profile:

```sh
CARGO_PROFILE_BENCH_OPT_LEVEL=s \
CARGO_PROFILE_BENCH_CODEGEN_UNITS=1 \
CARGO_PROFILE_BENCH_LTO=thin \
cargo bench --bench read
```

## Comparing two versions

Save the results of the old version as a named baseline, then run the new
version against it. Criterion reports the change of each benchmark and whether
it is significant.

```sh
cargo bench --bench read -- --save-baseline before # on the old version
cargo bench --bench read -- --baseline before # on the new version
```

To keep your working tree untouched, build the old version in a separate
worktree:

```sh
git worktree add ../jsonb-before <old-commit>
cp Cargo.lock ../jsonb-before/
export CRITERION_HOME="$PWD/target/criterion"
(cd ../jsonb-before && cargo bench --bench read -- --save-baseline before)
cargo bench --bench read -- --baseline before
git worktree remove ../jsonb-before
```

Keep these the same on both sides, or the comparison measures them instead of
the code:

- **Benches.** Both versions need the bench files and their `[[bench]]`
entries in `Cargo.toml`. For a commit older than these benches, copy
`benches/` and the entries over; older APIs may need small edits, for
example `Value::to_vec` returned `Vec<u8>` instead of `Result`.
- **`Cargo.lock`.** It is not committed, so each checkout resolves its own
dependency versions. Newer `itoa`, `zmij` or `fast-float2` releases alone
change number formatting and parsing by several percent.
- **Baseline directory.** Criterion keeps baselines in `$CRITERION_HOME`, or
else in `criterion/` under the Cargo target directory. Point both runs at
one directory, as `CRITERION_HOME` does above.
- **Build settings.** Use the same profile variables for both runs.

Results move by a few percent between runs, and a change elsewhere in the
crate can move a benchmark by several percent through code layout alone.
Rerun a benchmark before trusting a small change, and check large ones
against the other documents.
57 changes: 57 additions & 0 deletions benches/array.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
// Copyright 2023 Datafuse Labs.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//! The set-like array functions, each array taken against itself.
//! See `benches/README.md` for how to run and compare.

mod common;

use criterion::criterion_group;
use criterion::criterion_main;
use criterion::Criterion;
use jsonb::jsonpath::parse_json_path;
use jsonb::parse_owned_jsonb;
use jsonb::OwnedJsonb;

fn bench(c: &mut Criterion) {
for doc in common::docs() {
let array: OwnedJsonb = match doc.array_path {
Some(path) => {
let owned = parse_owned_jsonb(&doc.text).unwrap();
let path = parse_json_path(path.as_bytes()).unwrap();
owned.as_raw().select_value_by_path(&path).unwrap().unwrap()
}
None => common::SMALL_ARRAY.parse().unwrap(),
};
let raw = array.as_raw();

let mut group = common::group(c, doc.name);
group.bench_function("array_distinct", |b| {
b.iter(|| raw.array_distinct().unwrap())
});
group.bench_function("array_intersection", |b| {
b.iter(|| raw.array_intersection(&raw).unwrap())
});
group.bench_function("array_except", |b| {
b.iter(|| raw.array_except(&raw).unwrap())
});
group.bench_function("array_overlap", |b| {
b.iter(|| raw.array_overlap(&raw).unwrap())
});
group.finish();
}
}

criterion_group!(benches, bench);
criterion_main!(benches);
98 changes: 98 additions & 0 deletions benches/common/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
// Copyright 2023 Datafuse Labs.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//! Documents and settings shared by the `read`, `write` and `array` benches.
//! `benches/README.md` describes how to run and compare them.

#![allow(dead_code)]

use std::time::Duration;

use criterion::measurement::WallTime;
use criterion::BenchmarkGroup;
use criterion::Criterion;
use criterion::SamplingMode;

/// A small object with every scalar type, a nested object and an array.
pub const SMALL: &str = r#"{"id":12345,"name":"Alice","email":"alice@example.com","age":30,"active":true,"tags":["a","b","c"],"score":98.5,"addr":{"city":"Paris","zip":"75001"},"note":null}"#;

/// A small array of mixed types with duplicates, for the array functions.
pub const SMALL_ARRAY: &str = r#"[1,"a",true,null,{"k":1},[2,3],1,"a"]"#;

/// A JSON document with the paths and keys the benches look up in it.
pub struct Doc {
pub name: &'static str,
pub text: Vec<u8>,
/// A JSONPath that selects values deep in the document.
pub path: &'static str,
/// A top-level key of the object.
pub key: &'static str,
/// A JSONPath that selects a large array, for the array functions;
/// `None` uses [`SMALL_ARRAY`].
pub array_path: Option<&'static str>,
}

/// The three documents in `data/` and [`SMALL`]: canada is mostly numbers,
/// citm_catalog mostly objects, and twitter mostly strings.
pub fn docs() -> Vec<Doc> {
let read = |file: &str| {
let path = format!("{}/data/{file}.json", env!("CARGO_MANIFEST_DIR"));
std::fs::read(&path).unwrap_or_else(|err| panic!("{path}: {err}"))
};
vec![
Doc {
name: "canada",
text: read("canada"),
path: "$.features[0].geometry.coordinates[0][0]",
key: "type",
array_path: Some("$.features[0].geometry.coordinates[0]"),
},
Doc {
name: "citm_catalog",
text: read("citm_catalog"),
path: "$.performances[*].seatCategories[*].areas[*].areaId",
key: "events",
array_path: Some("$.performances"),
},
Doc {
name: "twitter",
text: read("twitter"),
path: "$.statuses[*].user.screen_name",
key: "statuses",
array_path: Some("$.statuses"),
},
Doc {
name: "small",
text: SMALL.as_bytes().to_vec(),
path: "$.addr.city",
key: "name",
array_path: None,
},
]
}

/// Returns a benchmark group named `name`. Operations on the large documents
/// take milliseconds, so those groups take fewer, flat samples to finish in a
/// few seconds per operation.
pub fn group<'a>(c: &'a mut Criterion, name: &str) -> BenchmarkGroup<'a, WallTime> {
let mut group = c.benchmark_group(name);
if name != "small" {
group
.sampling_mode(SamplingMode::Flat)
.sample_size(20)
.warm_up_time(Duration::from_secs(1))
.measurement_time(Duration::from_secs(3));
}
group
}
56 changes: 56 additions & 0 deletions benches/json_path.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Copyright 2023 Datafuse Labs.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//! `parse_json_path` on simple paths, filters with long `&&`/`||` chains,
//! and a name that contains `&&` but no operators.
//! See `benches/README.md` for how to run and compare.

use std::hint::black_box;

use criterion::criterion_group;
use criterion::criterion_main;
use criterion::Criterion;
use jsonb::jsonpath::parse_json_path;

/// A filter with `operators` logical operators joining `@.a == 1`.
fn chain(operators: usize, op: &str) -> String {
let conditions = vec!["@.a == 1"; operators + 1];
format!("$ ? ({})", conditions.join(&format!(" {op} ")))
}

fn bench(c: &mut Criterion) {
let paths = [
("simple", "$.store.book[0].title".to_string()),
(
"filter_1_op",
r#"$.store.book[*] ? (@.price < 10 && @.category == "fiction")"#.to_string(),
),
("filter_8_ops", chain(8, "||")),
("filter_64_ops", chain(64, "&&")),
("filter_256_ops", chain(256, "||")),
("name_with_&&", format!("$.{}a", "a&&".repeat(50))),
];
let mut group = c.benchmark_group("parse_json_path");
for (name, path) in &paths {
let bytes = path.as_bytes();
assert!(parse_json_path(bytes).is_ok(), "{name}: {path}");
group.bench_function(*name, |b| {
b.iter(|| parse_json_path(black_box(bytes)).unwrap())
});
}
group.finish();
}

criterion_group!(benches, bench);
criterion_main!(benches);
Loading
Loading