Repository navigation
Conversation
Add tests/defects/v1_review_2026_10.rs: 113 #[ignore]d tests encoding correct behaviour for every finding of the v1-readiness review (name resolution, precedence, comparison rules, deep expressions, column naming, REAL rendering and summation, verbatim writeback, load validation, table specs, writeback edge cases, file-type rules, CLI flags, error format, REPL behaviour, scalar functions, transactions), plus 5 passing guard tests for behaviour the fixes must keep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every statement now goes through vm::parse::parse_sql: - SqawkDialect (src/vm/dialect.rs) is Hive's dialect, reporting Hive's TypeId so the parser's Hive checks still apply, with SQLite operator precedence: = IS IN LIKE BETWEEN share one level; < <= > >= bind tighter; then bitwise, + -, * / %, ||, and unary - + ~ tightest. IS is parsed SQLite style: the right side of IS [NOT] [DISTINCT FROM] is parsed at the = level, and x IS y / x IS NOT y are NULL-safe equality. - Compound SELECTs are rebuilt left-deep in source order (SQLite gives UNION/UNION ALL/INTERSECT/EXCEPT equal precedence, left to right); parenthesized operands stay intact. - An iterative walk enforces SQLite's limits before anything recurses over the tree: expression depth 1000 (parentheses add none), 500 compound terms. - Derived tables execute their parsed Query directly instead of rendering it to SQL and parsing it again. - sqawk runs on a 256 MiB stack via stacker::grow on the main thread (a second thread would make glibc malloc lock on every call), so a 999-deep expression works in debug builds. - O(n^2) compile work removed: || no longer clones its operands, and aggregate-binding lookup renders a node only when its variant matches a bound expression. Red list: 12 tests un-ignored (precedence, compound order, deep expressions). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sums
Comparison: every comparison (=, <, IS, IN list, IN subquery, CASE, NULLIF,
hash join, ORDER BY, MIN/MAX, windows) uses one rule. Values that both have
a numeric reading (numbers, or text numeric_text accepts) compare as
numbers, text against text included ('10' > '9'); numbers sort before other
text; other text compares by bytes. compare_values is the matching total
order (number before equal-valued text, equal texts by spelling), so
GROUP BY / DISTINCT still keep '1' and 1 apart. IN (SELECT ...) sets and
hash-join buckets key values by an exact numeric key (cmp_key), so they
agree with = ; the hash join's Wild class is gone (booleans are 0/1).
Reals: Display and text_of render a REAL as SQLite's CAST(x AS TEXT)
(%!.15g, half-up digits, .0 on whole values, Inf/-Inf). c_general moved to
arith and shared with printf %g. Load inference: a field is REAL only if it
equals that rendering, else verbatim TEXT.
SUM/TOTAL/AVG: Kahan-Babuska-Neumaier summation as SQLite (integer
overflow reported at finalize unless a real was added after it).
ROUND(x, -N) treats N as 0. INTERSECT keeps the left side's values.
Un-ignores 14 red tests (A3 comparison, A7 reals/ROUND/sums); updates 24
test expectations for integral-real ".0" / 15-digit rendering / ROUND.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly aggregates rejected
- NameCtx::lookup / resolve_qualified lose the single-source fallback that
let any qualifier (zz.k, or a table's real name after it was aliased)
resolve to the one table in FROM; t.* likewise matches only the name a
source is known under. Unknown qualifiers fail as in SQLite:
'no such column: zz.k', 'no such table: zz'.
- UPDATE / DELETE scope the target under its alias (UPDATE d AS x ...
WHERE x.k), so an unknown qualifier there fails before anything is
written under --write.
- A derived table reading an outer-query column now fails cleanly
('no such column: a.aid') instead of binding to an inner column.
- Schema qualifiers: main.t (case-insensitive) is accepted in every table
position and main.t.c / main.t.* in expressions; any other schema is
'no such table: foo.t' / 'no such column: foo.t.c'.
- An aggregate whose arguments read only outer-query columns is rejected
(SQLite would move it to the outer query); one that also reads an inner
column aggregates in the subquery as before.
- ColumnNotFound / TableNotFound render as SQLite's 'no such column: X' /
'no such table: X'. Five tests matched the old 'not found' text and now
match the new message; no existing test relied on the removed fallback.
Red tests un-ignored: unknown_qualifier_is_error,
unknown_qualifier_in_update_is_error, unknown_qualifier_star_is_error,
correlated_derived_table_is_error, unknown_schema_is_error,
unknown_schema_in_update_is_error, outer_only_aggregate_in_subquery_is_error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unaliased result columns are named as SQLite 3.45 names them: - an alias wins; - a column reference, qualified or not, is named by the column as the table declares it (u.name -> name, NAME -> name), and * / t.* expand to bare column names, duplicates included (id,name,id,uid); - any other expression is named by its SQL text exactly as written (COUNT(*), n + 1, upper(name)||1, a IS 5 -- formerly COUNT, expr, a IS NOT DISTINCT FROM 5). The text comes from the source: parse::SqlSource tokenizes the statement with byte offsets, and each SELECT's result list is located from its SELECT token and split at top-level commas (sqlparser's expression spans do not cover -x, CAST(...), (a), ...). Without a locatable source the name falls back to the expression's Display form. The source is threaded from execute_vm through execute_statements and derived-table materialization into the compiler. Derived tables and CREATE TABLE ... AS make their column names unique as SQLite does (id, id:1, id:2); in a derived table a column reference is named as written. DDL instruction specs escape ':' / '|' / '%' in names and paths, since column names may now contain them (a||b). Differential check: 236 queries (columns, qualified refs, aliases, *, functions, aggregates, windows, CASE, CAST, subqueries, literals, IS / IS [NOT] DISTINCT FROM, comments and newlines inside items, compounds, derived tables, CTAS) against SQLite cursor.description / PRAGMA table_info: 234 match, 0 mismatch (1 both-error, 1 unsupported function). Test expectations: 140 header-only changes across tests/golden, tests/defects, join, join_on, aggregate, distinct, comparison, math_functions, stdin, window and tsq/tests -- every failing test was checked to differ from its expectation in header lines only. Red tests un-ignored: qualified_column_header_is_bare_name, star_over_join_header_is_bare_names, expression_header_is_sql_text, count_star_header_is_sql_text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- File types: .tsv/.tab are tab-separated, every other file and stdin
comma, unless -F. The first line is always the header unless
--no-header/--tabledef; the headerless guess is gone. '#' lines are data
unless the new --comments flag (any file type), which skips them on load
and keeps them in place on writeback.
- Verbatim writeback: a loaded table keeps the file's bytes (one buffer
plus per-row offsets, RawRows in table.rs). Rows no statement changed
are written back exactly as read (quoting, ragged rows, invalid UTF-8,
mixed line endings); UPDATE keeps a row's bytes when every value is
identical, DELETE drops them with the row, INSERT adds rows without,
ALTER ADD COLUMN and TRUNCATE discard them. The BOM, blank lines and the
header are kept; a row written after an unterminated last line gets the
file's line ending first.
- Load errors: an unclosed quote ("unterminated quoted field starting at
line N"); empty or duplicate (case-insensitive) header names.
- Column names: one case-insensitive lookup (Table::column_index) for
SELECT, INSERT column lists and UPDATE SET; ALTER TABLE ADD COLUMN of an
existing column and CREATE TABLE with a repeated column are errors.
- Table specs: one parser (parse_table_spec); NAME=PATH only when NAME is
identifier-like; empty table names are errors. Tables store canonical
absolute paths, so REPL .cd + .save writes the loaded file and the same
file cannot be loaded twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lags
Every error is one line on stderr, 'sqawk: <message>', exit status 1
(clap usage errors keep exit 2). SqawkError messages are self-contained
and no variant exposes source(), so causes are never printed twice; no
SQL echo or 'Caused by' chain. File errors name the user's file
('cannot open', 'cannot write'), never the writeback temp file; load
errors read 'FILE: line N: MESSAGE'. Unknown functions report 'no such
function: F', aggregates outside an aggregate context 'misuse of
aggregate function F()' (COUNT(*) included), ambiguous columns
'ambiguous column name: c'. The REPL prints errors in the same form.
All stderr diagnostics (-v, errors) and REPL stdout lines go through
diag!/outln!, which ignore write errors: a closed stream never panics.
-v is documented as debugging output whose format is not stable.
New flags: -f/--file FILE (repeatable, '-' for stdin, ordered with -s),
--field-separator (long -F), --output-delimiter (default: -F, else
comma), --no-output-header. Separator values are validated by clap.
The anyhow dependency is gone from sqawk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Transactions: BEGIN [DEFERRED|IMMEDIATE|EXCLUSIVE] [TRANSACTION], COMMIT / END, ROLLBACK, handled in vm/mod.rs against Database. The undo state is copy-on-write (a table is copied on its first change inside the transaction, moved there when dropped, recorded absent when created), and the modified-table set now lives in Database, so ROLLBACK restores both. SQLite's errors for nested BEGIN and COMMIT/ROLLBACK without one. A transaction open at the end of the run or REPL session is rolled back before writeback (-v notes it). In the REPL, .write on saves only outside a transaction; .save and .load are refused inside one. SAVEPOINT, RELEASE, ROLLBACK TO, transaction modes, AND CHAIN, START TRANSACTION and BEGIN blocks are rejected. The unused TransactionState and Begin/Commit/ Rollback/SavePoint/Release opcodes are removed. Writeback: every target is checked before any file is written; a read-only file (mode bits, or refused open for append) is "cannot write 'x.csv': file is read-only", and the directory must take a temp file. Scratch tables (no LOCATION, CTAS) are skipped; a dropped table leaves the modified set. Hard links are split by the atomic rename; documented. Docs no longer claim only INSERT/UPDATE/DELETE write back. Silent ignores: check_create_table destructures CreateTable exhaustively; STORED AS other than TEXTFILE, WITH options other than DELIMITER, other option forms, comments, TEMP with LOCATION, and LOCATION/STORED AS/WITH or a column list on CTAS are errors. --tabledef without ':', with an empty table or column name, a duplicate column, a table defined twice, or a table no input file loads is an error. Red list: 16 tests un-ignored (transactions, savepoints, read-only refusal, scratch skip, DROP of a modified table, STORED AS, tabledef). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
It is a large v1-readiness change spanning SQL parser precedence, transactions, float formatting, and error/file-handling refactors—high-correctness-risk areas that warrant final human review.
0 open findings
What changed in this PR
This "Updates" PR advances sqawk toward v1 readiness by aligning its SQL semantics more closely with SQLite 3.45 and adding transaction support. The changes shown here are mostly test updates that encode the new behavior, plus a substantial new end-to-end test file for transactions and writeback rules. They accompany broader source changes (a custom SqawkDialect with SQLite operator precedence, BEGIN/COMMIT/ROLLBACK with copy-on-write tables, SQLite-compatible real-number formatting, compensated summation, and error/file-handling refactors).
Changes:
- SQLite-compatible output: real numbers now render with a decimal point (e.g.
1→1.0,0.285714285714286), and auto-generated column names use the written expression text and are unqualified for joins (s.name→name,COUNT→COUNT(*)). - New
tests/transaction/mod.rsexercises BEGIN/COMMIT/ROLLBACK over in-memory tables,--writeinteraction, REPL behavior, and writeback rules (scratch tables, DROP, read-only files, hard links,--tabledefvalidation). - Reworded unsupported-feature errors (
"Unsupported"→"not supported") and addedunsupportedcases for CREATE TABLE options and transaction sub-statements.
| File | Description |
|---|---|
tsq/tests/tsq.rs |
Updates expected header COUNT → COUNT(*) for new auto-naming. |
tests/window/mod.rs |
Adjusts expected output for real formatting and unqualified join column names. |
tests/unsupported/mod.rs |
Adds create_table_options and transaction_clauses rejection tests. |
tests/transaction/mod.rs |
New suite covering transactions, --write, REPL, and writeback rules. |
tests/subquery/exact.rs |
Updates expected error text to "not supported". |
tests/stdin/mod.rs |
Updates expected headers (COUNT(*), unqualified name). |
tests/mod.rs |
Registers new file_io and transaction modules; clarifies headerless comment. |
tests/math_functions/mod.rs |
Updates ROUND/real expectations and negative-precision doc to match SQLite. |
tests/join/planner.rs |
Updates expected error text to "not supported". |
tests/join/mod.rs |
Join test expectation updates (truncated in diff). |
Verification notes: the reworded error assertions match source strings (src/vm/compiler_check.rs:26, src/vm/compiler_from.rs:674, src/vm/compiler_subquery.rs:147); the real-formatting and column-naming test updates are consistent with Value::field_str/Display and arith::real_text; transaction/writeback behavior is backed by the copy-on-write model in database.rs and REPL guards in repl.rs. I did not find concrete, reportable defects in the shown diffs.
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The test crate failed to compile on Windows: helpers called std::os::unix::fs::PermissionsExt::set_mode and std::os::unix::fs::symlink unconditionally. Gate those helpers (perms_enforced, chmod) and the tests that use them with #[cfg(unix)]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Piped (non-tty) stdin: read line by line; stdout carries only command output (no banner, prompts, CTRL-C/D echoes or status messages; status goes to stderr under -v). "Query returned N rows" is gone in both modes: results print exactly as with -s. - .exit N sets the exit status (mod 256); a script with any SQL or dot-command error keeps running and exits 1. Dot-command failures (unknown command, .schema of a missing table, failed .cd, bad .stats option, bad .exit code) are now errors. - A final statement without ';' runs at end of input; statement end and blank/comment-only input are found outside quotes and comments. - -i with -s/-f is a usage error (exit 2). - History only for a tty: $XDG_STATE_HOME/sqawk/history, else ~/.sqawk_history, else none; never the current directory. - .tables PATTERN uses the LIKE matcher; drop the regex dependency. - Tab completion: dot-commands, table-name arguments, SQL keywords, function names, table and column names, table.column. - Ctrl-C abandons the current entry instead of leaving the REPL. - Remove the REPL-only ResultSet/execute_sql/rows_as_strings path. - Un-ignore the REPL red-list tests; user guide REPL section updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verified by running the whole suite as x86_64-pc-windows-gnu binaries under Wine (all pass) as well as natively on Linux. - error.rs tests used raw OS error 13, which is EACCES on Unix but ERROR_INVALID_DATA on Windows; use the platform's access-denied code and read the OS's own wording instead of spelling it out. - repl.rs history_location used "/state", which is relative on Windows; use platform-absolute paths. - REPL tests spawned `cargo run` from inside the test run (nested cargo, build-lock contention, no cargo under Wine); spawn the built binary via CARGO_BIN_EXE_sqawk. - transaction::run wrote stdin to a process that may already have exited after rejecting its arguments, failing intermittently with BrokenPipe; a BrokenPipe on that write is not a test failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record with too many fields at line 4 of a CRLF file is reported at line 3. Windows CI exposed it: its checkout converts tests/data to CRLF, so too_many_fields_is_an_error saw line 5 instead of 6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- printf.rs: a port of SQLite 3.45's sqlite3_str_vappendf as the SQL printf drives it: every conversion (d i u f e E g G x X o c s z q Q w p r n %), flags - + space 0 # , !, * width/precision, l/ll, real decoding to 16 (26 with !) digits half up, NULL for an empty result. Widths/precisions past the length limit stay an error. - QUOTE(x). - datetime.rs: a port of date.c: negative years, whitespace rules, subsec, auto, julianday, +-HH:MM, +-YYYY-MM-DD[ HH:MM...], every 3.45 strftime conversion (%e %F %I %k %l %p %P %R %T %u ...), %J via %.16g, TIMEDIFF, and localtime/utc through the C library (libc). - Hex integer literals 0x1F / 0X1F (64-bit two's complement, "hex literal too big", unrecognized token for 0x alone); X'..' is the error "blob literals are not supported". `~~` splits into two `~`. - Bitwise & | << >> ~ with SQLite semantics; << and >> now parse. - Infinite real literals (1e999) no longer become 0.0. - Red list: the 13 remaining v1-review tests pass and are un-ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csv-core ends a CRLF record at the CR and counts its LF only when the next record is read, so every line number taken from its record positions was one too low in CRLF files (too many fields, bad header). read_table now numbers lines from the bytes: the line of a record is one plus the line breaks before the record's first byte, where a break is LF, CRLF, or a CR not followed by LF (the terminators the csv crate accepts). The count resumes from the previous record, so it stays linear. The unterminated-quote error uses the same counter, so CR-only files number lines correctly there too. Skipped blank and # lines, the BOM and quoted line breaks all fall out of counting bytes. Tests: the red CRLF test is no longer ignored; unit cases cover mixed endings, CR-only, quoted LF/CRLF, blank lines, BOM, --comments lines, unclosed quotes and bad headers; an integration test runs the binary on LF and CRLF twins of malformed files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's Windows runner checks out with core.autocrlf=true, which turned tests/data/* into CRLF files, so the fixtures the tests read were not the bytes in the repo. Mark the fixtures -text: git never converts their line endings, so they are byte-identical on every checkout. Scope: tests/data/** (every fixture the tests read) and tests/*.csv (the other CSV data). Sources are left alone. A CRLF-converted tree was simulated (native and under wine with --target x86_64-pc-windows-gnu), and with every file, Rust sources included, converted to CRLF the suite passes. Rustc normalizes CRLF in source, so multi-line string literals are unaffected. Forcing '* text=auto eol=lf' would buy nothing and would override contributors' own autocrlf settings. Every file in the index is already LF, so renormalization changes no content. Before the previous commit, the one test that depended on fixture line endings was csv_handler::tests::too_many_fields_is_an_error (line 5 vs 6, the CRLF line-number bug). That bug is fixed in the loader. This file makes sure fixtures cannot drift by platform in future. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete the unused index-based result types in table.rs, the IfNeg, Sort and Sequence opcodes the compiler never emits (and the cursor sort-key / sequence state only they used), the unreachable catch-all opcode arm, and every #[allow(dead_code)] that hid them. SqlCompiler::compile is test-only and now says so with #[cfg(test)]. Comments state current fact: no roadmap notes, removed-method notes or development phase labels. Doc footers that described future versions are gone. The empty test_complex_where_expressions is now seven exact-output checks of AND/OR/NOT precedence, parentheses and NULL truth tables, with expected output computed by SQLite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Version 0.9.0 (tsq stays 0.1.0). - Drop the parser-implementations category: sqawk is a command-line tool, not a parser crate. - [profile.release]: codegen-units = 1 and strip = true alongside lto. The release binary goes from 9,161,800 to 5,433,808 bytes on x86_64 Linux; best-of-5 timings on 200k rows are unchanged or a hair faster (GROUP BY 0.21s -> 0.20s, hash join 0.28s -> 0.27s, ORDER BY LIMIT 0.48s -> 0.47s). The profiling profile keeps its symbols with strip = false. - README: install with `cargo install --locked sqawk`. MSRV 1.88 re-verified: cargo +1.88.0 check --locked --workspace --all-targets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every example command in README.md and doc/user_guide.md and every value claim in doc/sql_reference.md was run against fixtures; fixed what did not match: - user guide: the header rule (first line always the header), an INSERT example that needed a three-column table, a DISTINCT example naming a column the documented file lacks, CREATE TABLE types (accepted, not enforced), the TOC, and install with --locked. - Memory: a table takes roughly six to seven times its file size (83 MB / 1M rows -> 530 MB peak for COUNT(*); SELECT * 1.4 GB), plus the result set. Documented in the user guide, database.md and README; "select fewer columns to save memory" advice removed (the whole table is loaded regardless). - SQL reference: TRIM takes one argument (TRIM(s, chars) does not parse; LTRIM/RTRIM take two); the precedence table gains the bitwise level and unary ~; the parser's nesting limit (47 levels of parentheses or function calls, about half that for subqueries: "recursion limit exceeded") is stated; REGEXP is rejected rather than unparsed. - database.md: TOC, column lookup, derived-table registration, output delimiter, result sets are materialized, memory figures. - CLAUDE.md: test directory list. CHANGELOG.md (Keep a Changelog) records 0.9.0: breaking changes (file types and headers, # lines, NULL and real output, column names, the comparison rule, integer arithmetic, LIKE, precedence, ROUND, errors and exit status, unknown qualifiers and schemas, REPL output, exit codes and history, -i with -s, packaging), additions and fixes since 0.8.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…zers harness.py now loads each cell into SQLite as the value sqawk infers (empty -> NULL, canonical integer, real only if it equals SQLite's CAST(x AS TEXT), else verbatim text) into columns with no affinity, and renders SQLite's results as sqawk prints them (NULL empty, reals via CAST(x AS TEXT), RFC 4180 quoting compared byte for byte). Every result set of a multi-statement test is compared; a both_err counter is added; data goes to a temp dir; SQAWK overrides the binary. NULL and '' both print empty, which the header documents. New, all self-contained (repo-relative, temp dirs, SUMMARY line): - diffh.py + hand1-3.sql: adversarial tables, hand corpora and a seeded random query generator. - wb.py: --write round-trip fuzzer (untouched records byte-identical). - csvfuzz.py: malformed and huge inputs. - fuzz.py: garbage SQL, limit values, deep nesting. Results: harness.py pass 522 / DIFF 3 / sqawk_err 14 / sqlite_err 1 / both_err 7; diffh hand 359 / 13 / 15 / both 36; random seeds 1-3 pass 676 / 684 / 682 of 700; wb.py 300/300 ok; csvfuzz crash 0, writeback_bad 0; fuzz findings 0. Every remaining DIFF and sqawk_err is a documented difference or unsupported feature; sql_reference.md gains the ones that were missing (SUM/AVG over text, bare columns in aggregate queries, integer ORDER BY / GROUP BY terms, x IN (), COLLATE, the RIGHT/FULL JOIN limits, INTERSECT/EXCEPT ALL). CLAUDE.md says how to run each tester and what its SUMMARY must show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ORDER BY +1` returned rows unsorted (SQLite: position 1); `ORDER BY -1` was accepted; `GROUP BY -1` gave a misleading error; integers beyond 32 bits and real literals errored instead of being constant terms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deep-expression red tests sent 20-70 KB statements with -s, past
Windows' 32,767-character command line limit ("The filename or extension
is too long", os error 206), so the process never started on the Windows
runner. run_one now feeds the SQL through `-f -` on stdin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A term is a result-column position exactly when SQLite's
sqlite3ExprIsInteger says so: an integer literal fitting in 32 bits, under
any parentheses and unary +/- (`+1`, `(1)`, `- -1`, `0x2`). It must lie in
1..=N, else "Nth ORDER BY term out of range - should be between 1 and N"
(GROUP BY likewise). Any other term (`1.0`, `4294967297`, `1+0`, `'1'`,
`-2147483648`) is an ordinary expression. In a compound such a term must
name a result column ("Nth ORDER BY term does not match any column in the
result set"). Window ORDER BY integers stay constants.
Un-ignores the four red tests; adds exact-output tests for simple, grouped,
compound, DISTINCT, subquery and window cases (SQLite 3.45 expectations).
Drops the "Differences from SQLite" entry in favour of the SQLite rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also drop the links to it from README.md and doc/user_guide.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…types CREATE TABLE ... LOCATION 'x.tsv' writes comma-separated text that does not load back as written, and declared column types other than the handful sqawk recognizes are recorded as TEXT. These #[ignore]d tests encode the correct behaviour, with controls for an explicit WITH (DELIMITER=...), .csv files, and declared types not being enforced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE TABLE ... LOCATION without WITH (DELIMITER=...) now writes a .tsv or .tab file tab-separated, as such a file is loaded, so it reads back as written; an explicit delimiter still wins. A declared column type (CREATE TABLE, ALTER TABLE ADD COLUMN) is recorded as written and shown so by .schema, instead of collapsing every name but INT/REAL/FLOAT/DOUBLE/BOOL to TEXT. Its affinity follows SQLite's rules (INT -> INTEGER; CHAR/CLOB/TEXT -> TEXT; BLOB -> BLOB; REAL/FLOA/DOUB -> REAL; else NUMERIC) and is what CREATE TABLE ... AS reports for a column of a declared table. Types are still not enforced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A FIFO queue is breadth-first: it holds one generation of rows, each followed by its children in order. When every recursive arm maps each working row independently (no DISTINCT, no wildcard), the whole generation is now the working table at once, each row tagged with its position; the children are put back in queue order by (position, arm) and UNION dedupes in that order -- the same rows in the same order as one run per queue row, which an ordered queue and the other arms still use. A 10,000-node tree walk drops from 7.2 s to 0.03 s (each run used to scan the whole parent-child table for a single row). Two tests pin the boundary: DISTINCT in an arm stays per row, and children follow their parent in queue order across a generation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sql_reference.md: WITH syntax, scoping, column lists, alias qualification, statement forms, errors; recursive CTEs (setup/recursive arms, FIFO or ORDER BY queue, UNION dedupe, LIMIT/OFFSET, no recursion limit, errors, performance); correlated CTE bodies listed under Differences from SQLite. user_guide.md: examples (run), README feature line, database.md: the pre-pass architecture and the unresolved-CTE guard, CLAUDE.md: vm/mod.rs, prepass.rs and parse.rs notes. queries2.sql gains 17 CTE queries (plain, DML, recursive counters, tree walks, depth-first), all matching SQLite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Foundation of the tsq rewrite. The old generator drew from libc srand/rand through `unsafe`, divided rand() (at most 2^31-1) by u32::MAX so every float landed in [0, 0.5) -- every customer active, nullable columns never NULL, every order to the top fifth of customers, value ranges halved -- truncated --seed to 32 bits, had modulo bias and Windows' RAND_MAX of 32767, and iterated HashMaps into its output files. - rng.rs: ChaCha8 seeded from the full 64-bit seed, unbiased ranges, floats uniform in [0, 1), independent per-item streams. The same seed gives the same bytes on every OS. No libc, `#![forbid(unsafe_code)]`. - gen.rs (`tsq gen`): customers (now with a referred_by hierarchy), products, orders, order_items, reviews with correct distributions and NULL rates; money in sqawk's canonical real spelling; `--edge-rate` salts non-key cells with edge values (i64 limits and just past them, 007, 1.50, 1e3, " 5", +5, -0, Unicode with combining marks, emoji and RTL, embedded LF/CR/quotes/delimiters, long strings, numeric-looking and NULL/true literal text, quoted empty fields) from its own stream. - data.rs: lossless writer for csv/tsv, --crlf, --bom, --no-header, --ragged-rate (rows leaving off trailing empty fields). - metadata.json records seed, options and row counts. - The 25 static query files and verify/run_verification.sh, which checked almost nothing, are gone; `tsq diff` replaces them. Tests: every output file identical across two runs (seed above 2^32), different seeds differ, foreign keys hold (sqawk anti-joins), and sqawk loads every layout with exact row counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`tsq diff` generates random statement sequences, runs each through sqawk (SQL on stdin, per-test time limit) and through SQLite 3.45 (rusqlite's bundled build) holding the same values, and compares every result set byte for byte, headers included. - Oracle (diff/oracle.rs): tables loaded into untyped columns with sqawk's per-cell inference (empty NULL, canonical integer, real only when CAST AS TEXT gives the text back, else verbatim text); each sequence runs on a fresh copy of the database, so transactions work; results rendered as sqawk prints them (NULL empty, reals via CAST AS TEXT, RFC 4180 quoting). .cargo/config.toml builds the bundled SQLite with its math functions and with LONGDOUBLE_TYPE=double, so the oracle's real formatting is the same on every OS (MSVC has no 80-bit long double; mingw under the Windows x87 control word got it wrong). - Generator (diff/sqlgen.rs, on shrinkable trees in diff/ast.rs): a port of diffh.py's Gen -- typed expressions, functions, CASE / CAST / COALESCE / IIF / NULLIF, LIKE / BETWEEN / IN, scalar / IN / EXISTS subqueries with correlation, every join kind including USING and NATURAL, GROUP BY / HAVING / FILTER / DISTINCT aggregates, windows with frames, compounds, derived tables -- plus CTEs (chained, column lists, recursive counters, Fibonacci, depth-bounded tree walks), DML sequences, transactions, CREATE TABLE AS, ALTER TABLE ADD COLUMN, printf and date functions. Column properties from the loaded data (diff/schema.rs) keep every comparison, ordering and MIN/MAX where the two engines' rules agree, so documented differences are avoided by construction; a result set compares in order only under an ORDER BY over every output column. - Classification: pass / diff / sqawk_err / sqlite_err / both_err / crash (panic, signal, abnormal exit, time limit); diff/allow.rs has one rule per documented difference (doc/sql_reference.md); SQLite running out of time is counted, not reported. Exit 1 on any finding. - --shrink greedily cuts the SQL tree (clauses, joins, list items, sub-expressions, literals), the tables (unnamed tables, row chunks, unnamed columns) and writes query.sql, the CSVs, both outputs and the replay command. - --tables gen|edge|harness, --scale small|large|ROWS, --edge-rate, --only N, --print, --jobs, --timeout, global --sqawk. Smoke test in cargo test: 300 tests on generated tables and 150 on the adversarial edge set (about 3 s on two cores). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`tsq corpus FILE...` runs hand-written query files with the oracle and
renderer of `tsq diff`, in harness.py's line format: one test per line,
`--` and blank lines skipped, `O:` for an ordered comparison,
`;`-separated statements with every result set compared (headers
included). A `-- tsq-tables: NAME` line picks the built-in table set:
`harness` (harness.py's TABLES, the default) or `edge` (diffh.py's
build_tables(7), transcribed cell for cell).
queries.sql, queries2.sql and hand1-3.sql move from tests/sqlite_diff/
to tsq/corpus/ (git mv; the hand files gain their tsq-tables line).
Parity with the Python tools on the same files:
harness.py queries + queries2: pass 544, DIFF 3, sqawk_err 9,
sqlite_err 1, both_err 7
tsq corpus: the same, every non-pass covered by a
documented-difference rule
diffh.py --only-hand hand1-3: pass 359, diff 13, sqawk_err 15,
both_err 36
tsq corpus: the same (tsq also compares headers and
every result set where diffh.py compared only the last; no case
changes category)
harness.py / diffh.py point at the new paths and stay until their
writeback and CLI testers are ported. README, CLAUDE.md and the user
guide describe the subcommands; a test pins both corpus summaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight #[ignore]d exact-output tests in tests/defects/tsq_2026_10.rs, each naming the tsq seed and test that found it (`tsq diff --seed S --only N --shrink` replays it). Expected answers are SQLite's; where SQLite builds disagree (x87 long double vs double-double, 3.45 vs 3.50) only values every build agrees on are asserted. Wrong answers: - CONCAT_WS writes a separator for an empty-string argument; SQLite 3.45/3.46 skip it (3.50 changed upstream -- fix or document). - Real to text: 443.85 / 5867 prints ...594, SQLite ...595 (SQLite rounds an ~18-digit expansion half up, not the exact binary value). - QUOTE(real) fallback (`%!.20e`) drops digits: 6.444918504005517e-04 for SQLite's 6.44491850400551701e-04. - `%` with a real operand rounds the integer operand through a double: 9007199254740993 % 2.0 is 0.0, SQLite 1.0 (doc: integers beyond 2^53 stay exact). - ROUND(1058.05, 1) is 1058.1; the stored value is below the half, and sqawk's own doc says halves are judged on the stored value (SQLite 1058.0). Performance cliffs (right answers, quadratic time; tsq diff --scale large reports them as crashes at its 10 s limit), each bounded at 5 s for work SQLite does in under one: - EXISTS whose only tie to the outer row is an outer-only predicate. - Window frames ending at UNBOUNDED FOLLOWING with a moving start. - An IN (SELECT ...) WHERE term on a join's first table applied only after the whole join. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment now names the evidence: SQLite 3.50 dropped x87 long double and MSVC never had it, and with it CAST(100.0/476.42 AS TEXT) ends in ...535 instead of ...536. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rustc 1.88 cannot infer `&str` through `Vec::push(r.pick(..))` the way current stable does; dereference explicitly. Generated data is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each performance test now uses data on which the quadratic code needs over a minute in a debug build (EXISTS 94 s, window 77 s, IN-subquery join 93 s measured) while linear code needs well under a second, and allows 20 s, so slow runners cannot make it flaky and the quadratic code cannot pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Real-to-text conversion now ports SQLite 3.50's sqlite3FpDecode, the portable path SQLite 3.45 also takes when built without long double (as MSVC builds and tsq's oracle are): the magnitude is scaled near 1e18 in Dekker double-double arithmetic, read as an 18- or 19-digit integer, and rounded half up. CAST(x AS TEXT), ||, output, writeback, QUOTE and every printf real conversion use it, so output is identical on every platform and equal to the oracle's: 443.85 / 5867 is 0.0756519515936595 (sqawk rounded the exact binary value, ...594), QUOTE(1 / 1551.61) keeps all of SQLite's digits, %!.20f of 0.1 is 0.1000000000000000055. Text-to-real conversion ports SQLite 3.50's sqlite3AtoF the same way (up to 19 significant digits, scaled in double-double arithmetic): SQL literals, CAST, arithmetic on text, date parsing and file loading read the double SQLite reads, which for extreme input can be a neighbour of the nearest one (1e-300). QUOTE's round-trip test uses it too, so QUOTE(1e-300) is 1.0e-300 as in SQLite. A real literal is now read from its own text rather than from Rust's shortest spelling of it. Load-time inference is unchanged in rule (a field is REAL only when its text is the rendering of its reading), now over the new conversions. Differential (scratch harness, 18997 doubles incl. random bit patterns, subnormals, 15/16/17-digit ties, huge/tiny; values passed bit-exactly): 0 mismatches against tsq's oracle (SQLite 3.45.0, LONGDOUBLE_TYPE=double) over CAST AS TEXT, QUOTE, ten printf forms and text-to-real reading; it agrees with SQLite 3.50.2 except where 3.50 changed printf (%.0f rounds, QUOTE of Inf). Load inference: 56991 fields, 0 mismatches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
% with a real operand now reads each operand as SQLite's sqlite3VdbeIntValue does -- integers exactly, reals truncated and clamped, text by its integer prefix -- and returns the integer remainder as a real. The integer operand used to pass through a double: 9007199254740993 % 2.0 was 0.0 (SQLite 1.0), and '1e3' % 7.0 was 6.0 (SQLite 1.0). ROUND ports SQLite's roundFunc: a magnitude above 2^52 is unchanged; to 0 places, x + 0.5 (- 0.5) truncated; otherwise %!.*f of x, with SQLite's decimal expansion, read back as a real. The digit-string rounding it replaces decided halves on Rust's 17-digit expansion, so ROUND(1058.05, 1) was 1058.1 although 1058.05 is stored below the half (SQLite 1058.0). d above 30 is now 30, as in SQLite. Checked bit-exactly against tsq's oracle over 18997 doubles (ROUND at 0/1/2/5/14/30 places, x % 7, 9007199254740993 % x, x % 2.5): 0 mismatches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sqawk writes the separator for an empty-string argument (a,,b), as SQLite 3.50 and PostgreSQL do; SQLite 3.45 skips it like NULL. Keep sqawk's behaviour: list it under Differences from SQLite, add the matching tsq allow rule (rows must agree once the CONCAT_WS separators are removed from both sides), and drop the red test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt ones first A WHERE conjunct containing a subquery was always evaluated at the innermost loop, after every join: expr_sources cannot see into a subquery. Such a conjunct (and any other that expr_sources cannot analyze) is now compiled in a scratch compiler and placed by the cursors its Column reads name, a correlated subquery's references included. t1.pid IN (SELECT pid FROM p WHERE q = 7) filters at t1's loop: the red test's join (40 million rows unfiltered) drops from 93 s to 0.4 s in a debug build. A WHERE conjunct that reads no table of the FROM -- a constant, an uncorrelated subquery, or in a subquery a test of the enclosing row only -- is tested once before the loop nest instead of once per row of its first table, as SQLite codes constant terms before the loop. A correlated EXISTS (SELECT 1 FROM r t3 WHERE t3.a < 19 AND t1.b < t1.c) no longer scans t3 for an outer row that fails t1.b < t1.c: 94 s to 0.04 s for the red test's 12000 rows (debug). EXISTS already stops at its first row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An aggregate whose frame does not keep a fixed start and a growing end was recomputed from scratch for every row: O(n^2) for frames ending at UNBOUNDED FOLLOWING with a moving start (the red test's 60000 rows took 77 s in a debug build; now 0.2 s), O(n*w) for sliding frames. The window pass now runs SQLite's sqlite3WindowCodeStep schedule: three cursors (next row to step into the frame, next to return, next to remove) advanced in SQLite's order for each frame shape, with its countdowns, by rows for ROWS and by peer groups for RANGE. Every row is stepped and removed at most once, so every frame shape is O(n). Removal is SQLite's inverse: COUNT decrements; SUM / AVG / TOTAL subtract (integers exactly, reals in the compensated sum, and a SUM that has seen a real stays a real); MIN / MAX keep the frame's candidate values in a monotonic queue (SQLite's ordered index, newest wins among equals); GROUP_CONCAT ports groupConcatStep / groupConcatInverse (drop the leaving value and the separator after it; an emptied string is NULL and starts over). So values now match SQLite where recomputation did not: SUM(x) OVER (ROWS 1 PRECEDING) over 1.5, 2, 3 is 1.5, 3.5, 5.0 (was ..., 5), inverse rounding residuals, emptied GROUP_CONCAT. Scratch differential against tsq's oracle, 62 window queries x 12 aggregates x every supported frame shape, seeds 1-8: 0 mismatching queries (the previous build: about 20 per seed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A CONCAT_WS result with a separator containing a comma is quoted by the CSV writer; removing the separator left the quotes on one side only, so tsq diff --seed 2 --only 14411 stayed a finding. Undo the quoting and compare field by field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yway tsq diff --seed 4 --only 3486 takes 91 s in sqawk, 0.02 s in SQLite: the CTE is read only by a compound arm whose WHERE is constant false, so SQLite never computes it, while sqawk materializes every referenced CTE before the statement runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A CTE or derived table was run in the pre-pass, before the statement: every referenced one cost its full evaluation even when no code that reads it ever ran. tsq diff --seed 4 --only 3486 reads a CTE (whose body self-joins another into 400 million rows) only from a compound arm whose WHERE is constant false; SQLite never runs it (0.02 s), sqawk took 91 s. The pre-pass now only compiles such a query and registers a lazy table (Database::add_lazy_table): its columns from the program's result schema, its rows from running the program when an OpenRead first opens it (VmEngine::fill_lazy), as SQLite runs a FROM subquery behind OP_Once. Recursive CTEs are still computed in the pre-pass. The FROM planner tests WHERE terms that read no table before it opens any cursor or builds a hash index (it tested them after), so a false constant WHERE opens nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uted Also covers, passing since 7429d16, a derived table read only under a false WHERE and a CTE read only by a CASE branch not taken or by the SELECT list of a query with no rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t reaches it Every table of a join was opened, and every hash index built, before the loop nest. A CTE or derived table is now computed when first opened (7429d16), so a join whose outer loop has no rows still computed its inner CTE: 12 s for a 64-million-row one where SQLite takes 0.0001 s. An inner level's OpenRead and hash build now sit behind a Once at the level's entry, where SQLite builds an automatic index. A FULL JOIN, whose unmatched-row pass reads its tables after the nest, and a RIGHT JOIN group's inner levels, which can be NULL-extended unreached, keep opening up front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe the lazy registration in prepass.rs and merge the duplicated mod.rs entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsq wb fuzzes --write: random files (delimiters, LF/CRLF/mixed line endings, BOM, ragged rows, blank and comment lines, quoted, multi-line and invalid-UTF-8 cells, gen's edge values, CREATE TABLE ... LOCATION files) and random DML / ALTER / TRUNCATE / transaction sequences, run statement by statement on the SQLite oracle as well. Each file must equal the oracle's final table rendered by the documented writeback rules (untouched rows are their source bytes, tracked by rowid); read-only targets must refuse with nothing written; SELECT output and a re-read must match; nothing stray may be left. On Unix, SIGKILL (through kill(1)) during a large write must leave the old or the new file. tsq cli fuzzes the command line and the REPL: flag and operand combinations, piped REPL scripts, malformed files, garbage, limit-value and huge SQL, closed stdout. Every run must exit 0/1/2 (or a .exit status) in time, report errors as one sqawk: line (clap's format for usage errors), never panic, and create nothing in cwd, HOME or XDG_STATE_HOME. tsq stress times 18 workloads over generated data (1M to 10M rows) and, on Linux, the peak RSS of sqawk (VmHWM polled from /proc); --json saves the results and --baseline exits 1 on a regression beyond --tolerance. run.rs gains one spawn helper (stdin, cwd, env, stdout kept / drained / closed early, peak RSS) that run_sqawk now uses. cargo test runs wb --iterations 50 and cli --iterations 100 next to the diff and corpus smoke runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…esters The Python testers under tests/sqlite_diff/ were deleted with the previous commit (554dd4c): harness.py and diffh.py were superseded by tsq corpus / tsq diff, wb.py by tsq wb, csvfuzz.py and fuzz.py by tsq cli. CLAUDE.md, the README and the user guide now describe every tsq subcommand and the cargo test smoke budgets; tsq's comments name the Python tools as retired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efix On Windows a --write error on a file outside the working directory printed the verbatim path fs::canonicalize returns (`cannot write '\\?\C:\...\ro.csv': file is read-only`). Found by tsq cli under wine; the test is Windows-only and ignored until fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in linear time Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table::column_count copied every column name to count them, and the Column opcode asks for the count on each read, so a row of n columns cost O(n^2) (SELECT * over 1000 columns x 1000 rows: 12.8 s). It is the length of the column list. Table::column_index looks a name up in a map built on first use for a table of more than 16 columns, and the header check finds repeated names with a hash set instead of rescanning the names before each one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…constant time A NATURAL JOIN of wide tables and a window over wide rows compiled in time quadratic in the column count: every reference scanned the list of merged USING columns or of the bindings of the materialized row, and the window stage's register remap scanned its register list per register. USING columns are indexed by name and by joined column (UsingCols), row bindings by column (ColBindings), the hidden-column set is a HashSet, and the window stage dedups and remaps registers through hash maps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refix fs::canonicalize returns the verbatim form on Windows, so a write error on a file outside the working directory read `cannot write '\\?\C:\...\ro.csv'`. Paths in open and write errors and in the -v writeback diagnostic go through display_path, which drops `\\?\` before a drive letter and turns `\\?\UNC\` into `\\`; the canonical path stays the table's identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wide tables now cost linear time per row, so the column-count cap that kept the quadratic code under the time limit is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.