Updates

.cargo/config.toml

@@ -0,0 +1,14 @@
+# Build flags for the bundled SQLite in tsq's differential oracle (rusqlite
+# -> libsqlite3-sys reads this variable; sqawk itself does not use SQLite):
+#
+# * SQLITE_ENABLE_MATH_FUNCTIONS: ceil, floor, ... as in SQLite's own
+#   builds (the bundled amalgamation leaves them out).
+# * LONGDOUBLE_TYPE=double: real <-> text conversion uses SQLite's
+#   double-double arithmetic on every platform, as MSVC builds always do and
+#   as current SQLite (3.50, which dropped long double) does everywhere,
+#   instead of x87 80-bit long double where the compiler offers it. Without
+#   this the oracle's 15th digit depends on the OS and on the x87 precision
+#   control word (CAST(100.0/476.42 AS TEXT) is ...535 with long double,
+#   ...536 without).
+[env]
+LIBSQLITE3_FLAGS = "-DSQLITE_ENABLE_MATH_FUNCTIONS -DLONGDOUBLE_TYPE=double"

.gitattributes

@@ -0,0 +1,5 @@
+# Test fixtures are byte-exact: tests compare their bytes, writeback output
+# and load-error line numbers. Never convert their line endings on checkout
+# (core.autocrlf=true on Windows would make them CRLF).
+tests/data/** -text
+tests/*.csv -text

CLAUDE.md

@@ -29,8 +29,11 @@ cargo run -- -s "SELECT * FROM data" data.csv
 # Run in interactive REPL mode
 cargo run -- --interactive data.csv
 
-# Generate test data with tsq (unpublished workspace crate in tsq/)
-cargo run -p tsq -- --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
+# tsq (unpublished workspace crate in tsq/): data, differential testing
+cargo run -p tsq -- gen --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
+cargo build --release --workspace
+target/release/tsq diff --seed 1 --queries 20000 --keep-going --shrink
+target/release/tsq corpus tsq/corpus/*.sql
 ```
 
 ## Architecture Overview
@@ -39,20 +42,31 @@ Sqawk is an SQL-based CLI tool for processing delimiter-separated files (CSV, TS
 
 ### Core Components
 
-- **`main.rs`**: Entry point orchestrating the pipeline: CLI parsing → file loading → SQL execution → output → optional writeback
-- **`database.rs`**: Central store for in-memory tables, manages table definitions and schemas
-- **`table.rs`**: In-memory table representation with columns, rows, and projection capabilities
-- **`sql_executor.rs`**: SQL parsing (via `sqlparser` crate) and execution against in-memory tables
+- **`main.rs`**: Entry point orchestrating the pipeline: CLI parsing → file loading → SQL execution → output → optional writeback. Any error is printed once as `sqawk: <message>` (one line, exit 1); clap usage errors exit 2
+- **`cli.rs`**: clap arguments; `-s`/`-f` are merged into one ordered script by command-line position
+- **`config.rs`**: `AppConfig`, the parsed options (-v, -F, --tabledef, --write, header/comment/output flags) passed to every component instead of global state
+- **`repl.rs`**: the interactive shell (`-i`). On a terminal: rustyline editor with prompt, multi-line statements, tab completion and history (`$XDG_STATE_HOME/sqawk/history` when that is absolute, else `~/.sqawk_history`); piped, it reads lines and prints only command output. Dot commands (`.tables`, `.schema`, `.save`, `.cd`, `.load`, ...) live here
+- **`error.rs`**: `SqawkError`; each `Display` is one complete line and no variant exposes a `source()` (that would print the cause twice). File I/O errors name the user's file (`SqawkError::open` / `write`), never a temp file
+- **`diag.rs`**: `diag!` / `outln!` -- every stderr diagnostic (errors, `-v`) and REPL stdout line goes through them; write errors are ignored, so a closed stream never panics. Do not use `eprintln!`/`println!`
+- **`database.rs`**: Central store for in-memory tables; owns modification tracking (`mark_modified` / `is_modified` / `modified_tables`, which decide what `--write` saves) and transactions (`begin` / `commit` / `rollback`; a table's original is kept the first time a transaction changes it)
+- **`table.rs`**: In-memory table representation with columns (name, type affinity, declared type name), rows, and projection capabilities
+- **`sql_executor.rs`**: `SqlExecutor`, the facade `main.rs` and the REPL use: `execute` hands SQL text to `vm::execute_vm` and applies the output delimiter to the result tables; it also saves modified tables (`save_modified_tables`, `save_table`), rolls back a transaction left open at exit (`finish`), and answers the REPL's table/column/row-count queries. It does no parsing itself
 
 ### SQL Execution (VM-based)
 
 SQL execution uses a bytecode VM inspired by SQLite's architecture:
-1. SQL is parsed via the `sqlparser` crate
+1. SQL is parsed by `src/vm/parse.rs` (`parse_sql`, sqlparser with `SqawkDialect`)
 2. The AST is compiled to bytecode (`src/vm/compiler.rs`)
 3. The VM engine executes the bytecode (`src/vm/engine.rs`)
 
 ### VM Module (`src/vm/`)
 
+- `mod.rs`: `execute_vm` / `execute_statements`, the statement loop: parses
+  the script, then per statement runs BEGIN/COMMIT/ROLLBACK directly on the
+  `Database`, runs the pre-pass, compiles and executes the statement, applies
+  the engine's `TableModification`s (insert/replace/delete rows, create/drop/
+  alter/truncate tables) to the `Database`, marking each changed table
+  modified, and drops the statement's registered tables
 - `bytecode.rs`: Bytecode instruction set and register/program types
 - `compiler.rs`: SQL AST to bytecode. Holds `code_expr`, the single expression
   compiler (the analogue of SQLite's `sqlite3ExprCode`) used from the SELECT
@@ -81,18 +95,41 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
   same query compiler as fragments of the enclosing program (`Once` when
   uncorrelated; correlation found by `NameCtx` outward resolution)
 - `compiler_check.rs`: rejects every parsed clause sqawk does not implement
-  (RETURNING, ON CONFLICT, UPDATE FROM, WITH, QUALIFY, ...) before
+  (RETURNING, ON CONFLICT, UPDATE FROM, QUALIFY, ...) before
   compilation; INSERT/UPDATE/DELETE/Query/Select are destructured
   exhaustively so a new sqlparser field fails to compile until handled
+- `parse.rs`: `parse_sql`, the only SQL-to-AST entry point: parses with
+  `dialect.rs`'s `SqawkDialect` (Hive syntax, SQLite operator precedence),
+  rewrites compound SELECTs left-deep (equal precedence, left to right), and
+  enforces SQLite's limits (expression depth 1000, 500 compound terms) with an
+  iterative walk before anything recurses over the tree
+- `prepass.rs`: runs before each statement compiles. Derived tables and CTEs
+  become registered tables (`sqawk_derived_N`, `sqawk_cte_N`) and FROM items
+  are rewritten to them. A non-recursive one is only compiled here and
+  registered lazily (`Database::add_lazy_table`); its rows are computed the
+  first time a cursor opens it (`OpenRead` → `VmEngine::fill_lazy`, SQLite's
+  `OP_Once`), so one read only by code that never runs is never computed.
+  Also CTE scoping (siblings, self, shadowing), `WITH ... DML` unwrapping, and
+  recursive CTEs, computed here (SQLite's queue algorithm, one run per
+  generation when the queue is FIFO). Other
+  one-part references sharing a CTE's name become `main.t`, and the compiler
+  refuses any one-part CTE name left over (`SqlCompiler::cte_names`), so a
+  missed rewrite cannot read the loaded table
+- `parse.rs`: `parse_sql` (SQLite precedence, hex literals, MATERIALIZED
+  hints dropped, compounds left-deep, depth / compound limits) and
+  `for_each_child`, the one enumeration of a statement's queries, FROM items
+  and expressions shared by the limit walk and the pre-pass
 - `ast_compat.rs`: thin accessors normalizing sqlparser AST shapes
 - `engine.rs`: Executes bytecode instructions
 - `arith.rs`: SQLite numeric semantics -- text-to-number coercion, i64
-  arithmetic with overflow to real, CAST, truth values
+  arithmetic with overflow to real, bitwise operators, CAST, truth values
 - `func.rs`: the scalar function table (`FuncDef`: name, arity, body) run by
   the `Function` opcode (SQLite's OP_Function); arity is checked at compile
   time. Lazy functions (COALESCE, IFNULL, IIF, NULLIF) are compiled to jumps
-- `datetime.rs`: DATE/TIME/DATETIME/STRFTIME/... (SQLite time strings and
-  modifiers, UTC)
+- `printf.rs`: SQLite's printf (a port of `sqlite3_str_vappendf`), used by
+  PRINTF/FORMAT, QUOTE and the date functions
+- `datetime.rs`: DATE/TIME/DATETIME/STRFTIME/TIMEDIFF/... (a port of SQLite
+  3.45's date.c: time strings, modifiers, `localtime` via the C library)
 - `like.rs`: the LIKE matcher (no regex; compiled pattern cached per
   instruction)
 - `agg.rs`: aggregate accumulators shared by AggStep/AggFinal and windows
@@ -102,12 +139,19 @@ effects of the ones before it.
 
 ### File Handling
 
-- **`file_handler.rs`**: Manages loading files into database tables
+- **`file_handler.rs`**: Manages loading files into database tables and saving
+  them. `parse_table_spec` splits a file operand `[NAME=]PATH` (NAME only when
+  identifier-like; otherwise the table is named after the file stem; `-` is
+  stdin, table `stdin`); `has_tab_extension` is the `.tsv`/`.tab` rule used
+  for loading and for `CREATE TABLE ... LOCATION`; `load_location` loads (or
+  starts) a LOCATION table under its declared columns
 - **`csv_handler.rs`**: The single loader (`read_table`, built on the `csv`
-  crate) for every delimiter and for stdin, plus writeback. `.csv` files always
-  have a header; other files keep the header guess and `#`-comment skipping.
-  Fields load as Integer/Float only when the text round-trips exactly, so
-  untouched cells write back byte-for-byte
+  crate) for every delimiter and for stdin, plus writeback. `.tsv`/`.tab` are
+  tab-separated, everything else comma (unless `-F`); the first line is always
+  the header unless `--no-header`/`--tabledef`; `#` lines are comments only
+  under `--comments`. Each table keeps its file's bytes (`RawRows` in
+  `table.rs`: one buffer + per-row offsets), so untouched rows write back
+  byte-for-byte; every row mutation in `table.rs` keeps the offsets in sync
 
 ### SQL Features
 
@@ -122,15 +166,16 @@ follow SQLite 3.45; deliberate differences are listed in
 
 ### Safe Writeback Model
 
-By default, all modifications remain in memory only. The `--write` flag must be explicitly provided to save changes back to source files. Only tables modified by INSERT/UPDATE/DELETE are written.
+By default, all modifications remain in memory only. The `--write` flag must be explicitly provided to save changes back to source files. Only tables changed by INSERT/UPDATE/DELETE/ALTER/TRUNCATE are written, and rows a statement did not change are written back byte for byte. `CREATE TABLE t (cols) LOCATION 'file'` follows Hive external-table semantics (`file_handler::load_location`): an existing file is loaded under the declared columns (its header line skipped, checked against the column count and written back verbatim; `WITH (HEADER=false)` makes the first line data; `WITH (COMMENTS=true)` is `--comments`) and loading it is not a modification; a file that does not exist yet is written even with no DML after it (created with just the header). The delimiter is WITH (DELIMITER=...) if given, else tab for `.tsv`/`.tab`, else comma. A LOCATION file already loaded as another table is an error. Scratch tables (CREATE TABLE without LOCATION, CREATE TABLE AS) are never written. Every target is checked for writability before any file is written; a read-only file aborts the whole writeback. An open transaction is rolled back before writeback.
 
 ## Test Organization
 
 Tests are in `tests/` organized by functionality:
 - `basic/`, `comparison/`, `join/`, `join_on/`, `order_by/`, `update/`
 - `unsupported/`: parsed-but-unimplemented syntax must error and modify nothing
 - `aggregate/`, `alias/`, `distinct/`, `group_by/`, `limit_offset/`
-- `delimiter/`, `csv_handler/`, `string_functions/`, `repl/`
+- `delimiter/`, `file_io/`, `headerless/`, `stdin/`, `string_functions/`, `repl/`, `transaction/`
+- loader and writer unit tests live in `src/csv_handler.rs` (`#[cfg(test)]`)
 - `helpers/`: Test utilities
 - `common.rs`: Shared test helpers including REPL script runner
 
@@ -145,10 +190,63 @@ Two suites carry the correctness contract:
   defects. Run with `cargo test --test mod defects -- --ignored`. Remove the
   `#[ignore]` when a defect is fixed; the list only shrinks.
 
-`tests/sqlite_diff/harness.py` is a differential tester against SQLite (via
-python3's `sqlite3` module), not part of `cargo test`. Run
-`cargo build --release && python3 -I tests/sqlite_diff/harness.py
-tests/sqlite_diff/queries.sql tests/sqlite_diff/queries2.sql`; the SUMMARY
-line counts pass / DIFF (wrong answer) / sqawk_err.
+`tsq/` (the unpublished `tsq` crate) is the differential tester and fuzzer.
+Build with `cargo build --release --workspace`; `--sqawk PATH` or
+`SQAWK=path` overrides the binary (default: the `sqawk` next to `tsq`). Data
+goes to temp dirs. Module map: `rng.rs` (ChaCha8, the only randomness),
+`data.rs` (tables, lossless CSV/TSV writer), `gen.rs` (`tsq gen`), `run.rs`
+(spawning sqawk: time limit, stdin, cwd/env, stdout kept / drained / closed
+early, Linux peak RSS from `/proc/<pid>/status`), `corpus.rs`, `wb.rs`
+(`tsq wb`), `clifuzz.rs` (`tsq cli`), `stress.rs` (`tsq stress`), and
+`diff/`: `oracle.rs`
+(rusqlite's bundled SQLite 3.45 -- `.cargo/config.toml` enables its math
+functions -- loaded with sqawk's per-cell inference into untyped columns,
+results rendered as sqawk prints them: reals via `CAST(x AS TEXT)`, NULL
+empty, RFC 4180 quoting, headers included, compared byte for byte),
+`schema.rs` (per-column properties deciding which comparisons both engines
+agree on), `sqlgen.rs` (the generator, built on `ast.rs` shrinkable trees),
+`allow.rs` (one rule per documented difference), `shrink.rs`, `tables.rs`
+(built-in `harness` / `edge` table sets).
+
+- `tsq diff --seed N --queries 20000 --keep-going [--scale large]
+  [--tables edge] [--edge-rate 0.05] [--shrink --out DIR]`: random SQL
+  vs SQLite. `DIFF SUMMARY {pass, diff, sqawk_err, sqlite_err, both_err,
+  crash}`, then the documented-difference counts, SQLite time-outs (too
+  heavy for the reference; not findings) and `findings N` (exit 1 when
+  non-zero). `--only N` replays one test; `--print` shows the SQL.
+- `tsq corpus tsq/corpus/*.sql`: the hand corpora (`queries*.sql` on the
+  `harness` tables, `hand*.sql` on `edge`, chosen by a `-- tsq-tables:`
+  line); same summary as `CORPUS SUMMARY`.
+- `tsq wb --seed N --iterations 2000 [--only N] [--kill-rate F]`:
+  `--write` fuzzer. Random files (delimiters, LF/CRLF/mixed, BOM, ragged
+  rows, blank and `#` lines under `--comments`, quoted / multi-line /
+  invalid-UTF-8 cells, gen's edge values, `--no-header`, `CREATE TABLE ...
+  LOCATION` files) and random UPDATE / DELETE / INSERT / ALTER / TRUNCATE /
+  SELECT with BEGIN / COMMIT / END / ROLLBACK, run statement by statement on
+  the oracle too. Each file must equal the oracle's final table rendered by
+  the writeback rules (untouched rows = source bytes, tracked by rowid);
+  read-only targets refuse with nothing written; SELECT output and a
+  re-read must match; no stray files. Unix: SIGKILL (`kill -KILL`) during a
+  large write leaves old or new content (temp `.tmp*` leftovers are counted,
+  not failures). `WB SUMMARY {ok, bad, crash}`, must be bad 0 crash 0.
+- `tsq cli --seed N --iterations 2000 [--only N]`: CLI / REPL surface fuzz
+  (flag and operand combinations, piped REPL scripts, the retired
+  csvfuzz.py's malformed files and mutations, the retired fuzz.py's garbage
+  SQL, limit values and huge statements via `-f`, closed stdout). Oracle: exit 0/1/2 (or `.exit
+  N`), no panic / overflow text, one `sqawk: ` line per error (clap's usage
+  format for 2), no hang, nothing created in cwd / HOME / XDG_STATE_HOME
+  (temp dirs), no banner or prompt in piped REPL output. `CLI SUMMARY {ok,
+  bad, crash}`, must be bad 0 crash 0.
+- `tsq stress --rows 1000000 [--workloads a,b] [--json OUT] [--baseline FILE
+  --tolerance PCT] [--repeat K]`: wall time and (Linux) peak RSS per
+  workload; with `--baseline`, exit 1 on a regression. Not in CI.
+- `cargo test --workspace` runs smoke budgets in `tsq/tests/tsq.rs`: `diff
+  --queries 300` (+150 on `edge`), `corpus` on all corpora, `wb --iterations
+  50`, `cli --iterations 100` -- about 10 s serialized in a debug build.
+
+Every remaining DIFF / sqawk_err must be a difference or an unsupported
+feature listed in `doc/sql_reference.md` ("Differences from SQLite",
+"Unsupported syntax"); panic and hang must be 0. NULL and `''` both print
+empty, so the testers cannot tell them apart.
 
 Tests use `assert_cmd` for CLI testing and `tempfile` for temporary test data.