Fixes

.github/workflows/ci.yml

@@ -46,20 +46,21 @@ jobs:
           components: clippy
       - uses: Swatinem/rust-cache@v2
       # Without dev-dependencies, which cargo test and clippy --all-targets
-      # merge in: this is the feature set users actually build.
+      # merge in: this is the feature set users actually build. --workspace
+      # adds tsq/, the unpublished test-data generator crate.
       - name: Build
-        run: cargo build --locked
-      # On Windows too, not just Linux: src/tsq.rs and src/storage/mmap.rs have
-      # cfg(unix) branches whose non-unix side is otherwise never linted. macOS
-      # is unix like Linux, so it adds nothing.
+        run: cargo build --locked --workspace
+      # On Windows too, not just Linux: tsq/src/main.rs has cfg(unix) branches
+      # whose non-unix side is otherwise never linted. macOS is unix like
+      # Linux, so it adds nothing.
       - name: Clippy
         if: runner.os != 'macOS'
-        run: cargo clippy --locked --all-targets -- -D warnings
+        run: cargo clippy --locked --workspace --all-targets -- -D warnings
       # Run even when Clippy failed, so a lint (often a new one from a stable
       # toolchain release) does not hide the test results on every OS.
       - name: Test
         if: ${{ !cancelled() }}
-        run: cargo test --locked
+        run: cargo test --locked --workspace
 
   test-release:
     name: Test (release)
@@ -72,7 +73,7 @@ jobs:
       - uses: dtolnay/rust-toolchain@stable
       - uses: Swatinem/rust-cache@v2
       - name: Test
-        run: cargo test --locked --release
+        run: cargo test --locked --workspace --release
 
   fmt:
     name: Format
@@ -107,7 +108,8 @@ jobs:
         with:
           toolchain: ${{ steps.msrv.outputs.version }}
       - uses: Swatinem/rust-cache@v2
-      # Library and binaries only: rust-version is a promise to consumers, and
-      # dev-dependencies are not part of it.
+      # Binaries only: rust-version is a promise to people building sqawk, and
+      # dev-dependencies are not part of it. --workspace includes tsq/, which
+      # declares the same rust-version.
       - name: Check build at MSRV
-        run: cargo check --locked
+        run: cargo check --locked --workspace

CLAUDE.md

@@ -9,7 +9,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 cargo build
 
 # Run all tests
-cargo test
+cargo test --workspace   # sqawk + the tsq dev crate
 
 # Run a specific test module
 cargo test --test mod basic       # Basic end-to-end tests
@@ -28,6 +28,9 @@ cargo run -- -s "SELECT * FROM data" data.csv
 
 # Run in interactive REPL mode
 cargo run -- --interactive data.csv
+
+# Generate test data with tsq (unpublished workspace crate in tsq/)
+cargo run -p tsq -- --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
 ```
 
 ## Architecture Overview
@@ -55,29 +58,67 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
   compiler (the analogue of SQLite's `sqlite3ExprCode`) used from the SELECT
   list, WHERE, HAVING, ORDER BY and SET clauses alike, plus `NameCtx`, which
   carries the FROM sources a name can resolve against
+- `compiler_select.rs`: the SELECT driver -- `compile_query`, compound
+  operators (UNION/INTERSECT/EXCEPT via keyed ephemeral tables) and the single
+  output stage (ORDER BY with hidden sort keys, DISTINCT, LIMIT/OFFSET) every
+  SELECT core ends in
 - `compiler_aggregate.rs`: aggregates, GROUP BY, HAVING
-- `compiler_join.rs`: joins
+- `compiler_from.rs`: the FROM planner used by every SELECT -- tables, comma
+  lists and join chains become nested loop levels (ON per level, WHERE after
+  outer-join NULL-extension, hash join on equi-conditions); the projection /
+  aggregate stage is a body callback emitted at the innermost level; a
+  FROM-less SELECT is a plan with zero sources (one row)
 - `compiler_dml.rs`: INSERT / UPDATE / DELETE
 - `compiler_ddl.rs`: CREATE / DROP / ALTER / TRUNCATE
-- `compiler_window.rs`: window functions
+- `compiler_window.rs`: the window stage -- finished rows (every column, or
+  a grouped query's keys and aggregates) plus window keys/arguments go into
+  an ephemeral table; one `WindowCompute` per distinct PARTITION BY/ORDER BY
+  sorts it and fills each function's result column (`src/vm/window.rs` is
+  the run-time half); the output stage reads rows back with each window call
+  bound like an aggregate, so window calls work inside any expression and in
+  ORDER BY
+- `compiler_subquery.rs`: scalar / EXISTS / IN subqueries, compiled by the
+  same query compiler as fragments of the enclosing program (`Once` when
+  uncorrelated; correlation found by `NameCtx` outward resolution)
+- `compiler_check.rs`: rejects every parsed clause sqawk does not implement
+  (RETURNING, ON CONFLICT, UPDATE FROM, WITH, QUALIFY, ...) before
+  compilation; INSERT/UPDATE/DELETE/Query/Select are destructured
+  exhaustively so a new sqlparser field fails to compile until handled
 - `ast_compat.rs`: thin accessors normalizing sqlparser AST shapes
 - `engine.rs`: Executes bytecode instructions
+- `arith.rs`: SQLite numeric semantics -- text-to-number coercion, i64
+  arithmetic with overflow to real, CAST, truth values
+- `func.rs`: the scalar function table (`FuncDef`: name, arity, body) run by
+  the `Function` opcode (SQLite's OP_Function); arity is checked at compile
+  time. Lazy functions (COALESCE, IFNULL, IIF, NULLIF) are compiled to jumps
+- `datetime.rs`: DATE/TIME/DATETIME/STRFTIME/... (SQLite time strings and
+  modifiers, UTC)
+- `like.rs`: the LIKE matcher (no regex; compiled pattern cached per
+  instruction)
+- `agg.rs`: aggregate accumulators shared by AggStep/AggFinal and windows
 
 Each statement is compiled and executed separately, so a statement observes the
 effects of the ones before it.
 
 ### File Handling
 
 - **`file_handler.rs`**: Manages loading files into database tables
-- **`csv_handler.rs`**: Standard CSV file parsing (comma-separated)
-- **`delim_handler.rs`**: Custom delimiter support via `-F` flag (TSV, colon-separated, etc.)
+- **`csv_handler.rs`**: The single loader (`read_table`, built on the `csv`
+  crate) for every delimiter and for stdin, plus writeback. `.csv` files always
+  have a header; other files keep the header guess and `#`-comment skipping.
+  Fields load as Integer/Float only when the text round-trips exactly, so
+  untouched cells write back byte-for-byte
 
 ### SQL Features
 
-Expression compilation, including all scalar and string functions, lives in
-`src/vm/compiler.rs` behind `code_expr`. Aggregates are compiled in
-`src/vm/compiler_aggregate.rs` and executed by the `AggStep`/`AggFinal`
-opcodes; `src/aggregate.rs` retains only the function-name lookup.
+Expression compilation lives in `src/vm/compiler.rs` behind `code_expr`;
+scalar functions are looked up in `src/vm/func.rs` and called through the
+`Function` opcode. Aggregates (including FILTER and GROUP_CONCAT's separator)
+are compiled in `src/vm/compiler_aggregate.rs` and executed by the
+`AggStep`/`AggFinal` opcodes over `src/vm/agg.rs` accumulators;
+`src/aggregate.rs` retains only the function-name lookup and arity. Semantics
+follow SQLite 3.45; deliberate differences are listed in
+`doc/sql_reference.md` ("Differences from SQLite").
 
 ### Safe Writeback Model
 
@@ -87,6 +128,7 @@ By default, all modifications remain in memory only. The `--write` flag must be
 
 Tests are in `tests/` organized by functionality:
 - `basic/`, `comparison/`, `join/`, `join_on/`, `order_by/`, `update/`
+- `unsupported/`: parsed-but-unimplemented syntax must error and modify nothing
 - `aggregate/`, `alias/`, `distinct/`, `group_by/`, `limit_offset/`
 - `delimiter/`, `csv_handler/`, `string_functions/`, `repl/`
 - `helpers/`: Test utilities
@@ -103,4 +145,10 @@ Two suites carry the correctness contract:
   defects. Run with `cargo test --test mod defects -- --ignored`. Remove the
   `#[ignore]` when a defect is fixed; the list only shrinks.
 
+`tests/sqlite_diff/harness.py` is a differential tester against SQLite (via
+python3's `sqlite3` module), not part of `cargo test`. Run
+`cargo build --release && python3 -I tests/sqlite_diff/harness.py
+tests/sqlite_diff/queries.sql tests/sqlite_diff/queries2.sql`; the SUMMARY
+line counts pass / DIFF (wrong answer) / sqawk_err.
+
 Tests use `assert_cmd` for CLI testing and `tempfile` for temporary test data.

Cargo.toml

@@ -7,11 +7,7 @@ description = "An SQL-based command-line tool for processing delimiter-separated
 license = "MIT"
 repository = "https://github.com/jgarzik/sqawk"
 homepage = "https://github.com/jgarzik/sqawk"
-documentation = "https://docs.rs/sqawk"
 readme = "README.md"
-# `cargo run` and `cargo install --path .` would otherwise be ambiguous across
-# the two [[bin]] targets below.
-default-run = "sqawk"
 keywords = ["csv", "tsv", "sql", "delimited", "awk"]
 categories = ["command-line-utilities", "text-processing", "parser-implementations"]
 # Minimum supported Rust version.
@@ -37,8 +33,6 @@ sqlparser = "0.62"
 csv = "1"
 anyhow = "1"
 regex = "1"
-memmap2 = "0.9"
-libc = "0.2"
 tempfile = "3"
 
 rustyline = "15"
@@ -58,6 +52,11 @@ debug = true
 name = "sqawk"
 path = "src/main.rs"
 
-[[bin]]
-name = "tsq"
-path = "src/tsq.rs"
+# `tsq`, the test-data generator, is a separate workspace member with
+# `publish = false`: it builds and runs locally (`cargo run -p tsq`) but is
+# not part of the published sqawk crate. `cargo package` skips nested crates.
+# Only sqawk is a default member, so plain `cargo build` / `cargo publish`
+# act on sqawk alone; use `--workspace` (as CI does) to include tsq.
+[workspace]
+members = ["tsq"]
+default-members = ["."]

README.md

@@ -1,7 +1,6 @@
 # Sqawk
 
 [![Crates.io](https://img.shields.io/crates/v/sqawk.svg)](https://crates.io/crates/sqawk)
-[![Docs.rs](https://docs.rs/sqawk/badge.svg)](https://docs.rs/sqawk)
 [![MIT licensed](https://img.shields.io/crates/l/sqawk.svg)](./LICENSE)
 
 Sqawk is an SQL-based command-line tool for processing delimiter-separated files (CSV, TSV, etc.), inspired by the classic `awk` command. It loads data into in-memory tables, executes SQL queries, and optionally writes results back to files.
@@ -10,14 +9,15 @@ Sqawk is an SQL-based command-line tool for processing delimiter-separated files
 
 - **SQL Query Engine** - SELECT, INSERT, UPDATE, DELETE with WHERE, ORDER BY, GROUP BY, HAVING, LIMIT/OFFSET
 - **Joins** - INNER, LEFT, RIGHT, FULL OUTER, and CROSS joins with ON conditions
-- **Aggregates** - COUNT, SUM, AVG, MIN, MAX with GROUP BY support
-- **Functions** - String (UPPER, LOWER, SUBSTR, REPLACE, etc.), math (ABS, ROUND, etc.), date/time
+- **Aggregates** - COUNT, SUM, TOTAL, AVG, MIN, MAX, GROUP_CONCAT/STRING_AGG with GROUP BY, DISTINCT and `FILTER (WHERE ...)`
+- **Functions** - String (UPPER, LOWER, LENGTH, SUBSTR, INSTR, REPLACE, TRIM, LPAD/RPAD, PRINTF, ...), math (ABS, ROUND, CEIL, FLOOR, SIGN, multi-argument MIN/MAX), conditional (COALESCE, IFNULL, NULLIF, IIF, TYPEOF), date/time (DATE, TIME, DATETIME, STRFTIME, JULIANDAY, UNIXEPOCH with modifiers)
+- **SQLite semantics** - 64-bit integer arithmetic, SQLite CAST rules, ASCII case-insensitive LIKE with ESCAPE, character-based strings; unimplemented syntax is an error, never silently ignored
 - **Subqueries** - Scalar, `IN (SELECT ...)`, and `EXISTS`, including correlated
 - **Set Operations** - UNION, UNION ALL, INTERSECT, EXCEPT
-- **Window Functions** - ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, and aggregates with `OVER (PARTITION BY ... ORDER BY ...)`
+- **Window Functions** - ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, LAG, LEAD, FIRST_VALUE, LAST_VALUE, NTH_VALUE, and aggregates with `OVER (PARTITION BY ... ORDER BY ...)`, ROWS frames and named windows
 - **DDL** - CREATE TABLE, CREATE TABLE AS SELECT, DROP, ALTER TABLE ADD COLUMN, TRUNCATE
-- **Expressions** - CASE, CAST, COALESCE, NULLIF, BETWEEN, IN, LIKE/ILIKE, `||`, arithmetic
-- **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--tabledef`
+- **Expressions** - CASE, CAST, BETWEEN, IN, LIKE/ILIKE ... ESCAPE, IS [NOT] DISTINCT FROM, `||`, arithmetic; ORDER BY ... NULLS FIRST/LAST
+- **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--no-header` or `--tabledef`; unchanged cells survive `--write` byte for byte
 - **Pipelines** - Reads standard input with a `-` file operand
 - **Safe by Default** - Files unchanged unless `--write` flag is specified
 - **Interactive REPL** - Explore data interactively with `-i` flag
@@ -28,8 +28,7 @@ Sqawk is an SQL-based command-line tool for processing delimiter-separated files
 cargo install sqawk
 ```
 
-This installs two binaries: `sqawk` and [`tsq`](#tsq---test-data-generator).
-Requires Rust 1.88 or newer.
+This installs the `sqawk` binary. Requires Rust 1.88 or newer.
 
 ## Quick Examples
 
@@ -50,13 +49,15 @@ sqawk -s "UPDATE data SET status = 'archived' WHERE year < 2020" data.csv --writ
 cat employees.csv | sqawk -s "SELECT name FROM stdin WHERE salary > 70000" -
 ```
 
-## `tsq` - test data generator
+## `tsq` - test data generator (development only)
 
-`cargo install sqawk` also installs `tsq`, which generates deterministic
-multi-table CSV data plus a corpus of SQL queries for exercising sqawk.
+The source tree also contains `tsq`, which generates deterministic multi-table
+CSV data plus a corpus of SQL queries for exercising sqawk. It is a separate
+workspace crate (`tsq/`) that is not published, so `cargo install sqawk` does
+not install it. Run it from a checkout:
 
 ```sh
-tsq --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
+cargo run -p tsq -- --seed 42 --rows 1000 --output-dir /tmp/sqawk-test
 sqawk -s "SELECT * FROM customers LIMIT 10" /tmp/sqawk-test/data/customers.csv
 ```