Updates

CLAUDE.md

@@ -39,7 +39,10 @@ Sqawk is an SQL-based CLI tool for processing delimiter-separated files (CSV, TS
 
 ### Core Components
 
-- **`main.rs`**: Entry point orchestrating the pipeline: CLI parsing → file loading → SQL execution → output → optional writeback
+- **`main.rs`**: Entry point orchestrating the pipeline: CLI parsing → file loading → SQL execution → output → optional writeback. Any error is printed once as `sqawk: <message>` (one line, exit 1); clap usage errors exit 2
+- **`cli.rs`**: clap arguments; `-s`/`-f` are merged into one ordered script by command-line position
+- **`error.rs`**: `SqawkError`; each `Display` is one complete line and no variant exposes a `source()` (that would print the cause twice). File I/O errors name the user's file (`SqawkError::open` / `write`), never a temp file
+- **`diag.rs`**: `diag!` / `outln!` -- every stderr diagnostic (errors, `-v`) and REPL stdout line goes through them; write errors are ignored, so a closed stream never panics. Do not use `eprintln!`/`println!`
 - **`database.rs`**: Central store for in-memory tables, manages table definitions and schemas
 - **`table.rs`**: In-memory table representation with columns, rows, and projection capabilities
 - **`sql_executor.rs`**: SQL parsing (via `sqlparser` crate) and execution against in-memory tables
@@ -84,6 +87,11 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
   (RETURNING, ON CONFLICT, UPDATE FROM, WITH, QUALIFY, ...) before
   compilation; INSERT/UPDATE/DELETE/Query/Select are destructured
   exhaustively so a new sqlparser field fails to compile until handled
+- `parse.rs`: `parse_sql`, the only SQL-to-AST entry point: parses with
+  `dialect.rs`'s `SqawkDialect` (Hive syntax, SQLite operator precedence),
+  rewrites compound SELECTs left-deep (equal precedence, left to right), and
+  enforces SQLite's limits (expression depth 1000, 500 compound terms) with an
+  iterative walk before anything recurses over the tree
 - `ast_compat.rs`: thin accessors normalizing sqlparser AST shapes
 - `engine.rs`: Executes bytecode instructions
 - `arith.rs`: SQLite numeric semantics -- text-to-number coercion, i64
@@ -104,10 +112,12 @@ effects of the ones before it.
 
 - **`file_handler.rs`**: Manages loading files into database tables
 - **`csv_handler.rs`**: The single loader (`read_table`, built on the `csv`
-  crate) for every delimiter and for stdin, plus writeback. `.csv` files always
-  have a header; other files keep the header guess and `#`-comment skipping.
-  Fields load as Integer/Float only when the text round-trips exactly, so
-  untouched cells write back byte-for-byte
+  crate) for every delimiter and for stdin, plus writeback. `.tsv`/`.tab` are
+  tab-separated, everything else comma (unless `-F`); the first line is always
+  the header unless `--no-header`/`--tabledef`; `#` lines are comments only
+  under `--comments`. Each table keeps its file's bytes (`RawRows` in
+  `table.rs`: one buffer + per-row offsets), so untouched rows write back
+  byte-for-byte; every row mutation in `table.rs` keeps the offsets in sync
 
 ### SQL Features
 
@@ -122,7 +132,7 @@ follow SQLite 3.45; deliberate differences are listed in
 
 ### Safe Writeback Model
 
-By default, all modifications remain in memory only. The `--write` flag must be explicitly provided to save changes back to source files. Only tables modified by INSERT/UPDATE/DELETE are written.
+By default, all modifications remain in memory only. The `--write` flag must be explicitly provided to save changes back to source files. Only tables changed by INSERT/UPDATE/DELETE/ALTER/TRUNCATE are written, and rows a statement did not change are written back byte for byte. Scratch tables (CREATE TABLE without LOCATION, CREATE TABLE AS) are never written. Every target is checked for writability before any file is written; a read-only file aborts the whole writeback. An open transaction is rolled back before writeback.
 
 ## Test Organization
 

Cargo.toml

@@ -31,11 +31,13 @@ exclude = [
 clap = { version = "4", features = ["derive"] }
 sqlparser = "0.62"
 csv = "1"
-anyhow = "1"
 regex = "1"
 tempfile = "3"
 
 rustyline = "15"
+# Runs sqawk on a large stack (see MAIN_STACK_BYTES in main.rs); already in
+# the build through sqlparser -> recursive.
+stacker = "0.1"
 
 [dev-dependencies]
 assert_cmd = "2"

README.md

@@ -17,7 +17,7 @@ Sqawk is an SQL-based command-line tool for processing delimiter-separated files
 - **Window Functions** - ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, LAG, LEAD, FIRST_VALUE, LAST_VALUE, NTH_VALUE, and aggregates with `OVER (PARTITION BY ... ORDER BY ...)`, ROWS frames and named windows
 - **DDL** - CREATE TABLE, CREATE TABLE AS SELECT, DROP, ALTER TABLE ADD COLUMN, TRUNCATE
 - **Expressions** - CASE, CAST, BETWEEN, IN, LIKE/ILIKE ... ESCAPE, IS [NOT] DISTINCT FROM, `||`, arithmetic; ORDER BY ... NULLS FIRST/LAST
-- **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--no-header` or `--tabledef`; unchanged cells survive `--write` byte for byte
+- **File Formats** - CSV, TSV (`.tsv`/`.tab`), and custom delimiters (`-F`); headerless files via `--no-header` or `--tabledef`; `#` comment lines via `--comments`; untouched rows survive `--write` byte for byte
 - **Pipelines** - Reads standard input with a `-` file operand
 - **Safe by Default** - Files unchanged unless `--write` flag is specified
 - **Interactive REPL** - Explore data interactively with `-i` flag
@@ -47,8 +47,14 @@ sqawk -s "UPDATE data SET status = 'archived' WHERE year < 2020" data.csv --writ
 
 # Read standard input as the table "stdin"
 cat employees.csv | sqawk -s "SELECT name FROM stdin WHERE salary > 70000" -
+
+# Run the statements in a file; print tab-separated results without a header
+sqawk -f report.sql --output-delimiter '\t' --no-output-header employees.csv
 ```
 
+Errors are one line on stderr (`sqawk: no such column: nmae`) with exit
+status 1; usage errors exit with status 2.
+
 ## `tsq` - test data generator (development only)
 
 The source tree also contains `tsq`, which generates deterministic multi-table

doc/database.md

@@ -36,22 +36,49 @@ Tables maintain an internal mapping of column names to their indices for efficie
   logic, so a comparison involving NULL is UNKNOWN and its row is filtered out
   rather than raising an error. Sorting and grouping treat NULL as equal to
   NULL, which SQL comparison must not — the two use separate paths deliberately
-- **Coercion**: A comparison mixing a number and a string converts the string
-  when it parses as a number, and compares textually otherwise. See
-  [SQL Reference](sql_reference.md) for the exact rules
+- **Comparison**: One rule everywhere (`=`, `<`, IN lists and subqueries,
+  joins, ORDER BY, MIN/MAX): two values that both read as numbers -- numbers,
+  or text spelling a number such as `'007'` or `'1e3'` -- compare as numbers,
+  text against text included (`'10' > '9'`); numbers sort before other text;
+  other text compares byte by byte. DISTINCT, GROUP BY and the compound
+  operators still keep text `'1'` and integer `1` apart. See
+  [SQL Reference](sql_reference.md#type-coercion) for the exact rules
 
 ### Table Lifecycle
 
 Tables are loaded from delimiter-separated files at startup and can be modified through SQL operations. Sqawk uses a safe, sed-like writeback model:
 
 1. **Default Behavior**: All modifications remain in memory only
-2. **Tracking Changes**: The system tracks which tables have been modified by any operation
+2. **Tracking Changes**: The database keeps the set of modified tables
+   (`Database::modified`): INSERT, UPDATE, DELETE, ALTER TABLE and TRUNCATE
+   mark a table, CREATE TABLE marks the new one, and DROP TABLE removes the
+   mark with the table
 3. **Write on Exit**: Modified tables are only written back to their source files if:
    - The `--write` (or `-w`) flag is explicitly provided
-   - The table was actually modified by an SQL operation (INSERT, UPDATE, DELETE)
+   - The table was actually modified by one of those statements
+   - The table has a file: a scratch table (CREATE TABLE without LOCATION,
+     CREATE TABLE ... AS) is skipped. A table read from standard input is an
+     error instead
 4. **Safe Execution**: Without the `--write` flag, source files remain untouched regardless of operations performed
 5. **Write Only Modified**: Only tables that were changed are written; unmodified tables are not rewritten
-6. **Format Preservation**: Original file formats and delimiters are preserved during writeback
+6. **Check First**: Every target is checked before any file is written; a
+   read-only file, or a directory that cannot take the temp file, fails the
+   run with no file written
+7. **Format Preservation**: Original file formats and delimiters are preserved during writeback
+
+### Transactions
+
+`BEGIN` / `COMMIT` (`END`) / `ROLLBACK` are handled in `vm/mod.rs` before
+compilation and act on the `Database` directly. The undo state is
+copy-on-write: `BEGIN` records only the modified-table set; the first
+`get_table_mut` of a table inside the transaction stores a deep copy of it
+(rows, source bytes, layout), `remove_table` moves the dropped table into
+the undo state, and `add_table` records that the name did not exist.
+`ROLLBACK` puts every recorded table back and restores the modified set;
+`COMMIT` discards the undo state. A transaction that only reads copies
+nothing, and one that changes a table copies it once. A transaction still
+open at the end of the run (or REPL session) is rolled back before writeback.
+Savepoints are not supported.
 
 This design ensures that users can experiment with data manipulations while maintaining the integrity of source files. The verbose mode (`-v`) provides additional confirmation about whether changes were saved or not.
 
@@ -177,15 +204,20 @@ lasts only for that statement.
 
 ### File Handlers
 
-- `FileHandler` (`file_handler.rs`) resolves `[table_name=]path` operands,
-  chooses the delimiter (`-F`, else comma for `.csv`, no extension and stdin,
-  tab otherwise) and the header policy (`.csv` and stdin: header always;
-  other files: headerless guess and `#` comments; `--no-header`/`--tabledef`:
-  no header), and routes writeback.
+- `FileHandler` (`file_handler.rs`) resolves `[table_name=]path` operands
+  (`parse_table_spec`: the prefix names the table only when it is
+  identifier-like), records each file's canonical absolute path (so the same
+  file cannot be loaded twice, and REPL `.cd` does not change what `.save`
+  writes), chooses the delimiter (`-F`, else tab for `.tsv`/`.tab`, comma for
+  everything else and stdin) and the header policy (the first line is the
+  header unless `--no-header`/`--tabledef`; `#` lines are comments only
+  under `--comments`), and routes writeback.
 - `csv_handler.rs` does all parsing and writing. Every format is read by the
   `csv` crate with one configuration (RFC 4180 quoting, multi-line quoted
-  fields, no trimming, BOM stripped, ragged rows padded), and written back by
-  a small RFC 4180 record writer that also quotes fields with leading or
+  fields, no trimming, ragged rows padded, an unclosed quote is an error,
+  header names must be non-empty and unique ignoring case). Untouched rows
+  are written back as their source bytes; changed and inserted rows by a
+  small RFC 4180 record writer that also quotes fields with leading or
   trailing whitespace.
 
 ## File I/O Patterns
@@ -205,7 +237,9 @@ This pattern provides several safety guarantees:
 - **Disk Full Protection**: If the disk runs out of space, the original file is preserved
 - **All-or-Nothing Semantics**: Either the complete new file exists, or the original file is unchanged
 
-The temporary file is created in the same directory as the target file to ensure the rename operation is atomic (same filesystem requirement on POSIX systems). A symlinked path is resolved first, so the link's target is rewritten and the link itself is left in place.
+The temporary file is created in the same directory as the target file to ensure the rename operation is atomic (same filesystem requirement on POSIX systems). A symlinked path is resolved first, so the link's target is rewritten and the link itself is left in place. The rename replaces the directory entry, so a file with several hard links is split from them: the other links keep the old contents.
+
+Because the rename would replace even a read-only file, `csv_handler::check_writable` runs for every target before anything is written: a file whose mode is read-only, or that cannot be opened for appending, is refused (`cannot write 'x.csv': file is read-only`), and a temp file is created and removed in its directory to prove the rename can happen.
 
 ### Delimiter Consistency
 
@@ -223,17 +257,26 @@ Sqawk supports two primary output modes:
 1. **Stdout Output**: Query results (SELECT) are written to standard output using
    the `-F` delimiter, or comma when `-F` was not given. This is independent of
    the delimiter a source table was parsed with
-2. **File Writeback**: Modified tables (via INSERT, UPDATE, DELETE) are written back to their source files when `--write` is specified, preserving that file's delimiter
+2. **File Writeback**: Modified tables (via INSERT, UPDATE, DELETE, ALTER TABLE, TRUNCATE) are written back to their source files when `--write` is specified, preserving that file's delimiter
 
 For stdout output, the data is streamed directly without buffering concerns. For file writeback, the atomic write pattern described above is used.
 
 ## Storage
 
 Every table holds its rows in memory as owned values (`Vec<Row>`). Files are
 read whole and parsed by the `csv` crate -- RFC 4180 quoting, quoted fields
-spanning lines, no trimming, UTF-8 byte order mark stripped -- and each cell's
-type is inferred with a rule that round-trips (see "Type inference" below), so
-an unmodified cell is written back byte for byte.
+spanning lines, no trimming, UTF-8 byte order mark set aside -- and each
+cell's type is inferred with a rule that round-trips (see "Type inference"
+below), so an unmodified cell of a changed row keeps its text.
+
+A loaded table also keeps the file's bytes (`RawRows` in `table.rs`): the
+whole file in one buffer plus a start/end offset per row, parallel to the
+rows. That costs the file's size plus 16 bytes per row. A row keeps its
+offsets while no statement changes it -- an UPDATE that sets every column to
+an identical value (same type, same value) counts as no change -- and is
+written back as exactly those bytes. INSERT adds rows without source bytes;
+DELETE drops a row's offsets with it; ALTER TABLE ADD COLUMN and TRUNCATE
+discard the source bytes of every row (every line changes).
 
 Standard input (the `-` operand) is read the same way, directly from the pipe.
 The resulting table has no file path, so `--write` reports it as unwritable
@@ -244,35 +287,38 @@ rather than writing anywhere.
 - empty field: NULL
 - canonical integer (`42`, `-7`, where formatting the number back gives the
   same text): INTEGER
-- canonical float (`1.5`, `-0.25`): REAL
-- anything else, verbatim: TEXT -- including `007`, `1.50`, `1e3`, ` 42`,
-  `true`, `NaN`
+- canonical float (`1.5`, `-0.25`, `2.0`, `1.0e+20`, where printing the real
+  the way SQLite's `CAST(x AS TEXT)` does gives the same text): REAL
+- anything else, verbatim: TEXT -- including `007`, `1.50`, `1e3`, `-0.0`,
+  `0.30000000000000004`, ` 42`, `true`, `NaN`
 
 Text that spells a number still acts as one where a number is needed:
-arithmetic, `SUM`/`AVG`, comparison against a number, and sort order.
+arithmetic, `SUM`/`AVG`, every comparison (against a number or against other
+numeric text), and sort order.
 
 ### Source layout
 
 Each table records how its file was laid out -- whether it had a header line,
-whether records ended in CRLF, and whether `#` lines were comments -- and
-writeback reproduces it. Writeback also keeps the file's permissions and, for
-a symlink, rewrites the link's target and leaves the link in place.
-
-In a file read with comment skipping (any delimited file that is not `.csv`),
-each `#` comment line is kept verbatim, with its own line ending, and
-anchored to the data that follows it: comments before the header (or, in a
-headerless file, before the first row) stay at the top, comments before a
-row stay with that row, and comments after the last row stay at the end. So
-a no-op UPDATE writes such a file back byte for byte. When rows change:
+a UTF-8 byte order mark, whether records ended in CRLF, and whether `#` lines
+were comments -- and writeback reproduces it. Writeback also keeps the file's
+permissions and, for a symlink, rewrites the link's target and leaves the
+link in place. When the file's last line had no line ending and a row is
+written after it, the file's line ending is inserted first.
+
+The lines between records -- blank lines, and under `--comments` each `#`
+comment line -- are kept verbatim, with their own line endings, and anchored
+to the data that follows: lines before the header (or, in a headerless file,
+before the first row) stay at the top, lines before a row stay with that
+row, and lines after the last row stay at the end. So a no-op UPDATE writes
+any file back byte for byte. When rows change:
 
 - UPDATE replaces rows in place, so every comment stays where it was.
 - DELETE leaves a deleted row's comments where they were: in front of the
   next surviving row, or of the trailing comments. Deleting every row (or
   TRUNCATE) keeps all comments, in order.
 - INSERT appends rows after the last row and before the trailing comments.
 
-Blank lines are not kept (the reader skips them), and `#` lines in a `.csv`
-file are data, not comments.
+Without `--comments`, `#` lines are data, not comments.
 
 ### Row Identity Across Updates
 
@@ -341,8 +387,9 @@ The database engine has several architectural limitations:
 
 - **No Index Structure**: Tables are scanned in full; the only index is the
   temporary hash index an equi-join builds for its inner table
-- **No Transaction Support**: A statement's modifications are applied once it
-  completes; there is no BEGIN/COMMIT and no rollback across statements
+- **Transactions without savepoints**: BEGIN / COMMIT / ROLLBACK are
+  supported (see "Transactions" above); SAVEPOINT, RELEASE and ROLLBACK TO
+  are not
 - **Schema Flexibility**: Types are inferred per value rather than enforced
 - **No Constraints System**: Referential integrity not enforced
 - **Whole-dataset residency**: A table must fit in memory