Fixes

CLAUDE.md

@@ -56,7 +56,10 @@ SQL execution uses a bytecode VM inspired by SQLite's architecture:
   list, WHERE, HAVING, ORDER BY and SET clauses alike, plus `NameCtx`, which
   carries the FROM sources a name can resolve against
 - `compiler_aggregate.rs`: aggregates, GROUP BY, HAVING
-- `compiler_join.rs`: joins
+- `compiler_from.rs`: the FROM planner used by every SELECT -- tables, comma
+  lists and join chains become nested loop levels (ON per level, WHERE after
+  outer-join NULL-extension, hash join on equi-conditions); the projection /
+  aggregate stage is a body callback emitted at the innermost level
 - `compiler_dml.rs`: INSERT / UPDATE / DELETE
 - `compiler_ddl.rs`: CREATE / DROP / ALTER / TRUNCATE
 - `compiler_window.rs`: window functions
@@ -69,8 +72,11 @@ effects of the ones before it.
 ### File Handling
 
 - **`file_handler.rs`**: Manages loading files into database tables
-- **`csv_handler.rs`**: Standard CSV file parsing (comma-separated)
-- **`delim_handler.rs`**: Custom delimiter support via `-F` flag (TSV, colon-separated, etc.)
+- **`csv_handler.rs`**: The single loader (`read_table`, built on the `csv`
+  crate) for every delimiter and for stdin, plus writeback. `.csv` files always
+  have a header; other files keep the header guess and `#`-comment skipping.
+  Fields load as Integer/Float only when the text round-trips exactly, so
+  untouched cells write back byte-for-byte
 
 ### SQL Features
 
@@ -103,4 +109,10 @@ Two suites carry the correctness contract:
   defects. Run with `cargo test --test mod defects -- --ignored`. Remove the
   `#[ignore]` when a defect is fixed; the list only shrinks.
 
+`tests/sqlite_diff/harness.py` is a differential tester against SQLite (via
+python3's `sqlite3` module), not part of `cargo test`. Run
+`cargo build --release && python3 -I tests/sqlite_diff/harness.py
+tests/sqlite_diff/queries.sql tests/sqlite_diff/queries2.sql`; the SUMMARY
+line counts pass / DIFF (wrong answer) / sqawk_err.
+
 Tests use `assert_cmd` for CLI testing and `tempfile` for temporary test data.

Cargo.toml

@@ -37,7 +37,6 @@ sqlparser = "0.62"
 csv = "1"
 anyhow = "1"
 regex = "1"
-memmap2 = "0.9"
 libc = "0.2"
 tempfile = "3"
 

README.md

@@ -17,7 +17,7 @@ Sqawk is an SQL-based command-line tool for processing delimiter-separated files
 - **Window Functions** - ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, and aggregates with `OVER (PARTITION BY ... ORDER BY ...)`
 - **DDL** - CREATE TABLE, CREATE TABLE AS SELECT, DROP, ALTER TABLE ADD COLUMN, TRUNCATE
 - **Expressions** - CASE, CAST, COALESCE, NULLIF, BETWEEN, IN, LIKE/ILIKE, `||`, arithmetic
-- **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--tabledef`
+- **File Formats** - CSV, TSV, and custom delimiters; headerless files via `--no-header` or `--tabledef`; unchanged cells survive `--write` byte for byte
 - **Pipelines** - Reads standard input with a `-` file operand
 - **Safe by Default** - Files unchanged unless `--write` flag is specified
 - **Interactive REPL** - Explore data interactively with `-i` flag

doc/database.md

@@ -107,23 +107,16 @@ lasts only for that statement.
 
 ### File Handlers
 
-The file handling system consists of multiple components:
-
-#### `FileHandler` Trait
-- Defines a common interface for different file format handlers
-- Provides abstraction for loading and saving tables from various file formats
-- Allows for consistent treatment of different delimiter-separated formats
-
-#### `CsvHandler` Implementation
-- Specialized handler for standard CSV files
-- Uses the csv crate for parsing and writing
-- Handles comma-separated values with standard CSV escaping rules
-
-#### `DelimHandler` Implementation
-- Handles files with custom delimiters (e.g., TSV, colon-separated, etc.)
-- Configured via the `-F` command-line option
-- Supports tab, colon, pipe, and other custom separators
-- Preserves the original delimiter and format during writeback
+- `FileHandler` (`file_handler.rs`) resolves `[table_name=]path` operands,
+  chooses the delimiter (`-F`, else comma for `.csv`, no extension and stdin,
+  tab otherwise) and the header policy (`.csv` and stdin: header always;
+  other files: headerless guess and `#` comments; `--no-header`/`--tabledef`:
+  no header), and routes writeback.
+- `csv_handler.rs` does all parsing and writing. Every format is read by the
+  `csv` crate with one configuration (RFC 4180 quoting, multi-line quoted
+  fields, no trimming, BOM stripped, ragged rows padded), and written back by
+  a small RFC 4180 record writer that also quotes fields with leading or
+  trailing whitespace.
 
 ## File I/O Patterns
 
@@ -132,23 +125,23 @@ The file handling system consists of multiple components:
 When saving modified tables back to disk (via the `--write` flag), Sqawk uses an atomic write pattern to ensure data integrity:
 
 1. **Write to Temporary File**: Data is first written to a temporary file in the same directory as the target file
-2. **Sync to Disk**: The temporary file is explicitly synced (`fsync`) to ensure data is durably stored on the physical device
-3. **Atomic Rename**: The temporary file is renamed to the target filename using an atomic rename operation
+2. **Keep Permissions**: The original file's permissions are copied onto the temporary file
+3. **Sync to Disk**: The temporary file is explicitly synced (`fsync`) to ensure data is durably stored on the physical device
+4. **Atomic Rename**: The temporary file is renamed to the target filename using an atomic rename operation
 
 This pattern provides several safety guarantees:
 
 - **Crash Safety**: If the process crashes during writing, the original file remains intact
 - **Disk Full Protection**: If the disk runs out of space, the original file is preserved
 - **All-or-Nothing Semantics**: Either the complete new file exists, or the original file is unchanged
 
-The temporary file is created in the same directory as the target file to ensure the rename operation is atomic (same filesystem requirement on POSIX systems).
+The temporary file is created in the same directory as the target file to ensure the rename operation is atomic (same filesystem requirement on POSIX systems). A symlinked path is resolved first, so the link's target is rewritten and the link itself is left in place.
 
 ### Delimiter Consistency
 
 Sqawk maintains delimiter consistency between input and output:
 
 - When a file is loaded with a specific delimiter (via `-F` flag or file extension detection), that delimiter is preserved in the table metadata
-- Query results are output using the same delimiter as the source table
 - When files are written back, the original delimiter format is preserved
 
 This ensures that TSV files remain TSV, colon-separated files remain colon-separated, and so on. The principle of least surprise: output format matches input format unless explicitly overridden.
@@ -164,76 +157,55 @@ Sqawk supports two primary output modes:
 
 For stdout output, the data is streamed directly without buffering concerns. For file writeback, the atomic write pattern described above is used.
 
-## Storage Backend Architecture
-
-Sqawk supports multiple storage backends for table row data, enabling efficient handling of different data sources:
+## Storage
 
-### Memory Storage
+Every table holds its rows in memory as owned values (`Vec<Row>`). Files are
+read whole and parsed by the `csv` crate -- RFC 4180 quoting, quoted fields
+spanning lines, no trimming, UTF-8 byte order mark stripped -- and each cell's
+type is inferred with a rule that round-trips (see "Type inference" below), so
+an unmodified cell is written back byte for byte.
 
-The `MemoryStorage` backend stores all row data in heap-allocated memory:
-- Used for tables with no backing file (`CREATE TABLE`, `CREATE TABLE AS
-  SELECT`), and for any mmap table after its first modification
-- All string values are owned (heap-allocated)
-- Supports all operations including INSERT, UPDATE, DELETE
-- Data is mutable throughout the session
+Standard input (the `-` operand) is read the same way, directly from the pipe.
+The resulting table has no file path, so `--write` reports it as unwritable
+rather than writing anywhere.
 
-### Mmap Storage (Default for on-disk files)
+### Type inference
 
-The `MmapStorage` backend uses memory-mapped files for zero-copy access:
-- Automatically used when loading files from disk
-- String values borrow directly from the mmap'd region (zero allocation)
-- Read-only by default to maintain zero-copy efficiency
-- Converts to MemoryStorage on first write operation
+- empty field: NULL
+- canonical integer (`42`, `-7`, where formatting the number back gives the
+  same text): INTEGER
+- canonical float (`1.5`, `-0.25`): REAL
+- anything else, verbatim: TEXT -- including `007`, `1.50`, `1e3`, ` 42`,
+  `true`, `NaN`
 
-### Automatic Backend Selection
+Text that spells a number still acts as one where a number is needed:
+arithmetic, `SUM`/`AVG`, comparison against a number, and sort order.
 
-The storage backend is selected automatically based on the data source:
-- **On-disk files**: Uses mmap for zero-copy loading
-- **Tables created by DDL**: Uses in-memory storage; there is no file to map
-- **Write operations**: Mmap tables convert to memory on first modification
+### Source layout
 
-Standard input (the `-` operand) is not seekable and cannot be mapped directly,
-so it is spooled to a temporary file and then loaded by the ordinary mmap path.
-The temp file handle is held for the run's duration, because the mmap borrows
-from it. The resulting table is then detached from that path, so `--write` has
-no target and reports the table as unwritable rather than rewriting a temp file
-that is about to be deleted.
+Each table records how its file was laid out -- whether it had a header line,
+whether records ended in CRLF, and whether `#` lines were comments -- and
+writeback reproduces it. Writeback also keeps the file's permissions and, for
+a symlink, rewrites the link's target and leaves the link in place.
 
 ### Row Identity Across Updates
 
 UPDATE replaces a row in place rather than deleting and re-appending it, so a
 table keeps its row order and `--write` does not reorder the user's file.
 
-### Write Operation Behavior
-
-When a table with mmap storage is modified (INSERT, UPDATE, DELETE):
-1. All rows are copied to heap memory (converting Cow::Borrowed to Cow::Owned)
-2. The storage backend switches from Mmap to Memory
-3. Subsequent operations use the in-memory copy
-4. The original file remains unchanged until `--write` is specified
-
-This design provides:
-- Zero-allocation loading for read-only queries
-- Transparent handling of write operations
-- Memory efficiency for large read-only datasets
-
 ## Performance Considerations
 
 Sqawk's in-memory database is optimized for:
 
 - **Fast Loading**: Delimiter-separated files are parsed directly into memory
-- **Zero-Copy Access**: Memory-mapped files enable direct access without allocation overhead
 - **Format Flexibility**: Support for CSV, TSV, and custom-delimited files
 - **Efficient Filtering**: WHERE clauses are applied in a single pass
 - **Low Memory Overhead**: Simple data structures minimize memory usage
 - **Zero Configuration**: No setup required, works directly with files in various formats
 
 For larger datasets, consider:
-- The entire dataset must fit in memory (or be addressable via mmap)
+- The entire dataset must fit in memory
 - Complex queries may require multiple passes over the data
-- Write operations on mmap tables trigger a copy to memory
-- Custom delimiters may have slightly different performance characteristics than standard CSV
-- Read-only operations on large files benefit most from mmap storage
 
 ## Join Implementation
 
@@ -252,35 +224,45 @@ The database engine supports combining data from multiple tables through SQL-sta
 - **ON constraints**: Supported, and the condition is a full expression, not
   just a single comparison: `ON a.id = b.id AND UPPER(a.name) = 'X'`
 - **Table aliases**: Supported (`FROM users u JOIN orders o ON u.id = o.user_id`)
-- **USING / NATURAL**: Not supported
-- **Multi-table joins**: Supports chaining 3+ tables in a single query
-- **Aggregates over joins**: Supported for inner joins, which are rewritten to
-  the equivalent comma join. Deliberately NOT applied to outer joins, since
-  moving the ON condition into WHERE would discard the NULL-extended rows an
-  outer join exists to produce
+- **USING / NATURAL**: Supported; the shared columns are merged into one
+  output column (the first non-NULL of the joined columns)
+- **Multi-table joins**: Any chain of INNER / LEFT / CROSS joins and comma
+  lists. RIGHT JOIN is supported when everything to its left is inner-joined;
+  FULL JOIN must be the last join
+- **Expressions**: The SELECT list, ON, WHERE, GROUP BY, HAVING and ORDER BY
+  of a join accept the same expressions as a single-table query
+- **Aggregates over joins**: Supported for every join type, including the
+  NULL-extended rows of outer joins
 
 ### Technical Implementation
 
-- Joins are compiled to VM bytecode for efficient execution
-- Nested loop algorithm with appropriate NULL handling for outer joins
-- Column references use table qualifiers for disambiguation in result sets
-- Type coercion rules are applied consistently in join conditions
+- One FROM planner (`src/vm/compiler_from.rs`) serves every SELECT: each
+  source table is a cursor and a loop level, outermost first
+- ON conditions are evaluated at their join's level; WHERE conditions after
+  outer-join NULL-extension, at the innermost level that has all their tables
+- Outer joins keep a per-level match flag; an unmatched row re-runs the inner
+  levels once with the inner tables on a NULL row (`NullRow`). RIGHT JOIN is a
+  LEFT JOIN with the loop order swapped; FULL JOIN adds a pass over the right
+  table for rows that never matched
+- An equality between the inner table and the outer tables becomes a hash
+  join (`HashInsert` / `HashProbe` / `HashNext`). Keys are normalized so that
+  values `=` treats as equal (1, 1.0, '1') share a bucket; NULL keys never
+  match; the full join condition is re-checked on every candidate
 
 ## Current System Limitations
 
 The database engine has several architectural limitations:
 
-- **No Index Structure**: All operations perform full table scans
+- **No Index Structure**: Tables are scanned in full; the only index is the
+  temporary hash index an equi-join builds for its inner table
 - **No Transaction Support**: A statement's modifications are applied once it
   completes; there is no BEGIN/COMMIT and no rollback across statements
 - **Schema Flexibility**: Types are inferred per value rather than enforced
 - **No Constraints System**: Referential integrity not enforced
-- **Whole-dataset residency**: A table must fit in memory, or be addressable
-  via mmap
+- **Whole-dataset residency**: A table must fit in memory
 
 Unsupported SQL of note: CTEs (`WITH`), explicit window frames
-(`ROWS BETWEEN ...`), `USING`/`NATURAL` joins, and aggregates over outer
-joins.
+(`ROWS BETWEEN ...`), parenthesized joins, and window functions over joins.
 
 ---
 

doc/sql_reference.md

@@ -10,7 +10,12 @@
 | `BOOLEAN` | `BOOL` |
 | `NULL` | |
 
-Type inference on load: Integer → Float → Boolean → String. Empty values become NULL.
+Type inference on load, per cell: an empty field is NULL; a canonical integer
+(`42`, `-7` -- printing the number gives back the same text) is INTEGER; a
+canonical float (`1.5`, `-0.25`) is REAL; everything else is TEXT, verbatim and
+untrimmed -- including `007`, `1.50`, `1e3`, ` 42`, `true` and `NaN`. Booleans
+are never inferred. The rule guarantees that every loaded cell prints, and is
+written back, exactly as it was read.
 
 ## SELECT
 
@@ -145,12 +150,27 @@ SELECT * FROM t1 LEFT JOIN t2 ON t1.id = t2.id
 SELECT * FROM t1 RIGHT JOIN t2 ON t1.id = t2.id
 SELECT * FROM t1 FULL JOIN t2 ON t1.id = t2.id
 
--- Multiple joins
+-- Multiple joins, any mix of INNER / LEFT / CROSS
 SELECT * FROM t1
   JOIN t2 ON t1.id = t2.t1_id
-  JOIN t3 ON t2.id = t3.t2_id
+  LEFT JOIN t3 ON t2.id = t3.t2_id
+
+-- USING and NATURAL: the shared columns appear once in SELECT *
+SELECT * FROM t1 JOIN t2 USING (id)
+SELECT * FROM t1 NATURAL LEFT JOIN t2
 ```
 
+WHERE filters the joined rows, including the NULL-extended rows of an outer
+join (`... LEFT JOIN t2 ON ... WHERE t2.id IS NULL` is an anti-join). An
+unqualified column name must be unambiguous across the joined tables.
+
+A RIGHT JOIN whose left side already contains an outer join, and a FULL JOIN
+followed by further joins, are rejected as unsupported.
+
+Joins on an equality (`ON a.k = b.k`, or the same condition in WHERE for a
+comma or inner join) run as hash joins: the inner table is indexed once
+instead of being rescanned for every outer row.
+
 ## Subqueries
 
 ```sql
@@ -288,7 +308,8 @@ TRUNCATE TABLE name
 
 ## NULL handling
 
-An empty field reads as NULL.
+An empty field reads as NULL, and NULL is printed (and written back) as an
+empty field.
 
 Comparison follows SQL three-valued logic: any comparison involving NULL is
 UNKNOWN, and a row whose `WHERE` evaluates to UNKNOWN is not returned. This
@@ -306,8 +327,17 @@ floats and text.
 When a comparison mixes a number and a string, the string is converted to a
 number if it parses as one, and the two are compared textually otherwise. This
 matches arithmetic, so `x + '1'` and `x > '1'` agree — awk-like rather than
-strict SQL, which suits untyped delimited input.
+strict SQL, which suits untyped delimited input. Two strings compare as text.
+
+Text that spells a number (`007`, `1.50`, `1e3`, which load as text to keep
+their spelling) is used as that number by arithmetic and by `SUM` and `AVG`.
+`ORDER BY`, `GROUP BY`, `MIN` and `MAX` order such text among the numbers by
+value; values equal as numbers but spelled differently (`7` and `007`) remain
+distinct groups.
 
 ## Writeback
 
 Modifications (INSERT, UPDATE, DELETE) remain in-memory unless `--write` flag is specified.
+Cells a statement did not change are written back byte for byte; NULL is
+written as an empty field; the source file's delimiter, header line (or its
+absence), line endings and permissions are kept.