Spring Cleaning (#308)

* Unify inference logic; add explicit column_names flag; fix stream/mmap parity regressions

* Clean up MSVC compatibility macros

* Simplify IBasicCSVParser construction

* Refactor get_col_names()

Use parse() and move to utility header

* Get rid of get_file_size()

This returned the file size of an mmap source for the MmapParser, but used ifstream. Not efficient!

* Delete sh.hpp

* Cleanup constructors for CSVReader & IBasicCSVParser

* Clean up CSVField duplication

* Figured out what was wrong with clang & CSVWriter

Also add variadic write_row() method

* Fix MSVC pedantry

* Potential fix for pull request finding 'CodeQL / Large object passed by value'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Large object passed by value'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Update writing documentation

* Fix clanker-induced error

Also fix shadowing issues

* Fix CSV writing issues

* Bump version

* Delete sh.hpp

* Reduce duplication in get_csv_head_stream()

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Author: vincentlaucsb

.github/workflows/codeql.yml

@@ -46,7 +46,7 @@ jobs:
         cd build
         cmake .. \
           -DCMAKE_BUILD_TYPE=Release \
-          -DCSV_CXX_STANDARD=17 \
+          -DCSV_CXX_STANDARD=20 \
           -DCMAKE_CXX_COMPILER=g++ \
           -DCMAKE_C_COMPILER=gcc
     

AGENTS.md

@@ -38,33 +38,7 @@ CSVReader reader(infile, format);
 
 **Testing requirement:** Use ≥500K rows to cross 10MB boundary.
 
-## Key Files
-
-| File | Contains |
-|------|----------|
-| `csv_reader.hpp` | Mmap vs stream constructors |
-| `csv_reader.cpp` | Delimiter guessing, header detection |
-| `basic_csv_parser.hpp` | Parser base class (IBasicCSVParser, MmapParser, StreamParser) |
-| `basic_csv_parser.cpp` | Chunk transitions, worker thread |
-| `raw_csv_data.hpp` | Internal parser data structures (RawCSVField, CSVFieldList, RawCSVData) |
-| `thread_safe_deque.hpp` | Producer-consumer queue for parser→main thread communication |
-| `csv_row.hpp` | Public API types (CSVField, CSVRow) |
-| `test_round_trip.cpp` | Exemplar test patterns |
-
-## Data Flow: Parser → Row API
-
-```
-Parser Thread                      Main Thread
-	↓                                  ↓
-RawCSVData (shared_ptr) ─────────────→ CSVRow
-	↓                                  ↓
-CSVFieldList → RawCSVField[]       CSVField (lazy unescaping)
-	↓
-ThreadSafeDeque<CSVRow>
-(producer-consumer queue)
-```
-
-**Thread Safety:** Parser populates `RawCSVData`, pushes `CSVRow` to `ThreadSafeDeque`, main thread pops and reads. The `CSVFieldList` uses chunked allocation (~170 fields/chunk) for cache locality. See `raw_csv_data.hpp` and `thread_safe_deque.hpp` for implementation details.
+For detailed file mapping, parser data flow, and component relationships, see `ARCHITECTURE.md` and `include/internal/ARCHITECTURE.md`.
 
 ## Common Pitfalls
 
@@ -73,18 +47,29 @@ ThreadSafeDeque<CSVRow>
 3. **Don't use uniform values:** Each column needs distinct values to detect corruption.
 4. **Don't ignore async:** Worker thread means exceptions must use `exception_ptr`.
 5. **Don't change one constructor:** Likely affects both mmap and stream paths.
-7. **Compatibility macros defined in `common.hpp` MUST be referenced only after including `common.hpp`.** Any macro (such as `CSV_HAS_CXX20`) that is defined in `common.hpp` must not be used or checked before `#include "common.hpp"` appears in the file. This ensures feature detection and conditional compilation work as intended across all supported compilers and build modes.
-8. **`CSVReader` is non-copyable and move-enabled.** Prefer explicit ownership transfer (`std::move`) or `std::unique_ptr<CSVReader>` when sharing/handing off parser ownership across APIs.
-9. **Prefer trailing underscore for private members** (for example `source_`, `leftover_`). When you touch code with mixed private-member naming styles, normalize the edited region toward trailing underscores instead of introducing more leading-underscore or unsuffixed names.
-10. **Prefer user-friendly API constraints.** Do not narrow template constraints unless required for correctness, safety, or a measured performance win. If an implementation already handles common standard-library containers/ranges correctly, keep those inputs accepted instead of over-constraining APIs for aesthetic purity.
-11. **Respect existing compile-time compatibility macros.** Keep `IF_CONSTEXPR`, `CONSTEXPR_VALUE`, and similar macros unless there is a correctness bug.
-12. **Do not replace compile-time constructs with runtime control flow to silence warnings.** Prefer smallest scoped warning suppression at the exact site (for example, local `#pragma warning(push/pop)` on MSVC) over semantic rewrites.
-13. **Opportunistic rewrites/refactors are allowed when they are safe and justified.** Keep them separated from build-fix urgency where possible, and avoid bundling unrelated churn with compiler triage unless explicitly requested.
-14. **When proposing changes that affect compile-time behavior, explain the tradeoff clearly.** Call out any impact to codegen, performance, portability, and readability before applying the change.
-15. **If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.** Provide a short justification before expanding further.
+6. **`CSVReader` is non-copyable and move-enabled.** Prefer explicit ownership transfer (`std::move`) or `std::unique_ptr<CSVReader>` when sharing/handing off parser ownership across APIs.
+7. **Prefer user-friendly API constraints.** Do not narrow template constraints unless required for correctness, safety, or a measured performance win. If an implementation already handles common standard-library containers/ranges correctly, keep those inputs accepted instead of over-constraining APIs for aesthetic purity.
+8.  **Opportunistic rewrites/refactors are allowed when they are safe and justified.** Keep them separated from build-fix urgency where possible, and avoid bundling unrelated churn with compiler triage unless explicitly requested.
+9. **When proposing changes that affect compile-time behavior, explain the tradeoff clearly.** Call out any impact to codegen, performance, portability, and readability before applying the change.
+10. **If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.** Provide a short justification before expanding further.
 
 See `tests/AGENTS.md` for test strategy, checklist, and conventions.
 
+### Rules for Coding
+1. **Use compatibility macros defined in `common.hpp`** for cross-compiler or cross-standard concerns. If it doesn't exist, consider creating one.
+2. **Compatibility macros defined in `common.hpp` MUST be referenced only after including `common.hpp`** to ensure correctness.
+3. **Prefer compile time control flow and assertions where possible**. For example, if a branch may be safely written with `if constexpr`, then use the `IF_CONSTEXPR` macro (from `common.hpp`) to ensure C++11 compatibility while ensuring optimal control flow for C++17 and later users.
+   1. **If this causes compiler warnings, always silence the compiler. Do not revert to unnecessary runtime flow.**
+4. **Prefer trailing underscore for private members** (for example `source_`, `leftover_`). When you touch code with mixed private-member naming styles, normalize the edited region toward trailing underscores instead of introducing more leading-underscore or unsuffixed names.
+5. **Apply the 5/2 anti-duplication rule.**
+	1. If equivalent behavior exists in 2 or more code paths and each copy is about 5+ meaningful lines, extract a shared helper.
+	2. If duplication is intentionally kept, add a brief comment explaining why (for example performance, API boundary, or template constraints).
+	3. For behavior-sensitive duplicated logic, keep at least one regression test that exercises each path (for example mmap and stream via separate Catch2 `SECTION`s).
+6. If a class has both a `.hpp` and `.cpp` file, put methods inside the `.cpp` and prefix the definition with `CSV_INLINE` to ensure proper single-header compilation (the macro is `inline` in the generated single-header and empty otherwise). Exceptions:
+   - **Templates must stay in `.hpp`** — the compiler needs the definition at instantiation time. `init_from_stream` is the standing example.
+   - **Trivial one-liner accessors** may be unconditionally `inline` in the header when the call overhead is measurable and the body will never change.
+   - **Consolidation:** If a `.cpp` would be under ~100 lines *and* the split causes excessive comment duplication between the two files, prefer a single `.hpp` with definitions marked `inline` (free functions and methods alike). Do not use `CSV_INLINE` for consolidated definitions — `CSV_INLINE` expands to empty in multi-header mode, which would produce ODR violations across TUs. Do not consolidate just for brevity — only when duplication is the dominant cost.
+
 ### Rules for Comments
 1. **Always update or remove incorrect comments.**
 2. **Don't reference internal functions in public API comments.** Public API docs should describe user-visible behavior and contracts; internal helper/function details belong in internal docs.

ARCHITECTURE.md

@@ -14,6 +14,7 @@ Operational/testing guidance:
 
 Notes:
 - Internal architecture content lives under include/internal to stay close to implementation.
+- Detailed file map, parser data flow, and component relationship diagrams are maintained in include/internal/ARCHITECTURE.md.
 - Queue synchronization details are maintained only in THREADSAFE_DEQUE_DESIGN.md to avoid duplication.
 - Always update or remove incorrect comments.
 - Public API comments should remain user-facing and avoid references to internal helper/function details.
@@ -27,4 +28,5 @@ Notes:
 - Opportunistic rewrites are acceptable when safe/justified, but should be kept separate from urgent compiler triage unless requested.
 - When changing compile-time behavior, explicitly document tradeoffs (codegen, performance, portability, readability).
 - If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.
+- Apply the 5/2 anti-duplication rule: if equivalent behavior exists in 2+ code paths and each copy is ~5+ meaningful lines, extract a shared helper; if duplication remains, document why and keep regression coverage for each path.
 

CLAUDE.md

@@ -21,13 +21,8 @@
 - Exceptions propagate via `std::exception_ptr`
 - Tests must use ≥500K rows to cross chunk boundary
 
-## Key Files
-- `csv_reader.hpp` — mmap vs stream constructors
-- `basic_csv_parser.hpp` — MmapParser, StreamParser implementations
-- `basic_csv_parser.cpp` — chunk transitions, worker thread
-- `raw_csv_data.hpp` — RawCSVField, CSVFieldList, RawCSVData
-- `thread_safe_deque.hpp` — producer-consumer queue
-- `csv_row.hpp` — CSVField, CSVRow public API
+## Architecture Detail
+- File mapping, parser data flow, and component relationships are maintained in `ARCHITECTURE.md` and `include/internal/ARCHITECTURE.md`
 
 ## Common Pitfalls
 - Always test both mmap and stream paths
@@ -48,6 +43,8 @@
 - **Opportunistic rewrites are allowed when safe and justified** — avoid mixing unrelated churn into urgent compiler triage unless requested
 - **Explain compile-time tradeoffs explicitly** — when a change affects compile-time behavior, call out impact on codegen/perf/portability/readability
 - **Scope guard for build fixes** — if a fix grows beyond roughly 3 files or 60 changed lines, pause and confirm scope with justification
+- **Apply the 5/2 anti-duplication rule** — if equivalent behavior exists in 2+ code paths and each copy is ~5+ meaningful lines, extract a shared helper; if duplication remains, document why; keep at least one regression test that exercises each path
+- **Non-trivial methods go in `.cpp` with `CSV_INLINE`** — `CSV_INLINE` is `inline` in the generated single-header and empty otherwise; omitting it causes ODR violations. Exceptions: templated methods must stay in `.hpp` (`init_from_stream` is the standing example); trivial one-liner accessors may stay `inline` in the header when call overhead matters. Consolidate into a single `.hpp` only when the `.cpp` would be under ~100 lines *and* the split causes excessive comment duplication — consolidated definitions (free functions and methods alike) must use `inline`, not `CSV_INLINE`, to avoid ODR violations across TUs.
 
 ## Tests
 See `tests/AGENTS.md` for full test strategy, checklist, and conventions.

Doxyfile

@@ -2037,7 +2037,8 @@ INCLUDE_FILE_PATTERNS  =
 # recursively expanded use the := operator instead of the = operator.
 # This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
 
-PREDEFINED             = DOXYGEN_SHOULD_SKIP_THIS
+PREDEFINED             = DOXYGEN_SHOULD_SKIP_THIS \
+                         CSV_HAS_CXX20=1
 
 # If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
 # tag can be used to specify a list of macro names that should be expanded. The

README.md

@@ -385,6 +385,12 @@ format.delimiter('\t')
 // Alternatively, we can use format.delimiter({ '\t', ',', ... })
 // to tell the CSV guesser which delimiters to try out
 
+// Inference contract:
+// - Single delimiter: delimiter is fixed (no delimiter inference)
+// - Header row is still inferred unless you explicitly call header_row(...), no_header(),
+//   or provide column_names(...)
+// - Multiple delimiters: delimiter inference is enabled
+
 CSVReader reader("weird_csv_dialect.csv", format);
 
 for (auto& row: reader) {
@@ -577,6 +583,8 @@ for (auto& row : df) {
 ```
 
 ### Writing CSV Files
+*For a more in-depth guide, check out the [Doxygen page on CSV writing](https://vincentlaucsb.github.io/csv-parser/csv_writing_guide.html).*
+
 Writing CSVs is powered by the generic `DelimWriter`, with helpful factory functions like `make_csv_writer()` and `make_tsv_writer()` that cut down on boilerplate.
 
 ```cpp
@@ -599,8 +607,12 @@ writer << vector<string>({ "A", "B", "C" })
     << deque<string>({ "I'm", "too", "tired" })
     << list<string>({ "to", "write", "documentation." });
 
+// Uses compile time templates
 writer << array<string, 3>({ "The quick brown", "fox", "jumps over the lazy dog" });
-writer << make_tuple(1, 2.0, "Three");
+writer << make_tuple(1, 2.0, "Three", "Quatro");
+
+// write_row() does everything operator<< does and then some
+writer.write_row(67, "six", "seven", 6.7, "mogged");
 ...
 ```
 

docs/source/Doxy.md

@@ -28,7 +28,7 @@ For quick examples, go to this project's [GitHub page](https://github.com/vincen
    * csv::get_col_pos(): Returns the zero-based index of a named column
 
  #### See also
- [Dealing with Variable Length CSV Rows](md_docs_source_variable_row_lengths.html)
+ [Dealing with Variable Length CSV Rows](variable_row_lengths.html)
 
  #### Working with parsed data
  * csv::CSVRow: \copybrief csv::CSVRow
@@ -97,7 +97,7 @@ column extraction, editing, and grouping.
    * csv::CSVStat::get_col_names()
 
 ### CSV Writing
-The [CSV Writing Guide](\ref md_docs_2source_2csv__writing) contains a 
+The [CSV Writing Guide](@ref csv_writing_guide) contains a 
 high-level overview of writing CSVs.
 
  * csv::make_csv_writer(): Construct a csv::CSVWriter
@@ -126,4 +126,5 @@ experimenting should follow these guidelines:
      create separate threads to process each column
  * csv::CSVRow may be safely processed from multiple threads
  * csv::CSVField objects should only be read from one thread at a time
-   * **Note**: csv::CSVRow::operator[]() produces separate copies of `csv::CSVField` objects
+   * **Note**: csv::CSVRow::operator[]() produces separate copies of `csv::CSVField` objects
+

docs/source/csv_writing.md

@@ -1,3 +1,5 @@
+@page csv_writing_guide CSV Writing Guide
+
 # CSV Writing Guide
 
 This page summarizes write-side APIs and practical usage patterns for emitting
@@ -18,13 +20,21 @@ Any row-like container of string-convertible values can be streamed directly.
 
 \snippet tests/test_write_csv.cpp CSV Writer Example
 
-## Writing Tuples and Custom Types
+### Writing Tuples and Custom Types
 
 `DelimWriter` can also serialize tuples and custom types that provide a string
 conversion.
 
 \snippet tests/test_write_csv.cpp CSV Writer Tuple Example
 
+## Using `write_row()`
+
+The `write_row()` method can be used to write rows with arbitrary fields and mixed types without having to construct a container first.
+
+Through the magic of SFINAE, `write_row()` also supports any of the operations of `operator<<`.
+
+\snippet tests/test_write_csv.cpp CSV write_row Variadic Example
+
 ## Data Reordering Workflow
 
 For read-transform-write pipelines, `csv::CSVRow` supports conversion to
@@ -39,3 +49,22 @@ Typical flow:
 4. Emit with `CSVWriter`
 
 \snippet tests/test_write_csv.cpp CSV Reordering Example
+
+### C++20 Ranges Version
+
+With C++20, you can use `std::ranges::views` to elegantly reorder fields in a single expression:
+
+\snippet tests/test_write_csv.cpp CSV Ranges Reordering Example
+
+## DataFrame with Sparse Overlay
+
+When working with DataFrames, you can efficiently update specific cells without reconstructing entire rows. The overlay mechanism stores only the changed cells and writes them correctly:
+
+\snippet tests/test_write_csv.cpp DataFrame Sparse Overlay Write Example
+
+## End-to-End Round-Trip Integrity Example
+
+The following test is intentionally write-first then read/verify, but it validates
+the same data-integrity guarantee as read-transform-write user workflows.
+
+\snippet tests/test_round_trip.cpp Round Trip Distinct Field Values Example

docs/source/scientific_notation.md

@@ -1,3 +1,5 @@
+@page scientific_notation Scientific Notation Parsing
+
 # Scientific Notation Parsing
 
 This library has support for parsing scientific notation through `csv::internals::data_type()`,

docs/source/variable_row_lengths.md

@@ -1,3 +1,5 @@
+@page variable_row_lengths Dealing with Variable Length CSV Rows
+
 # Dealing with Variable Length CSV Rows
 
 `csv::CSVReader` generally assumes that most rows in a CSV are of the same length.