Generalize StreamParser; restore C++11 support; add std::ranges support (#305)

* Added StringViewStream

 * Added opt-in footgun
 * Added opt-in safety switch for StreamParser path

* Fix single header issue

* Updated comments on CSVReader ownership

* Reverted parse(). Added parse_unsafe() for non-owning SVs.

* Potential fix for pull request finding 'CodeQL / Large object passed by value'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Generalize StreamParser to support non-seekable streams

* Update basic_csv_parser.hpp

* Fix test race condition

* Update basic_csv_parser.hpp

* Add writer extensions; fix hanging

* Update csv_row.hpp

* Fix C++20 issue

* Make tests more robust

* Add C++ 11 to CI matrix

* Fix C++11 compatibility

* Fix C++11 unique_ptr issues

* Fix more C++11 issues

* Fix C++11 compiler warnings and errors

* Update common.hpp

Fix pedantic ass g++ error with std=c++11

* Version bump

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Author: vincentlaucsb

.github/workflows/cmake-multi-platform.yml

@@ -29,7 +29,7 @@ jobs:
         os: [windows-latest, ubuntu-latest]
         build_type: [Release]
         c_compiler: [gcc, cl, clang]
-        cxx_standard: [17, 20]
+        cxx_standard: [11, 17, 20]
         include:
           - os: windows-latest
             c_compiler: cl

AGENTS.md

@@ -74,7 +74,15 @@ ThreadSafeDeque<CSVRow>
 4. **Don't ignore async:** Worker thread means exceptions must use `exception_ptr`.
 5. **Don't change one constructor:** Likely affects both mmap and stream paths.
 6. **Don't delete or simplify comments** unless they are trivially obvious (e.g. `// increment i`) or factually incorrect. Comments in this codebase frequently encode concurrency invariants, non-obvious design decisions, and hard-won bug context that cannot be recovered from the code alone.
+7. **Compatibility macros defined in `common.hpp` MUST be referenced only after including `common.hpp`.** Any macro (such as `CSV_HAS_CXX20`) that is defined in `common.hpp` must not be used or checked before `#include "common.hpp"` appears in the file. This ensures feature detection and conditional compilation work as intended across all supported compilers and build modes.
 7. **Don't reference internal functions in public API comments.** Public API docs should describe user-visible behavior and contracts; internal helper/function details belong in internal docs.
-8. **`CSVReader` is non-copyable and non-movable.** The preferred idiom for sharing or transferring a reader is `std::unique_ptr<CSVReader>`. Document this at any API surface that might tempt callers to copy or move.
+8. **`CSVReader` is non-copyable and move-enabled.** Prefer explicit ownership transfer (`std::move`) or `std::unique_ptr<CSVReader>` when sharing/handing off parser ownership across APIs.
+9. **Prefer trailing underscore for private members** (for example `source_`, `leftover_`). When you touch code with mixed private-member naming styles, normalize the edited region toward trailing underscores instead of introducing more leading-underscore or unsuffixed names.
+10. **Prefer user-friendly API constraints.** Do not narrow template constraints unless required for correctness, safety, or a measured performance win. If an implementation already handles common standard-library containers/ranges correctly, keep those inputs accepted instead of over-constraining APIs for aesthetic purity.
+11. **Respect existing compile-time compatibility macros.** Keep `IF_CONSTEXPR`, `CONSTEXPR_VALUE`, and similar macros unless there is a correctness bug.
+12. **Do not replace compile-time constructs with runtime control flow to silence warnings.** Prefer smallest scoped warning suppression at the exact site (for example, local `#pragma warning(push/pop)` on MSVC) over semantic rewrites.
+13. **Opportunistic rewrites/refactors are allowed when they are safe and justified.** Keep them separated from build-fix urgency where possible, and avoid bundling unrelated churn with compiler triage unless explicitly requested.
+14. **When proposing changes that affect compile-time behavior, explain the tradeoff clearly.** Call out any impact to codegen, performance, portability, and readability before applying the change.
+15. **If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.** Provide a short justification before expanding further.
 
 See `tests/AGENTS.md` for test strategy, checklist, and conventions.

ARCHITECTURE.md

@@ -16,4 +16,13 @@ Notes:
 - Internal architecture content lives under include/internal to stay close to implementation.
 - Queue synchronization details are maintained only in THREADSAFE_DEQUE_DESIGN.md to avoid duplication.
 - Public API comments should remain user-facing and avoid references to internal helper/function details.
+- Private member naming should prefer trailing underscores; when editing mixed-style code, normalize the touched region toward that convention.
+- Compatibility macros defined in `common.hpp` must only be referenced after including `common.hpp`. See AGENTS.md and CLAUDE.md for details.
+- API constraints should be user-friendly: do not over-constrain templates unless needed for correctness, safety, or a measured performance win.
+- `CSVReader` is intentionally non-copyable and move-enabled; use explicit ownership transfer patterns (`std::move`, `std::unique_ptr`) at API boundaries.
+- Respect existing compile-time compatibility macros (`IF_CONSTEXPR`, `CONSTEXPR_VALUE`, etc.) unless correctness requires change.
+- Avoid semantic rewrites to silence compiler warnings; prefer precise scoped suppression where appropriate.
+- Opportunistic rewrites are acceptable when safe/justified, but should be kept separate from urgent compiler triage unless requested.
+- When changing compile-time behavior, explicitly document tradeoffs (codegen, performance, portability, readability).
+- If a build fix appears to require more than ~3 files or ~60 changed lines, pause and confirm scope first.
 

CLAUDE.md

@@ -36,8 +36,16 @@
 - Exceptions from worker thread need `exception_ptr`
 - Changes to one constructor likely affect both paths
 - **Do not delete or simplify comments** unless trivially obvious or factually wrong — comments encode concurrency invariants and bug history
+- **Compatibility macros defined in `common.hpp` MUST be referenced only after including `common.hpp`.** Any macro (such as `CSV_HAS_CXX20`) that is defined in `common.hpp` must not be used or checked before `#include "common.hpp"` appears in the file. This ensures feature detection and conditional compilation work as intended across all supported compilers and build modes.
 - **Do not reference internal functions in public API comments** — public API docs should remain user-facing; internal details belong in internal docs
-- **`CSVReader` is non-copyable and non-movable** — the preferred sharing/transfer idiom is `std::unique_ptr<CSVReader>`
+- **`CSVReader` is non-copyable and move-enabled** — prefer explicit ownership transfer (`std::move`) or `std::unique_ptr<CSVReader>` when handing off parser ownership
+- **Prefer trailing underscore for private members** — when touching mixed-style code, normalize the edited region toward names like `source_` and `leftover_`
+- **Prefer user-friendly API constraints** — do not narrow template constraints unless required for correctness, safety, or a measured performance win; if common containers/ranges already work, keep them accepted
+- **Respect compile-time compatibility macros** — keep constructs like `IF_CONSTEXPR` and `CONSTEXPR_VALUE` unless there is a correctness bug
+- **Do not rewrite compile-time logic to silence warnings** — prefer tightly scoped suppression at the exact site when needed
+- **Opportunistic rewrites are allowed when safe and justified** — avoid mixing unrelated churn into urgent compiler triage unless requested
+- **Explain compile-time tradeoffs explicitly** — when a change affects compile-time behavior, call out impact on codegen/perf/portability/readability
+- **Scope guard for build fixes** — if a fix grows beyond roughly 3 files or 60 changed lines, pause and confirm scope with justification
 
 ## Tests
 See `tests/AGENTS.md` for full test strategy, checklist, and conventions.

README.md

@@ -22,7 +22,7 @@
       - [Avoid cloning with FetchContent](#avoid-cloning-with-fetchcontent)
   - [Features \& Examples](#features--examples)
     - [Reading an Arbitrarily Large File (with Iterators)](#reading-an-arbitrarily-large-file-with-iterators)
-      - [Memory-Mapped Files vs. Streams](#memory-mapped-files-vs-streams)
+    - [Memory-Mapped I/O and Streams](#memory-mapped-io-and-streams)
     - [Indexing by Column Names](#indexing-by-column-names)
     - [Numeric Conversions](#numeric-conversions)
     - [Converting to JSON](#converting-to-json)
@@ -33,14 +33,19 @@
     - [Parsing an In-Memory String](#parsing-an-in-memory-string)
     - [DataFrames for Random Access and Updates](#dataframes-for-random-access-and-updates)
     - [Writing CSV Files](#writing-csv-files)
+      - [C++20 Ranges: Efficient writing for `CSVRow`, `DataFrameRow`, and STL containers](#c20-ranges-efficient-writing-for-csvrow-dataframerow-and-stl-containers)
 
 ## Motivation
-There's plenty of other CSV parsers in the wild, but I had a hard time finding what I wanted. Inspired by Python's `csv` module, I wanted a library with **simple, intuitive syntax**. Furthermore, I wanted support for special use cases such as calculating statistics on very large files. Thus, this library was created with these following goals in mind.
+I wanted a CSV library that was fast and reliable without forcing you into either:
+ * A 1990s C-style API
+ * A high-level wrapper that murders `malloc()` and your memory cache
+
+This library tries to be **fast for developers** and **fast for your computer**.
 
 ### Performance and Memory Requirements
-A high-performance CSV parser lets you take advantage of large datasets efficiently. This library combines SIMD-accelerated parsing, memory-mapped I/O, careful memory layout, minimal allocation, and background parsing to process large CSV files quickly, even when they exceed available RAM.
+This library combines SIMD-accelerated parsing, memory-mapped I/O, careful memory layout, minimal allocation, and background parsing to process large CSV files quickly, even when they exceed available RAM.
 
-In fact, [according to Visual Studio's profiler](https://github.com/vincentlaucsb/csv-parser/wiki/Microsoft-Visual-Studio-CPU-Profiling-Results) this
+[According to Visual Studio's profiler](https://github.com/vincentlaucsb/csv-parser/wiki/Microsoft-Visual-Studio-CPU-Profiling-Results) this
 CSV parser **spends almost 90% of its CPU cycles actually reading your data** as opposed to getting hung up in hard disk I/O or pushing around memory.
 
 #### Show me the numbers
@@ -52,7 +57,7 @@ All benchmarks shown are warm cache runs to focus on parser/CPU performance rath
 
 #### Chunk Size Tuning
 
-By default, the parser reads CSV data in 10MB chunks. This balance was determined through empirical testing to optimize throughput while minimizing memory overhead and thread synchronization costs.
+By default, the parser reads CSV data in 10MB chunks. This balance was determined through empirical testing to optimize throughput while minimizing memory overhead and thread synchronization costs, but feel free to experiment and measure with different numbers yourself.
 
 If you encounter rows larger than the chunk size, pass a custom `CSVFormat` with `chunk_size()`:
 
@@ -65,13 +70,11 @@ for (auto& row : reader) {
 }
 ```
 
-**Tuning guidance:** The default 10MB provides good balance for typical workloads. Smaller chunks (e.g., 500KB) increase thread overhead without meaningful memory savings. Larger chunks (e.g., 100MB+) reduce thread coordination overhead but consume more memory and delay the first row. Feel free to experiment and measure with your own hardware and data patterns.
-
 ### Robust Yet Flexible
 #### RFC 4180 and Beyond
 This CSV parser is much more than a fancy string splitter, and parses all files following [RFC 4180](https://www.rfc-editor.org/rfc/rfc4180.txt).
 
-However, in reality we know that RFC 4180 is just a suggestion, and there's many "flavors" of CSV such as tab-delimited files. Thus, this library has:
+However, in reality we know that RFC 4180 is just a suggestion, so this library has:
  * Automatic delimiter guessing
  * Ability to ignore comments in leading rows and elsewhere
  * Ability to handle rows of different lengths
@@ -81,32 +84,34 @@ By default, rows of variable length are silently ignored, although you may elect
 
 #### Encoding
 This CSV parser is encoding-agnostic and will handle ANSI and UTF-8 encoded files.
-It does not try to decode UTF-8, except for detecting and stripping UTF-8 byte order marks.
+It does not try to decode UTF-8, except for detecting and stripping UTF-8 byte order marks (BOM).
 
 ### Well Tested
 This CSV parser has:
  * An extensive Catch2 test suite
  * Tests of various CMake and non-CMake builds across g++, clang, MSVC, and MinGW
  * Address, thread safety, and undefined behavior checks with ASan, TSan, and Valgrind (see [GitHub Actions](https://github.com/vincentlaucsb/csv-parser/actions))
-
+  
 #### Bug Reports
-Found a bug? Please report it! This project welcomes **genuine bug reports brought in good faith**:
- * ✅ Crashes, memory leaks, data corruption, race conditions
- * ✅ Incorrect parsing of valid CSV files
- * ✅ Performance regressions in real-world scenarios
- * ✅ API issues that affect **practical, real-world use cases**
 
-When reporting integration or compiler issues, please state which library form you are using:
- * Single-header
- * Unamalgamated headers/library (`include/` with your own build system, CMake, etc.)
+I welcome genuine bug reports brought in good faith. This includes:
+
+- Crashes, memory leaks, data corruption, or race conditions
+- Incorrect parsing of valid CSV files
+- Performance regressions on real-world data
+- API issues that affect practical use cases
 
-Please keep reports grounded in real use cases—no contrived edge cases or philosophical debates about API design, thanks!
+When reporting compiler or integration issues, please mention which form of the library you're using:
+- Single-header
+- Regular headers + your own build system
+- CMake
 
-**Design Note:** `CSVReader` uses `std::input_iterator_tag` for single-pass streaming of arbitrarily large files. If you need multi-pass iteration or random access, copy rows to a `std::vector` first. This is by design, not a bug.
+**Note:** Please keep reports focused on real-world problems. 
+Questions about extremely edge-case behavior (e.g. "what should `,,,` return?") do not belong in the issue tracker.
 
 ## Documentation
 
-In addition to the [Features & Examples](#features--examples) below, a [fully-fledged online documentation](https://vincentlaucsb.github.io/csv-parser/) contains more examples, details, interesting features, and instructions for less common use cases.
+In addition to the [Features & Examples](#features--examples) below, an [extensive documentation site](https://vincentlaucsb.github.io/csv-parser/) contains more examples, details, interesting features, and instructions for less common use cases.
 
 ## Sponsors
 If you use this library for work, please [become a sponsor](https://github.com/sponsors/vincentlaucsb). Your donation
@@ -121,7 +126,7 @@ This library was developed with Microsoft Visual Studio and is compatible with >
 All of the code required to build this library, aside from the C++ standard library, is contained under `include/`.
 
 ### C++ Version
-While C++17 is recommended, C++11 is the minimum version required. This library makes extensive use of string views, and uses
+While C++20 is recommended, C++11 is the minimum version required. This library makes extensive use of string views, and uses
 [Martin Moene's string view library](https://github.com/martinmoene/string-view-lite) if `std::string_view` is not available.
 
 This library requires C++ exceptions to be enabled (for example, do not compile with `-fno-exceptions`).
@@ -226,40 +231,18 @@ while (reader.read_row(row)) {
 ...
 ```
 
-#### Memory-Mapped Files vs. Streams
-By default, passing in a file path string to the constructor of `CSVReader`
-causes memory-mapped IO to be used. In general, this option is the most
-performant.
-
-However, `std::ifstream` may also be used as well as in-memory sources via `std::stringstream`.
-
-**Note**: Currently CSV guessing only works for memory-mapped files. The CSV dialect
-must be manually defined for other sources.
-
 **⚠️ IMPORTANT - Iterator Type and Memory Safety**:  
 `CSVReader::iterator` is an **input iterator** (`std::input_iterator_tag`), NOT a forward iterator.
-This design enables streaming large CSV files (50+ GB) without loading them entirely into memory.
+This design enables streaming large CSV files (50+ GB) without loading them entirely into memory, but may fail with some standard algorithms that require forward iterators.
 
-**Why Forward Iterator Algorithms Don't Work**:
-- As the iterator advances, underlying data chunks are automatically freed to bound memory usage
-- Algorithms like `std::max_element` require ForwardIterator semantics (multi-pass, hold multiple positions)
-- Using such algorithms directly on `CSVReader::iterator` will cause **heap-use-after-free** when the
-  algorithm tries to access iterators pointing to already-freed data chunks
-- While it may appear to work with small files that fit in a single chunk, it WILL fail with larger files
+If you need to get around this, I suggest either loading all rows into an STL container, e.g. `std::vector<CSVRow>`, or using the `DataFrame` class which supports row and column random access.
 
-**✅ Correct Approach for ForwardIterator Algorithms**:
-```cpp
-// Copy rows to vector first (enables multi-pass iteration)
-CSVReader reader("large_file.csv");
-std::vector<CSVRow> rows(reader.begin(), reader.end());
-
-// Now safely use any algorithm requiring ForwardIterator
-auto max_row = std::max_element(rows.begin(), rows.end(), 
-    [](const CSVRow& a, const CSVRow& b) { 
-        return a["salary"].get<double>() < b["salary"].get<double>(); 
-    });
-```
+### Memory-Mapped I/O and Streams
+When passing in a file path to `CSVReader`, memory-mapped I/O is used as it is the most performant.
+
+However, most finite steams implementing `std::istream`, such as `std::stringstream` and `std::ifstream` are supported as well as non-seekable streams. `CSVReader` is capable of taking a stream by reference, although it is recommended to pass in an owning `std::unique_ptr<std::istream>` for memory safety.
 
+Both memory-mapped and `std::istream` paths benefit from having a background parsing thread, unless disabled.
 
 ```cpp
 CSVFormat format;
@@ -483,7 +466,7 @@ for (auto& r: rows) {
 
 ### DataFrames for Random Access and Updates
 
-For files that fit comfortably in memory, `DataFrame` provides fast and powerful keyed access, in-place updates, and grouping operations—all built on the same high-performance parser. It uses the same parsing pipeline as `CSVReader` but retains the results in memory for random access.
+For files that fit comfortably in memory, `DataFrame` provides fast and powerful keyed access, in-place updates, and grouping operations—all built on the same high-performance parser. It uses the same parsing pipeline as `CSVReader` but retains the results in memory for both row-wise and column-wise random access.
 
 **Creating a DataFrame with Keyed Access**
 ```cpp
@@ -590,17 +573,8 @@ for (auto& row : df) {
 }
 ```
 
-**When to Use DataFrame vs. CSVReader:**
-- **Use CSVReader** for: Large files (>1GB), streaming pipelines, minimal memory footprint
-- **Use DataFrame** for: Files that fit in RAM, frequent lookups/updates, grouping operations, data that needs random access
-
-**When Not to Use DataFrame:**
-- Extremely large files that do not fit in RAM
-- Streaming pipelines where you only need single-pass access
-
-Both options deliver the same parsing performance—DataFrame simply keeps the results in memory for convenience.
-
 ### Writing CSV Files
+Writing CSVs is powered by the generic `DelimWriter`, with helpful factory functions like `make_csv_writer()` and `make_tsv_writer()` that cut down on boilerplate.
 
 ```cpp
 # include "csv.hpp"
@@ -627,5 +601,9 @@ writer << make_tuple(1, 2.0, "Three");
 ...
 ```
 
-You can pass in arbitrary types into `DelimWriter` by defining a conversion function
-for that type to `std::string`.
+You can pass in arbitrary types into `DelimWriter` by defining a conversion function for that type to `std::string`.
+
+#### C++20 Ranges: Efficient writing for `CSVRow`, `DataFrameRow`, and STL containers
+If compiling with C++20 or later, the `DelimWriter` uses efficient `std::ranges` over string views for zero-copy writing.
+
+You can still serialize `CSVRow` or `DataFrameRow` in older versions, but you will have to use the `std::vector<std::string>()` conversion operator.