Clean up CSVFieldList

And rename it to RawCSVFieldList

Author: vincentlaucsb

AGENTS.md

@@ -2,7 +2,7 @@
 
 Architectural overview for AI assistants working with this codebase.
 
-> **Maintenance rule:** Whenever this file is changed, `CLAUDE.md` in the same directory must be updated to reflect the changes. `CLAUDE.md` is a bullet-point summary of this file and must stay in sync.
+> **Maintenance rule:** Whenever this file is changed, update both `CLAUDE.md` and `ARCHITECTURE.md` in the same directory to reflect relevant changes. `CLAUDE.md` is a bullet-point summary and `ARCHITECTURE.md` is the top-level architecture index; both must stay in sync with this guidance.
 
 ## Critical: single_include/csv.hpp Is A Shim
 

ARCHITECTURE.md

@@ -0,0 +1,18 @@
+# Architecture Index
+
+This file is a top-level index for architecture documentation.
+
+Primary architecture document:
+- include/internal/ARCHITECTURE.md
+
+Subsystem deep-dive:
+- include/internal/THREADSAFE_DEQUE_DESIGN.md
+
+Operational/testing guidance:
+- AGENTS.md
+- tests/AGENTS.md
+
+Notes:
+- Internal architecture content lives under include/internal to stay close to implementation.
+- Queue synchronization details are maintained only in THREADSAFE_DEQUE_DESIGN.md to avoid duplication.
+

CLAUDE.md

@@ -2,6 +2,9 @@
 
 > **`AGENTS.md` is the source of truth.** This file is a bullet-point summary only. Always load and follow `AGENTS.md` — it takes precedence over anything here.
 
+## Maintenance Rule
+- When `AGENTS.md` changes, update both `CLAUDE.md` and root `ARCHITECTURE.md` to keep guidance and architecture index references aligned.
+
 ## single_include/csv.hpp
 - Non-functional shim — do **not** compile against it
 - For single-header use: generate `build/.../single_include_generated/csv.hpp` via `generate_single_header` target

include/internal/ARCHITECTURE.md

@@ -0,0 +1,184 @@
+# Internal Architecture
+
+This document describes the high-level architecture of the CSV parser internals and how major classes interact.
+
+Scope:
+- Internal component responsibilities
+- End-to-end data flow
+- Core invariants for safe changes
+- Where to add tests for subsystem changes
+
+For queue synchronization protocol details, see THREADSAFE_DEQUE_DESIGN.md.
+For AI-agent workflow and guardrails, see ../../AGENTS.md and ../../tests/AGENTS.md.
+
+## 1. System Shape
+
+The parser is a streaming producer-consumer pipeline:
+
+1. A parser implementation reads source bytes in chunks.
+2. Parsed rows are emitted into a queue.
+3. Consumer-side APIs expose rows and fields lazily.
+
+Two independent parser paths exist and must be kept behaviorally aligned:
+
+- File constructor path: memory-mapped parser
+- iostream constructor path: stream parser
+
+## 2. Major Components
+
+### API-facing core
+
+- CSVReader
+  - Orchestrates parser lifecycle, worker cycle, and row retrieval.
+  - Holds parser, queue, format, and exception propagation state.
+
+- CSVRow
+  - Lightweight row view over shared chunk data.
+  - Resolves field slices and supports index/name access.
+
+- CSVField
+  - Field-level typed conversion facade.
+  - Defers conversion work until requested.
+
+- CSVFormat
+  - Parse configuration (delimiter/quote/trim/header/chunk size/policies).
+
+### Parsing core
+
+- IBasicCSVParser
+  - Shared parse loop and field/row state machine.
+
+- MmapParser
+  - Reads chunks from memory maps and handles chunk-transition remainder.
+
+- StreamParser
+  - Reads chunks from stream sources.
+
+### Internal storage and transport
+
+- RawCSVData
+  - Shared chunk payload and per-chunk parse metadata.
+
+- RawCSVFieldList
+  - Compact field metadata storage (start/length/quote flags).
+
+- ThreadSafeDeque<CSVRow>
+  - Parser-to-consumer transport queue.
+  - Synchronization protocol is documented in THREADSAFE_DEQUE_DESIGN.md.
+
+### Relationship diagrams
+
+Parser hierarchy:
+
+```text
+                  +------------------+
+                  | IBasicCSVParser  |
+                  | (abstract base)  |
+                  +---------+--------+
+                            ^
+                 +----------+----------+
+                 |                     |
+        +--------+--------+    +-------+--------+
+        |   MmapParser    |    |  StreamParser  |
+        | concrete parser |    | concrete parser|
+        +-----------------+    +----------------+
+```
+
+Reader + row/data ownership:
+
+```text
+CSVReader
+  -> parser->next() builds RawCSVData chunk
+  -> emits CSVRow objects into ThreadSafeDeque
+
+         +--------------------------+
+         | RawCSVData               |
+         | - _data: shared_ptr<void>|
+         | - data: string_view      |
+         | - fields: RawCSVFieldList|
+         +------------+-------------+
+                      ^
+                      | shared_ptr<RawCSVData>
+          +-----------+-----------+-----------+
+          |                       |           |
+       CSVRow #1              CSVRow #2    CSVRow #N
+```
+
+Notes:
+- Multiple CSVRow instances can share the same RawCSVData chunk.
+- RawCSVData lifetime extends until the last referencing CSVRow is destroyed.
+- RawCSVFieldList is contained inside RawCSVData and indexes slices into the backing data payload.
+
+CSVRow -> CSVField lazy materialization:
+
+```text
+RawCSVData
+  |- data (chunk bytes)
+  |- fields[i] = {start, length, has_double_quote}
+  v
+CSVRow::get_field_impl(i)
+  -> slice = data.substr(start, length)
+  -> if quoted: unescape/cached materialization
+  -> if trim enabled: apply trim at access time
+  v
+CSVField(string_view)
+  -> typed conversion only when get<T>() / try_get<T>() is called
+```
+
+Implication:
+- Parser throughput stays focused on boundary detection and row emission; expensive string work is deferred until fields are actually accessed.
+
+## 3. End-to-End Flow
+
+Source bytes -> parser chunk read -> parse loop -> RawCSVData + RawCSVFieldList -> CSVRow enqueue -> CSVReader read_row / iteration -> CSVField materialization
+
+Operationally:
+
+1. CSVReader starts a read cycle with current chunk size.
+2. Parser next(bytes) ingests one chunk and emits complete rows.
+3. Queue buffers rows for consumer-side retrieval.
+4. CSVRow/CSVField lazily materialize trim/unescape/conversion behavior.
+5. Worker completion and errors are signaled back to the consumer side.
+
+## 4. Key Invariants
+
+### Chunk boundary integrity
+
+Fields spanning chunk boundaries must not be split/corrupted.
+
+### Path parity
+
+Mmap and stream parsers must preserve the same externally observable behavior.
+
+### Lazy materialization contract
+
+Trimming/unescaping/conversion behavior must remain coherent across parser and field-access layers.
+
+### Bounded streaming semantics
+
+Avoid designs that force retaining all parsed chunks globally.
+
+## 5. Change Impact Map
+
+- Parser state machine changes:
+  - basic_csv_parser.hpp, basic_csv_parser.cpp
+
+- Chunk transition changes:
+  - basic_csv_parser.cpp (MmapParser/StreamParser next)
+
+- Reader worker/iteration behavior:
+  - csv_reader.hpp, csv_reader.cpp, csv_reader_iterator.cpp
+
+- Field extraction and trimming/unescaping:
+  - csv_row.hpp, csv_row.cpp, raw_csv_data.hpp
+
+- Parse configuration behavior:
+  - csv_format.hpp, csv_format.cpp
+
+- Queue synchronization semantics:
+  - thread_safe_deque.hpp, THREADSAFE_DEQUE_DESIGN.md
+
+## 6. Test Guidance by Subsystem
+
+For full testing strategy, checklist, and conventions, see:
+- ../../tests/AGENTS.md

include/internal/THREADSAFE_DEQUE_DESIGN.md

@@ -42,7 +42,7 @@ The producer is intentionally not always running.
 
 ### 1. **Early Guard Check in `wait()`**
 
-**Location:** [thread_safe_deque.hpp:85](thread_safe_deque.hpp#L85)
+**Location:** [thread_safe_deque.hpp](thread_safe_deque.hpp)
 
 ```cpp
 void wait() {
@@ -167,7 +167,7 @@ The atomic store has memory ordering but does **not** prevent the lost-wakeup wi
 
 ## Batching Predicate: The `notify_size` Parameter
 
-**Location:** [thread_safe_deque.hpp:27](thread_safe_deque.hpp#L27)
+**Location:** [thread_safe_deque.hpp](thread_safe_deque.hpp)
 
 ```cpp
 ThreadSafeDeque(size_t notify_size = 100) : _notify_size(notify_size) {}
@@ -247,17 +247,17 @@ t=5:   Consumer: pop_front() → returns row1
 
 | Component | Where | What |
 |-----------|-------|------|
-| **Consumer** | [csv_reader.cpp:310](../csv_reader.cpp#L310) | `read_row()` – main thread |
-| **Consumer** | [csv_reader.cpp:313-315](../csv_reader.cpp#L313) | Check empty, is_waitable, call wait() |
-| **Producer** | [csv_reader.cpp:261](../csv_reader.cpp#L261) | `read_csv()` – worker thread |
-| **Producer** | [csv_reader.cpp:274](../csv_reader.cpp#L274) | notify_all() at start |
-| **Producer** | [csv_reader.cpp:278](../csv_reader.cpp#L278) | parser->next() pushes rows |
-| **Producer** | [csv_reader.cpp:291](../csv_reader.cpp#L291) | kill_all() at end |
-| **Queue** | [thread_safe_deque.hpp:57](thread_safe_deque.hpp#L57) | push_back() – producer |
-| **Queue** | [thread_safe_deque.hpp:67](thread_safe_deque.hpp#L67) | pop_front() – consumer |
-| **Queue** | [thread_safe_deque.hpp:84](thread_safe_deque.hpp#L84) | wait() – consumer |
-| **Queue** | [thread_safe_deque.hpp:108](thread_safe_deque.hpp#L108) | notify_all() – producer |
-| **Queue** | [thread_safe_deque.hpp:115](thread_safe_deque.hpp#L115) | kill_all() – producer |
+| **Consumer** | [csv_reader.cpp](../csv_reader.cpp) | `read_row()` flow on main thread |
+| **Consumer** | [csv_reader.cpp](../csv_reader.cpp) | Empty/is_waitable checks and wait() call |
+| **Producer** | [csv_reader.cpp](../csv_reader.cpp) | `read_csv()` worker lifecycle |
+| **Producer** | [csv_reader.cpp](../csv_reader.cpp) | notify_all() at cycle start |
+| **Producer** | [csv_reader.cpp](../csv_reader.cpp) | parser->next() push cycle |
+| **Producer** | [csv_reader.cpp](../csv_reader.cpp) | kill_all() terminal signal |
+| **Queue** | [thread_safe_deque.hpp](thread_safe_deque.hpp) | push_back() producer path |
+| **Queue** | [thread_safe_deque.hpp](thread_safe_deque.hpp) | pop_front() consumer path |
+| **Queue** | [thread_safe_deque.hpp](thread_safe_deque.hpp) | wait() condition protocol |
+| **Queue** | [thread_safe_deque.hpp](thread_safe_deque.hpp) | notify_all() wake-up state |
+| **Queue** | [thread_safe_deque.hpp](thread_safe_deque.hpp) | kill_all() terminal publication |
 
 ---
 

include/internal/basic_csv_parser.cpp

@@ -74,6 +74,7 @@ namespace csv {
             _ws_flags = internals::make_ws_flags(
                 format.trim_chars.data(), format.trim_chars.size()
             );
+            _has_ws_trimming = !format.trim_chars.empty();
         }
 
         CSV_INLINE void IBasicCSVParser::end_feed() {
@@ -99,10 +100,6 @@ namespace csv {
             using internals::ParseFlags;
             auto& in = this->data_ptr->data;
 
-            // Trim off leading whitespace
-            while (data_pos < in.size() && ws_flag(in[data_pos]))
-                data_pos++;
-
             if (field_start == UNINITIALIZED_FIELD)
                 field_start = (int)(data_pos - current_row_start());
 
@@ -122,34 +119,23 @@ namespace csv {
 
             field_length = data_pos - (field_start + current_row_start());
 
-            // Trim off trailing whitespace, this->field_length constraint matters
-            // when field is entirely whitespace
-            for (size_t j = data_pos - 1; ws_flag(in[j]) && this->field_length > 0; j--)
-                this->field_length--;
+            // Whitespace trimming is deferred to get_field_impl() so callers that never
+            // read field values (e.g. row counting) pay no trimming cost.
         }
 
         CSV_INLINE void IBasicCSVParser::push_field()
         {
             // Update
-            if (field_has_double_quote) {
-                fields->emplace_back(
-                    field_start == UNINITIALIZED_FIELD ? 0 : (unsigned int)field_start,
-                    field_length,
-                    true
-                );
-                field_has_double_quote = false;
-
-            }
-            else {
-                fields->emplace_back(
-                    field_start == UNINITIALIZED_FIELD ? 0 : (unsigned int)field_start,
-                    field_length
-                );
-            }
+            fields->emplace_back(
+                field_start == UNINITIALIZED_FIELD ? 0 : (unsigned int)field_start,
+                field_length,
+                field_has_double_quote
+            );
 
             current_row.row_length++;
 
             // Reset field state
+            field_has_double_quote = false;
             field_start = UNINITIALIZED_FIELD;
             field_length = 0;
         }
@@ -245,6 +231,8 @@ namespace csv {
         CSV_INLINE void IBasicCSVParser::reset_data_ptr() {
             this->data_ptr = std::make_shared<RawCSVData>();
             this->data_ptr->parse_flags = this->_parse_flags;
+            this->data_ptr->ws_flags = this->_ws_flags;
+            this->data_ptr->has_ws_trimming = this->_has_ws_trimming;
             this->data_ptr->col_names = this->_col_names;
             this->fields = &(this->data_ptr->fields);
         }

include/internal/basic_csv_parser.hpp

@@ -123,6 +123,7 @@ namespace csv {
             ) : _parse_flags(parse_flags), _ws_flags(ws_flags) {
                 const char d = internals::infer_delimiter(parse_flags);
                 _simd_sentinels = SentinelVecs(d, internals::infer_quote_char(parse_flags, d));
+                _has_ws_trimming = std::any_of(ws_flags.begin(), ws_flags.end(), [](bool b) { return b; });
             }
 
             virtual ~IBasicCSVParser() {}
@@ -155,7 +156,7 @@ namespace csv {
             CSVRow current_row;
             RawCSVDataPtr data_ptr = nullptr;
             ColNamesPtr _col_names = nullptr;
-            CSVFieldList* fields = nullptr;
+            RawCSVFieldList* fields = nullptr;
             int field_start = UNINITIALIZED_FIELD;
             size_t field_length = 0;
 
@@ -190,6 +191,11 @@ namespace csv {
              *  be trimmed
              */
             WhitespaceMap _ws_flags;
+
+            /** True when at least one whitespace trim character is configured.
+             *  Used to skip trim loops entirely in the common no-trim case.
+             */
+            bool _has_ws_trimming = false;
             bool quote_escape = false;
             bool field_has_double_quote = false;
 

include/internal/common.hpp

@@ -24,8 +24,6 @@
   *  in the single header version
   */
 #define CSV_INLINE
-
-#pragma once
 #include <type_traits>
 
 #if defined(__EMSCRIPTEN__)
@@ -57,14 +55,6 @@
     #define CSV_NON_NULL(...)
 #endif
 
-#if defined(__GNUC__) || defined(__clang__)
-    #define CSV_UNREACHABLE() __builtin_unreachable()
-#elif defined(_MSC_VER)
-    #define CSV_UNREACHABLE() __assume(0)
-#else
-    #define CSV_UNREACHABLE() abort()
-#endif
-
 // This library uses C++ exceptions for error reporting in public APIs.
 #if defined(__cpp_exceptions) || defined(_CPPUNWIND) || defined(__EXCEPTIONS)
     #define CSV_EXCEPTIONS_ENABLED 1