Skip to content

Commit 58f773c

Browse files
committed
Improve GNU compatibility through byte processing
The original Rust implementation assummed a UTF-8 environment, as is common in modern systems. This commit improves GNU sed compatibility by changing most processing to run on bytes rather than Rust's Unicode characters and strings. This change also improves the handling of binary files. The commit adds a simple POSIX-compatible locale configuration to support UTF-8 processing when specified. While at it, also fix script-specified file handling to take place through OsString. The change fixes two failing GNU test suite cases.
1 parent 30607c4 commit 58f773c

13 files changed

Lines changed: 1596 additions & 605 deletions

README.md

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -116,12 +116,19 @@ cargo test
116116
sequences.
117117

118118
### Incompatibilities
119-
* The input is assumed to be valid UTF-8 (this includes 7-bit ASCII).
120-
If the input is in another code page, consider converting it through UTF-8
121-
in order to avoid errors on invalid UTF-8 sequences and for the correct
122-
handling of regular expressions.
123-
This _sed_ program can also handle arbitrary byte sequences if no part of the
124-
input is treated as string.
119+
* Similarly to GNU _sed_, input is processed as raw bytes or as valid UTF-8
120+
(this includes 7-bit ASCII) based on the locale as specified by the
121+
`LC_ALL`, `LC_CTYPE`, and `LANG` environment variables,
122+
with the default being byte processing.
123+
However, in contrast with GNU _sed_, other locales (e.g. ISO-8859-1)
124+
are not supported. If the input is in another code page or encoding
125+
and requires locale-specific processing (e.g. ignore/map case,
126+
character classes), consider converting it through UTF-8 to ensure
127+
the correct handling of locale-specific regular expressions.
128+
This _sed_ program can also handle arbitrary byte sequences
129+
if no part of the input requires treating it as a Rust String.
130+
* Back-references aren't supported when input is processed as bytes
131+
(`LC_ALL=C`).
125132
* The command will report an error and fail if duplicate labels are found
126133
in the script.
127134
This matches the BSD behavior. The GNU version accepts duplicate labels.

0 commit comments

Comments
 (0)