checksum: add SHA-3 (FIPS 202) via OpenSSL EVP

[fusionfoto]

Summary

Adds sha3-256, sha3-384, and sha3-512 to --checksum-choice.
Implementation is entirely through OpenSSL's EVP interface — no third-party
hash code is vendored. Picks up automatically when rsync is built with
--enable-openssl against libcrypto >= 1.1.1 (released September 2018, on
every supported distro). On older libcrypto the existing verify_digest()
probe cleanly demotes the entry to CSUM_gone, so listing the algorithms
in valid_checksums_items[] is safe across the board.

Defaults are unchanged. SHA-3 is never selected by auto negotiation;
it has to be requested explicitly on both ends:

rsync -a --checksum-choice=sha3-256 src/ dst/

An unmodified peer keeps negotiating xxh128/xxh3/md5/MD4 as before.

Motivation

SHA-3 is the FIPS 202 standard and a NIST-approved hash for FIPS 140-3
modules. Some environments — federal, healthcare, regulated finance —
require checksum algorithms drawn from that approved list. With this
patch rsync can satisfy those requirements without changing defaults
for everyone else.

Diffstat

 NEWS.md                      | 11 +++++++
 checksum.c                   | 20 ++++++++++++
 lib/md-defines.h             | 12 ++++++++
 rsync.1.md                   |  7 +++++
 testsuite/checksum-sha3.test | 60 ++++++++++++++++++++++++++++++++++++
 5 files changed, 110 insertions(+)

110 lines added, 0 removed.

How it fits the existing code

Every checksum entry point — get_checksum2, file_checksum,
sum_init/update/end — already short-circuits to the EVP path when
xfer_sum_evp_md / file_sum_evp_md / cur_sum_evp_md is set. So no
extra case arms are needed for the rolling-block, whole-file, or
streaming code paths — SHA-3 just flows through them.

lib/md-defines.h gains three digest-length constants
(SHA3_256_DIGEST_LEN = 32, SHA3_384_DIGEST_LEN = 48,
SHA3_512_DIGEST_LEN = 64) since OpenSSL never published
SHA3_*_DIGEST_LENGTH macros. SHA3-512's 64-byte digest matches
SHA-512's, so MAX_DIGEST_LEN is unaffected.

Test coverage

testsuite/checksum-sha3.test exercises, for each of the three SHA-3
variants:

• whole-file path (transfer checksum only)
• pre-transfer + transfer path (-c)
• comma-paired form (--checksum-choice=sha3-N,sha3-N)
• delta-update path (mutate destination, re-sync, confirm match)

The test self-skips if the local rsync was built without OpenSSL or its
libcrypto lacks SHA-3, so it's safe to include unconditionally.

Verified

Built and tested against this branch on Ubuntu 22.04 / OpenSSL 3.0.2 /
gcc 11.4 / libxxhash 0.8.1:

Check	Result
./configure --enable-openssl && make	clean
rsync --version "Checksum list"	xxh128 xxh3 xxh64 (xxhash) sha3-512 sha3-384 sha3-256 md5 md4 sha1 none
rsync -a --checksum-choice=sha3-{256,384,512} round trip	byte-identical (verified with diff -r)
rsync -ac --checksum-choice=sha3-512 (pre-transfer + transfer)	works
rsync -ac --checksum-choice=sha3-256,sha3-256 (comma pair)	works
Delta-update path (modify dst, resync)	works
SHA3-256("abc") digest	matches FIPS 202 vector 3a985da7…1532
Auto negotiation	still picks xxh128 by default; SHA-3 only when asked
Behavior on older libcrypto (probed via verify_digest)	algorithm cleanly demoted to CSUM_gone; --cc=sha3-* reports "unknown checksum name"
make check (full suite)	new checksum-sha3 test passes; rest of the suite unaffected. The pre-existing devices failure reproduces on unmodified upstream master and is unrelated.

Backward compatibility

• Wire protocol: unchanged. SHA-3 isn't in the auto set, so peers
  that don't know it never see it advertised.
• Build matrix: builds without OpenSSL are unaffected — the new
  entries in valid_checksums_items[] are inside #ifdef USE_OPENSSL.
• Older libcrypto: verify_digest() already handles "EVP knows the
  name but the lib doesn't support it" by marking the entry
  CSUM_gone; users see "unknown checksum name" if they request
  sha3-* against a libcrypto that doesn't have it.

Authorship and scope

The patch was authored by Deepak Jain (deepak@ai.net),
Signed-off-by line preserved) as part of work needed for a regulated
deployment that requires FIPS-approved digests. It is being submitted
upstream because there's no reason the rest of the rsync community
shouldn't benefit from it.

Andrew Tridgell indicated on the mailing list that he would accept a
PR along these lines, which is why this is going to GitHub directly
rather than circulating as an RFC first.

Happy to split, rebase, or rework anything that doesn't match house
style. The commit is one atomic change so it should be straightforward
to drop, squash, or amend as needed.

[tridge]

@fusionfoto thanks! I'm planning some more work on new features in a couple of weeks time, will look at this one more closely then