feat: Add replicator connect_to support for outbound requests.

[willholley]

Overview

This adds a feature to the CouchDB replicator to override the DNS target for specific host
patterns (including wildcards) when making outbound requests. This has the same semantics as curl's [--connect-to](https://curl.se/docs/manpage.html#--connect-to option:

For a request intended for the "HOST1:PORT1" pair, connect to "HOST2:PORT2" instead. This option is only used to establish the network connection. It does NOT affect the hostname/port number that is used for TLS/SSL (e.g. SNI, certificate verification) or for the application protocols.

There is a new configuration option to specify the overrides:

[replicator]
connect_to = host:target, host2:target

The replicator resolves the configured host patterns to the alternative connection targets while
preserving the request URL host (applies to regular requests and session-auth requests).

Note this depends on the connect_to option in ibrowse.

Testing recommendations

Testing this is with TLS a bit involved as it relies on setting up an SNI proxy. I did it using nginx in docker with the configuration attached to proxy to a cloudant.com database. The proxy was running on a non-standard port (e.g. 8443) so that any replications connecting directly to cloudant.com would fail.

I then set connect_to = *.cloudant.com:443:127.0.0.1:8443 in default.ini and configured a replication from myaccount.cloudant.com/mydb. The test succeeds if the proxy logged the connection and the replication completed.

The feature also works without TLS - you can just use it to direct an arbitrary hostname to your local couchdb, for instance. e.g. if you use connect_to = *.cloudant.com:80:127.0.0.1:15984 and couchdb is running on 127.0.0.1:15984, set up a replication with source or target as http://foo.cloudant.com/db1 and it will be redirected to 127.0.0.1:15984/db1.

nginx.conf.zip

Related Issues or Pull Requests

• feat: add connect_to option couchdb-ibrowse#9

Checklist

• This is my own work, I did not use AI, LLM's or similar technology
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[nickva]

This is very nice! I didn't get to play with it locally just did a quick look-over with some comments first

[nickva]

It looks great, almost there but noticed a few more things about setting SNI and ipv6 handling

[nickva]

Even if we strip [] bracket from an ipv6 address tcp, when we pass an ip like "::1" to gen_tcp|ssl:connect/3 I think it will still fail, it can only handle a tuple

(node1@127.0.0.1)15> ssl:connect("[2607:f8b0:4023:100d::8b]", 443, [{verify, verify_none}]).
{error,nxdomain}

(node1@127.0.0.1)16> ssl:connect("2607:f8b0:4023:100d::8b", 443, [{verify, verify_none}]).
{error,nxdomain}

> ssl:connect({9735,63664,16419,4109,0,0,0,139}, 443, [{verify, verify_none}]).
{ok,{sslsocket,{gen_tcp,#Port<0.28>,tls_connection, undefined}, [<0.57527.0>,<0.57526.0>]}}

(address is from dig AAAA google.com)

Wonder if we then take the TargetHost and parse with a helper as:

parse_target(Bin) ->
case inet:parse_strict_address(binary_to_list(Bin)) of
    {ok, Tuple} -> {ok, Tuple}; 
    _ -> {ok, Bin};
end

Then connect_to can can either the get a string or a tuple.

[willholley]

Yep - good shout. I've update the branch to strictly parse the TargetHost address and, if valid, store it as a tuple which gets passed through to connect_to. In other cases, it assumes the TargetHost is a hostname and passes it through as a string.

[nickva]

It's a bit of a gnarly expression but I couldn't come up with anything smaller, after staring at it for a while it does seems right

[nickva]

all_but_first

Hadn't seen that used before, that's neat:

All but the first matching subpattern, that is, all explicitly captured subpatterns, but not the complete matching part of the subject string. This is useful if the regular expression as a whole matches a large part of the subject, but the part you are interested in is in an explicitly captured subpattern.

[nickva]

+1 nicely done, Will!

Don't forget to squash the fixup commits.