Fail permanently on other errors too

[timmc]

I have some older accounts with media that can no longer be fetched for various reasons, ranging from 500 errors from the server to domains that no longer resolve. Each new backup takes a very long time because of these outstanding errors. A recent run took all night.

Desired properties of a solution:

• One-time errors do not cause a permanent media download failure
• Continue to handle 429 as temporary (backoff and retry in same session)
• Continue to handle 401 and 404 as immediately permanent (mark as missing and skip in the future)

I'll add a description of an implementation in the comments but wanted to check if a contribution like this would be desirable.

[timmc]

I think what I'd want to do is recognize situations where the same error is showing up several weeks apart, and not changing. Something like this:

• When a media request fails for a reason other than 429, 401, or 404, the error information is appended to a JSONL file located at {file_name}.errors.
  • Each line is a JSON dict containing timestamp and error (the repr of the HTTPError or URLError). For example: {"timestamp":"2025-08-25 23:35:33+00:00", "error":"<HTTPError 500: 'Internal Server Error'>"}
• After an error is recorded, the errors file is read and the file is marked as missing if a threshold has been reached.
  • The errors file is processed by looking at the last entry, then collecting all of the consecutive tail entries that have the same error as the last one.
  • If the timestamp of the first and last of that series are more than (let's say) two weeks apart, the file is to be considered missing.

Thoughts?

[kensanata]

Sounds good to me!

[kensanata]

I recently looked at HTTP status 410 Gone which also seems to fit the bill as "immediate stop".

[kensanata]

I'm wondering about the 14-day period. I rarely run mastodon-archive (less than once per 14 days). From my point of view, too many errors are too many errors and the time window is irrelevant unless there is some sort of expiration of errors.

Such an expiration, however, would be an entirely different concern.

My hunch would be to say: errors are forever. Or: Errors expire after 1y. Anyway: Something long.

[kensanata]

As for consecutive errors: It seems to me that a "good" result should immediately wipe the error trail since there can no longer be a "consecutive errors" situation.

[timmc]

My thinking is that if the local or remote server happens to be having difficulty at the time that you run a backup, you probably want to treat that as temporary -- no matter how many times you run a backup during that interval. And I've seen servers be out of commission for a few days, but always less than 2 weeks. So that's where I'm getting that ballpark number from.

I don't know about the value of expiring errors. If your two consecutive backup runs are a month apart, or 2 years apart, and you get the same error both times -- you probably want to treat it as a permanent failure either way.

Good point about a successful download -- I'll add a PR to wipe the errors file on success, as it would no longer be needed. (In contrast, it might be a good idea to keep it even after the .missing file is created, as it provides context. But I could go either way on that.)

[timmc]

I've been running this branch locally, and so far I've found a number of permanent errors, but also recorded some errors on media that I later successfully downloaded. If you're curious, these were the truly transient errors:

• <HTTPError 403: 'Forbidden'> (once, on remote)
• URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')) (three on the same remote)
• <HTTPError 500: 'Internal Server Error'> (three on local)
• URLError(OSError(101, 'Network is unreachable')) (once, on a different local)

So I think that's working well.

[timmc]

Here's a PR for cleaning up error histories on success: #126