http_client enhancement: pagination

[sonnens]

(copied from #20844 )

I'm dealing with this one again with yet another API so I want to revive this discussion

I don't know what the correct general way to handle pagination is, since every API is going to be different. Okta handles it with a Link header, Kandji handles it with a next link in the payload, Slack handles it with response_metadata.next_cursor in the payload

I think that maybe one solution is to add a pagination section to the http_client source and allow the user to write VRL that will generate the next URL to fetch (or null to reset to whatever the original URL is?)

For something like the Okta streaming API (just as an example. I've written the Okta source to handle this specific case) the API always returns a rel=next link, and the client is meant to pause when the payload is empty but continue processing with that next link (or, in other words, the URL to process next cycle is updated rather than reset.) I suppose something like a reset_after_pagination option or some such?

The issues here I guess is how to expose a state machine to VRL (what indicates that it should stop?) and the internal configuration (ie, the url that it should hit next time)

I'm happy to actually make the change myself, and write an RFC around it but I'd like to hear from others first

[pront]

Hey @sonnens, thanks for offering to write an RFC and the code for this!

Key points for the RFC to address before we come up with an implementation plan:

1. VRL target shape -- what does the VRL expression receive as input?
2. State across intervals
3. Timeout semantics
4. Safeguards
5. Observability -- pagination depth and cursor extraction failure metrics.

Happy to review the RFC when you have one.