feat: support pagination in list_* methods in rest catalog

[jayceslesar]

Closes #2084

Rationale for this change

Support pagination!

I looked at the rust iceberg implementation and noticed there is no laziness there either so seems fine for the non-lazy list methods to just exhaust into a list.

Also needed to move all requests behind a self._request_with_retries method that now handles all previous retry logic

Are these changes tested?

Added tests

Are there any user-facing changes?

Sort of. Existing compatability is not broken but there are new methods to lazily use the list_* methods.

[jayceslesar]

Current CI failures look like they should be fixed in a different commit

[kevinjqliu]

i think we should support pageSize as well https://github.com/apache/iceberg/blob/17f9a9fd28bbcb37745f13ebfcd57cd5a96e0a5d/open-api/rest-catalog-open-api.yaml#L244-L258

[jayceslesar]

@kevinjqliu wdyt about a list_*_raw() method instead of supporting page size in the list_*_lazy() and list_*() that provides no abstraction to the user? I think that gives users the most control

[jayceslesar]

lmk what you think this needs, imo if a user really needs to be able to make single requests they can write the thin layer to do that but if we want to provide the most basic level of call against the list API's I am happy to implement.

[jayceslesar]

@kevinjqliu im gonna take another stab at this to see if we can accomplish with less changes

[jayceslesar]

closing in favor of #2158