pudl_datastore AccessDenied

[krivard]

Describe the bug

A clear and concise description of what the bug is.

If I attempt to fetch a zenodo ID that's not in the S3 cache, I get an AccessDenied error from pudl_datastore.

Bug Severity

How badly is this bug affecting you?

• High: This bug is preventing me from running the ETL locally on this branch.

To Reproduce

1. Checkout Brian's EPA MATS branch: https://github.com/bsousa22/pudl/tree/extract-mats-5358
2. Run pixi run pudl_datastore epamats
3. Observe the following stacktrace:

Traceback (most recent call last):
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 169, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/aiobotocore/context.py", line 36, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/aiobotocore/client.py", line 436, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDenied: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/katie/Documents/work/catalyst/pudl/src/pudl/workspace/resource_cache.py", line 229, in add
    path.write_bytes(content)
    ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/pathlib_abc/__init__.py", line 444, in write_bytes
    with vfsopen(self, mode='wb') as f:
         ~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/spec.py", line 2275, in __exit__
    self.close()
    ~~~~~~~~~~^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/spec.py", line 2226, in close
    self.flush(force=True)
    ~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/spec.py", line 2089, in flush
    if self._upload_chunk(final=force) is not False:
       ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 2759, in _upload_chunk
    self.commit()
    ~~~~~~~~~~~^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 2785, in commit
    write_result = self._call_s3("put_object", **kw, **match)
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 2618, in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 464, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
        method, kwargs=additional_kwargs, retries=self.retries
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/katie/Documents/work/catalyst/pudl/.pixi/envs/default/lib/python3.13/site-packages/s3fs/core.py", line 215, in _error_wrapper
    raise err
PermissionError: Access Denied

I get the same error when trying to run the raw_epamats group of the ETL from the Dagster UI.

Expected behavior

Upload failures should not prevent local runs of the ETL.

Any subset of the following would be fine:

• Successfully downloading the epamats archives to local cache
• Printing a warning if my current AWS credentials don't have the proper permissions
• Telling me where it got bogus AWS credentials (the environment? a dotfile?)

[zaneselvans]

Hmm. So this is the result of it attempting to populate the cache in the process of downloading the data for you, because it's never been seen by the cache before? Which means it's never been pushed to GitHub by one of us before (since any change to zenodo_dois.yml results in the S3 cache being warmed).

But yeah this is not great. It seems like the correct behavior would be for it to realize it can't write to S3 and just get you the data (slowly) direclty from Zenodo, and maybe log a warning.