CODEWITHSUNDEEP
pythonpython-s3fsfsspec
Roelant
Roelant

Reputation: 5119

Using fsspec with aws profile_name

With S3fs we can set

fs = s3fs.S3FileSystem(profile=profile_name)

However this passing doesn't work for fsspec with caching:

 fs = fsspec.filesystem(
            "filecache",
            target_protocol="s3",
            cache_storage="some/dir",
            cache_check=10,  
            expiry_time=60, 
            profile=profile_name, 
        )

Of course we can use AWS_PROFILE to circumvent fsspec, but is there any way to pass it onwards?

Upvotes: 0

Views: 190

Answers (1)

mdurant
mdurant

Reputation: 28683

The caching filesystems take target_options to pass on to the inner filesystem (this is a convention shared with many other implementations designed to chain well). So your invocation would be

fs = fsspec.filesystem(
            "filecache",
            target_protocol="s3",
            cache_storage="some/dir",
            cache_check=10,  
            expiry_time=60, 
            target_options=dict(profile=profile_name), 
        )

Upvotes: 1

Related Questions