Questions Tagged [Python-S3Fs]
Ask Question for Questions Related to the Python S3Fs Library 87 Questions More Bountied 0 Unanswered Frequent Score Unanswered (My Tags) Apply Filter 64 Votes...
![Questions Tagged [Python-S3Fs]](https://www.alledaags.nl/uploads/questions-tagged-python-s3fs_1200.jpg)
For questions related to the Python s3fs library
87
questions
64
votes
5
answers
102k
views
How to read partitioned parquet files from S3 using pyarrow in python
I looking for ways to read data from multiple partitioned directories from s3 using python.
data_folder/serial_number=1/cur_date=20-12-2012/abcdsd0324324.snappy.parquet
data_folder/serial_number=2/...
19
votes
3
answers
5k
views
Overwrite parquet file with pyarrow in S3
I'm trying to overwrite my parquet files with pyarrow that are in S3. I've seen the documentacion and I haven't found anything.
Here is my code:
from s3fs.core import S3FileSystem
import pyarrow as ...
14
votes
1
answer
12k
views
s3fs suddenly stopped working in Google Colab with error "AttributeError: module 'aiobotocore' has no attribute 'AioSession'" [closed]
Yesterday the following cell sequence in Google Colab would work.
(I am using colab-env to import environment variables from Google Drive.)
This morning, when I run the same code, I get the following ...
13
votes
1
answer
8k
views
How to read parquet file from s3 using dask with specific AWS profile
How to read a parquet file on s3 using dask and specific AWS profile (stored in a credentials file). Dask uses s3fs which uses boto. This is what I have tried:
>>>import os
>>>...
12
votes
4
answers
64k
views
ImportError: Missing optional dependency 'S3Fs'. The S3Fs package is required to handle S3 files. Use pip or conda to install S3Fs
I am using AWS Cloud9 as my IDE.
import boto3
import pandas as pd
# import s3fs
# s3_ob=boto3.resource('s3',aws_access_key_id="xxxxxxxxxx",aws_secret_access_key="xxxxxxxxxxxx")
client=boto3.client('...
12
votes
1
answer
16k
views
S3FS python, credential inline
I am trying to use python s3fs to read files in S3 AWS.
I could not find the code to put credential (Access key + Secret) into s3fs code.
Can anyone please help me how to set this info along with ...
9
votes
2
answers
17k
views
download file using s3fs
I am trying to download a csv file from an s3 bucket using the s3fs library. I have noticed that writing a new csv using pandas has altered data in some way. So I want to download the file directly in ...
8
votes
5
answers
5k
views
Pandas read_csv specify AWS Profile
Pandas (v1.0.5) use s3fs library to connect with AWS S3 and read data. By default, s3fs uses the credentials found in ~/.aws/credentials file in default profile. How do I specify which profile should ...
7
votes
2
answers
3k
views
s3fs gzip compression on pandas dataframe
I'm trying to write a dataframe as a CSV file on S3 by using the s3fs library and pandas.
Despite the documentation, I'm afraid the gzip compression parameter it's not working with s3fs.
def ...
6
votes
3
answers
19k
views
How to mount S3 bucket as local FileSystem?
I have a python app running on a Jupiter-notebook on AWS. I loaded a C-library into my python code which expects a path to a file.
I would like to access this file from the S3 bucket.
I tried to use ...
5
votes
3
answers
7k
views
aiobotocore - ImportError: cannot import name 'InvalidIMDSEndpointError'
The code below raise an import exception
import s3fs
fs = s3fs.S3FileSystem(anon=False)
Exception
Traceback (most recent call last):
File "issue.py", line 1, in <module>
import ...
5
votes
1
answer
2k
views
Resolving dependencies fails on boto3 and s3fs using poetry
I can install boto3, s3fs and pandas using :
pip install boto3 pandas s3fs
But it fails with poetry :
poetry add boto3 pandas s3fs
Here is the error :
Because no versions of s3fs match >2023.3.0,&...
4
votes
2
answers
5k
views
cannot import s3fs in pyspark
When i try importing the s3fs library in pyspark using the following code:
import s3fs
I get the following error:
An error was encountered: cannot import name 'maybe_sync' from
'fsspec.asyn' (/usr/...
4
votes
1
answer
3k
views
Load CSV file into Pandas from s3 using chunksize
I'm trying to read a very big file from s3 using...
import pandas as pd
import s3fs
df = pd.read_csv('s3://bucket-name/filename', chunksize=100000)
But even after giving the chunk size it is taking ...
4
votes
1
answer
3k
views
pytest How to mock s3fs.S3FileSystem open file
I am trying to mockup the call to open a file in a S3 bucket. The code that I have is:
# mymodule.py
import s3fs
#...
def __init__(self):
self.s3_filesystem = s3fs.S3FileSystem(anon=False, key=...
Robert Thorne covers electric vehicle innovations, autonomous driving systems, global mobility trends, and automotive engineering developments.
