Databricks Dbutils. Fs. Ls Shows Files. However, Reading Them Throws an Io Error
I Am Running a Spark Cluster and When I'm Executing the Below Command on Databricks Notebook, It Gives Me the Output: Dbutils. Fs. Ls("/Mnt/Test_File. Json")...
I am running a Spark Cluster and when I'm executing the below command on Databricks Notebook, it gives me the output:
dbutils.fs.ls("/mnt/test_file.json")
[FileInfo(path=u'dbfs:/mnt/test_file.json', name=u'test_file.json', size=1083L)]
However, when I'm trying to read that file, I'm getting the below mentioned error:
with open("mnt/test_file.json", 'r') as f:
for line in f:
print line
IOError: [Errno 2] No such file or directory: 'mnt/test_file.json'
What might be the issue here? Any help/support is greatly appreciated.
2 Answers
In order to access files on a DBFS mount using local file APIs you need to prepend /dbfs to the path, so in your case it should be
with open('/dbfs/mnt/test_file.json', 'r') as f:
for line in f:
print(line)
See more details in the docs at especially regarding limitations. With Databricks Runtime 5.5 and below there's a 2GB file limit. With 6.0+ there's no longer such a limit as the FUSE mount has been optimized to deal with larger file sizes.
you could directly use
dbutils.fs.head("file path")
if you intended to view the data.
Alexander Ross has covered the video game industry for a decade, writing deep dives on game design, esports tournaments, VR developments, and gaming culture.
