How to Convert Csv File to Gzip File with Use of Multiprocessing. Pool in Python
I Am Using Gzip to Convert Csv File into Gunzip File. I Am Trying to Implement Multiprocessing to Make the Process Faster. but When I Run My Code It Doesn't...
I am using gzip to convert csv file into gunzip file.I am trying to implement multiprocessing to make the process faster. But when i run my code it doesn't even give me the output, it keeps on running and doesn't give error either. Below is my code
from multiprocessing import Pool
import gzip
path= r'D:\All Files'
def convertToGzip(file):
file_ = os.path.join(path, file)
print('File {} is being Converted'.format(file_))
with open(os.path.join(path, file_), 'rb') as f_in:
with gzip.open(file+'.gz', 'wb') as f_out:
shutil.copyfileobj(f_in, f_out)
a_pool = Pool(processes=4)
result = a_pool.map(convertToGzip, os.listdir(path))
I donot understand where is the issue here , since I do not know much about Multiprocessing
1 Answer
You can try this:
import gzip
import os
import shutil
from multiprocessing import Pool
PATH = "D:\All Files"
def convert_to_gzip(fname):
fpath = os.path.join(PATH, fname)
print("File {} is being Converted".format(fpath))
with open(fpath, "rb") as f_in:
with gzip.open(fpath + ".gz", "wb") as f_out:
shutil.copyfileobj(f_in, f_out)
def main():
with Pool() as pool:
result = pool.map(convert_to_gzip, os.listdir(PATH))
if __name__ == "__main__":
main()
Alexander Ross has covered the video game industry for a decade, writing deep dives on game design, esports tournaments, VR developments, and gaming culture.
