trjhtr

Task: I'm importing a .CSV files as a pandas dataframes. I'm writing one column of that dataframe to a .txt file. Importantly, each row of the column must only be written as one row in the text file, never more (hence stripping /n)!

I have a very large dataframe (2 million rows), and this loop is naturally very slow given the I/O overhead. Any suggestions for improvements?

for i in tqdm(data['column'].head(500)):
 f = open("Questions.txt","a", newline="n",encoding='utf-8')
 f.write(i.strip("/n"))
 f.write("n")
 f.close()

and this loop is naturally very slow given the I/O overhead

Most of the overhead is probably produced from the way you are opening/closing the file with each iteration. That could be easily fixed by moving these operations out of the loop:

f = open("Questions.txt","a", newline="n",encoding='utf-8')
for i in tqdm(data['column'].head(500)):
 f.write(i.strip("/n"))
 f.write("n")
f.close()

Long story short: you don't deal with the I/O and call into numpy.savetext instead. Consider the following code:

import numpy as np

np.savetext("Questions.txt", data['column'].map(strip_newlines).head(500), newline="n", encoding="utf-8")

This makes abundantly clear that you only care about a newline-stripped representation of the column 'column' in your dataframe. Note that I removed the progress bar from this. I expect this code to be blazingly fast in comparison to yours, because it does two things:

1. expensive I/O operations (open and close) are only done once


2. I/O is pushed from python into C++