trjhtr

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0

Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns contain text posts of var length. I then loop through each of these rows and query the post against an imported function that returns a specific counter of variable length. The counter tells me about the presence of certain words in the text. On average the func takes less than 1 ms to return this counter. (Timer for the "Func" inserted at the end to prove this)

Overheads: The code i'm looking to improve is the loop. I accept a certain level of overhead with the "func" which can't be improved at this minute. I have considered looking at Spark or Dask to parallelize the loop and speed up the process. Suggestions are welcome

#Import data
import pandas as pd 
from func import func 
data = pd.read_csv('Dataset.csv')

print(len(data))
>> 1900000

print(data.columns)
>> Index(['type', 'body'], dtype='object')


#Create new DF
data2 = pd.Dataframe()

for post in data['post']:
 post = str(post)
 scores = func.countWords(posts)
 data2 = data2.append(scores,ignore_index=True)

print(scores)
>> Counter(0: 306,
 1: 185,
 2: 61,
 45: 31,
 87: 23,
 92: 5,
 94: 3,
 102: 30,)


 import time
 start = time.time()
 score = func.countWords("Slow down Sir, you're going to give yourself skin faliure!")
 end = time.time()
 print(end - start)
 >> 0.0019948482513427734

Several errors:

I then loop through each of these loops

I take it you mean "I then loop through each of these columns".

for post in data['post]:

missing end quote mark

scores = Func.countWords(posts)

You imported func (lowercase) and now you're calling Func (uppercase)

data2 = data2.append(scores,ignore_index=True)

append should take a row-type object. If the function returns a numeric, then you shouldn't be appending it. Instead you can do:

def post_to_count(post):
 return func.countWord(str(post))

scores = data['post'].apply(post_to_count)