trjhtr

I have a large file that I need to parse - and since it will be regenerated from external queries every time script runs so there is no way to parse it once and cache the results.

I want to save memory footprint and read and parse only logical "chunks" of that file everything between open 'product' and closing curly bracket.
Not sure what is the canonical way in Python (I am new to the language).

Here's what I tried so far:

def read_chunk(file_name, pattern_open_line, pattern_close_line):
 with open(file_name,"r") as in_file:
 chunk = 
 in_chunk = False
 open_line = re.compile(pattern_open_line);
 close_line = re.compile(pattern_close_line)
 try:
 for line in in_file:
 line = line.strip()
 if in_chunk:
 chunk.append(line)
 if close_line.match(line):
 yield chunk
 if open_line.match(line):
 chunk = 
 chunk.append(line)
 in_chunk = True
 continue
 except StopIteration:
 pass

def get_products_buffered(infile):
 chunks = read_chunk(infile, r'^products*$', r'^s*}s*')
 products = 
 for lines in chunks:
 for line in lines:
 if line.startswith('productNumber:'):
 productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
 products.append(productNumber)
 continue
 return products

def get_products_unbuffered(infile):
 with open(infile) as f:
 lines = f.readlines()
 f.close()
 products = 
 for line in lines:
 if line.startswith('productNumber:'):
 productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
 products.append(productNumber)
 continue
 return products

I profiled both runs and while unbuffered reading is faster:

Buffered reading
Found 9370 products:
Execution time: 3.0031037185720177
Unbuffered reading
Found 9370 products:
Execution time: 1.2247122452647523

it also incurs a much bigger memory hit when file is essentially read into memory:

Line # Mem usage Increment Line Contents
================================================
 29 28.2 MiB 0.0 MiB @profile
 30 def get_products_buffered(infile):
 31 28.2 MiB 0.0 MiB chunks = read_chunk(infile, '^products*$', '^s*}s*')
 32 28.2 MiB 0.0 MiB products = 
 33 30.1 MiB 1.9 MiB for lines in chunks:

versus:

Line # Mem usage Increment Line Contents
================================================
 42 29.2 MiB 0.0 MiB @profile
 43 def get_products_unbuffered(infile):
 44 29.2 MiB 0.0 MiB with open(infile) as f:
 45 214.5 MiB 185.2 MiB lines = f.readlines()

I would be grateful for any pointers/suggestions.

You called it unbuffered, but these lines:

with open(infile) as f:
 lines = f.readlines()
 f.close()

slurp the entire file into memory, while your 'buffered' version only pulls in a line at a time, returning chunks.

I note that you're not doing anything with the entire chunk, just the lines starting with 'productNumber:', so I think a rework of your 'unbuffered' code will actually be fastest, as well as clearest:

def get_products_unbuffered(infile):
 products = 
 with open(infile) as f:
 for line in f:
 if line.startswith('productNumber:'):
 productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
 products.append(productNumber)
 return products

as this will read the file a line at a time and only keep desired info (productNumbers) in memory..