trjhtr

Each line of the JSON has a structure looks as the following

|- Sentence
| |-- WordPosition:
| |-- |-- Candidate
| |-- |-- |-- index:
| |-- |-- |-- retrievals

Example of a line in JSON output (parameter num_retrieval=2)

Current implementation

def output_conll_retrievals_to_json(output_file, conll_mcs,conll_sentences, num_retrieval):

 num_sentence = len(conll_sentences)

 with open(output_file, 'w') as f:
 for i in range(num_sentence):
 num_position = len(conll_sentences[i])
 level2 = dict()
 level1 = dict()
 for j in range(num_position):
 level0 = dict()
 for mc in conll_mcs[i][j]:
 mc_str = ' '.join(mc.mention_string)
 level0[mc_str] = dict()
 level0[mc_str]['index'] = mc.mention
 level0[mc_str]['retrievals'] = self.get_one_mc_retrievals(mc, num_retrieval)
 level1[conll_sentences[i][j]] = level0
 level2[' '.join(conll_sentences[i])] = level1
 json.dump(level2, f)
 f.write('n')

Note that self.get_one_mc_retrievals is implemented in parallel, accessing a memory map and performs a hierarchical search. Executing the snippet of code, status showing that CPUs are under-used (like 10-20% for each core as writing is serial programming) and the whole JSON output takes 8 days (approximately 3000 sentences with num_retrievals of 100) to generate. Would it be possible to boost the speed (i.e., Cython, parallel or something else)?

** Edited **

This just comes off my head, would it be possible to utilize multiprocessing to speed up JSON writing? Right now, my code is kind of serially doing a process that retrieves a dict from reading conll_mcs[i] and writes a line to the JSON file. If I could simultaneously perform many of such a process, speed will very likely improve, however I am still trying to find how to put such an idea into practice. Any help would be greatly appreciated.