trjhtr

My program works with two types of files.
File 1 contains 500,000 distinct words.
File set 2 contains 173 text files, each containing 500 paragraphs, that I scraped from Wikipedia.
The program counts how many times each word from the first file appears in the second set of files.

The main problem I have is that it's taking around 4 seconds per word to process so it will take around 24 days to complete all 500k words in a core5 7th gen 8gb ram laptop. Is it possible to make it more process efficient?

I am still learning Java so my knowledge is not that vast. I am using Java 8, with IntelliJ as my IDE.

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.*;

public class Main 

public static void main(String args) 

 //This is the map that will contain each word
 Map<String, Integer> map = new HashMap<>();
 //int that will count how manny times the word is in the File set 2
 int wordCounter = 0;
 //List that contain arround 500k unrepeted words
 List<String> list = new ArrayList<>();
 //List that contains the current file words
 List<String> list1 = new ArrayList<>();


 try 
 //scans the file that contains the 500k unrepeted words
 Scanner s = new Scanner(new File("C:\Users\filepath"));
 //while loop that add the words to a list so it can manipulate it latter on
 while (s.hasNext()) 
 list.add(s.next());
 
 //random output to see the Set size
 System.out.println(list.size());


 //main loop that will cheek each word in the 500k file
 for (int i = 0; i < list.size(); i++) 
 //loop to se each file of words
 for (int j = 0; j < 100; j++) 
 try 
 //read each file
 Scanner d = new Scanner(new File("C:\Users\filepath" + j));
 //add the information of each file
 while (d.hasNext()) 
 list1.add(d.next());
 
 d.close();
 //this code counts the number of words in all the files a
 wordCounter = wordCounter + Collections.frequency(list1, list.get(i).toLowerCase());
 //clears the list so it has more space and not run out of it
 list1.clear();

 catch (IOException k) 
 k.printStackTrace();
 
 
 //adds the information to the map
 map.put(list.get(i), wordCounter);
 //this sorts the information and discard the words that only has 1 or less matches
 if (wordCounter > 1) 
 try 
 FileWriter fw = new FileWriter("C:\Users\filePath", true);
 PrintWriter pw = new PrintWriter(fw);
 pw.append("n");
 pw.append(map.toString());
 pw.close();

 catch (IOException f) 
 f.printStackTrace();
 
 
 //this clean the map so it doesnt run out of memory
 map.clear();
 //resets the counter to 0
 wordCounter = 0;
 //simple display so it seems nice
 System.out.println(i);

 
 catch (IOException f)
 f.printStackTrace();
 



 

 

Somewhere I read that because of Java using a virtual machine it makes the processing of data much slower. Would that be something to consider?

You should try to switch inner and outer loop, because it will be much faster to read each wikipedia articles in and count the word frequencies for all 500k words (you anyway have the 500k word list in memory all the time). What you do now is reading 500k times all articles into memory which is time consuming.

To sum up all usages of a word, you can use the map which already exists. Just read out the current sum for a given word, add the occurrences in the current article and write it back to the map. Right now you are just writing one entry to the map, convert it to a string and clear it right afterwards. I assume, you had the idea in mind to do it how I described.

Don't worry about Java execution speed in general, because the code will get compiled (just-in-time-compilation) to machine code eventually.

First of all you should really use more meaningful names for your variables.

wordCount says a lot more than map, uniqueWords a lot more than list and same for wordsInCurrentFile instead of list1.

Just renaming these makes it so much easier to follow what your program is doing.

You should follow Markus's advice to flip the loops. In the outer loop, you should iterate over each file. And then for each file, count the occurences for each of the words.

We can also optimise the use of our variables a bit so we don't even need the 2 lists at all.

Here's the main idea:

Map<String, int> wordCount = new HashMap<>();
File wordFile = new File( ...); //open the file with unique words
while(wordFile.hasNext())
 wordCount.put(wordFile.next(), 0);

At this point we got an entry in our count map for each of the 500k words.

for( each file with wiki text) 

loop over each file in outer loop. Either the same way, opening the file like you did now based on a fixed name and number appended. Or by putting all the files in a specific directory and in java iterate over all files you find in that directory.

 while(file.hasNext())
 String word = file.next();

At this point we're looping over each word in the file and want to update the total count.

 Map<String, Integer> wordCount = new HashMap<>();
 if (wordCount.containsKey(word)) 
 wordCount.put(word, wordCount.get(word)+1);
 
 

It may also be worth looking up how to list all files in a directory in java 8 (I'm not familiar with this myself yet).

And it's also a good idea to look up how to use a try-with-resources since java 7 which helps you handling closing the file after you're done with it. The way you wrote it now doesn't properly close the file if you encounter an error.