trjhtr

What am I trying to do

I'm trying to scan a disk and save (cache) all the paths of files and directories into an array so that I could perform a quick file/directory search later on.

Implementation

I'm using a Node.js module called walkdir.

cacheDisk () 
 let searchDir = 'C:/'
 let cache = 

 let options = 
 "max_depth": 5, // only recurse down to max_depth. if you need more than no_recurse
 "track_inodes": true // should be used with max_depth to prevent infinite loop. On windows or with hardlinks some files are not emitted due to inode collision.
 

 let paths = walkdir(searchDir, options, (path) => 
 cache.push(path)
 )
 .on('end', () => 
 console.log(cache)
 this.findItem(cache)
 )
,

findItem (cache) 
 let filename = 'file_1.txt'
 const matches = cache.filter(element => element.includes(filename))
 console.log(matches)

Questions / Problems:

FYI: I only need it to work in the latest Chrome (it's an Electron app)

• Is using cache.push(path) a good way (performance-wise) to save paths to the array or is the module already doing it for me and I'm just repeating this action on every iteration? It takes about 30 seconds and 500-800 MB of RAM on a core i7 to finish that caching operation for my ssd with 1 million paths

• Is using cache.filter(element => element.includes(filename)) a good / safe way to find an element in an array that can potentially be ~10,000,000 paths in length?

• When I'm trying to scan disk's root (e.g. C:/) I get 2 slashes after the disk letter in every path (e.g. C:\Program Filestest). But when I'm scanning something like C:/test/ everything is fine. I guess that's Node's doing? How do I avoid that? (do I just replace all \ with in all the paths afterwards?)

I have to ask why you would want to find a file anywhere on the filesystem. That seems like a pretty odd requirement. Most apps (take node module loading for example) check a limited number of folders which can be done very quickly.

To answer your questions:

1. the OS does not do this kind of caching.



2. 
   

   No, I wouldn't consider this safe. For two reasons. There are many other names that could match. For example video_file_1.txt, file_1.txt.old or some_path/file_1.txt/cat.jpg. I would check against the basename something like

   
   
   

   function basename(path) 
    return path.match(/[^\/]+$/)[0];
   
   const matches = cache.filter(element => basename(element) == filename) 
   

   
   



3. Someone already answered this in the comments. It is really just an artifact of attempting to handle windows and unix paths consistently.

Also if retrieval efficiency is important then I would store the data as a hash not an array, i.e:

let cache = ;

let paths = walkdir(searchDir, options, (path) => 
 const name = basename(path);
 if (!cache[basename])
 cache[basename] = ; 
 cache[basename].push(path)
);

Then finding an element is a simple matter of:

findItem (cache) 
 let filename = 'file_1.txt';
 const matches = cache[filename];
 console.log(matches);

Note that, since you are on Windows, you should also consider case-insensitivity of filenames.