I have a Node.js app / Web API that runs on an Azure app service with a single CPU.

One of the functions needs to run for a long time, perhaps tens of seconds, while the server should continue to process other requests, i.e. the function should not be blocking.

My idea is to use await new Promise(resolve => setTimeout(resolve, 0)); at the end of each loop cycle to requeue the microtask at the end of the queue, giving other users a chance to receive the response too.

The code is processing data using the node-tfidf package like this:

const Tfidf = require('node-tfidf');
const tfidf = new Tfidf();

for (let document of documents) 
 tfidf.addDocument(document);

 // Break the blocking code:
 await new Promise(resolve => setTimeout(resolve, 0));


for (let keyword of keywords) 
 tfidf.tfidfs(keyword, function(i, measure) 
 results.add(keyword, i, measure);
 );

 // Break the blocking code:
 await new Promise(resolve => setTimeout(resolve, 0));

I want to get your comments about whether using Promise + setTimeout is a sensible way to break long-running tasks and whether there is a big performance drawback (perhaps break it every 10 loop cycles?).

Room to improve.

There is an overhead associated with interrupting the execution flow with await though it is relatively minor.

As represented in the question there is a little room for improvement.

Problems with your implementation.

Heaping closure

The way you have implemented it can be improved

await new Promise(resolve => setTimeout(resolve, 0));

You create a new function each iteration. This means there is also a new closure created for each iteration that takes heap space, an unneeded overhead, which can be avoided by pre-defining the function (see suggested improvements below).

No zero timeout

V8 (and other browsers) throttle the timeout, it is never 0ms. What it is for V8 on the version of node you are using I don't know as the throttling value has gone through changes, and may well do again.

So if you iterate 100 times and the timeout minimum is 1ms the timeout will add 100ms to the total time to complete the iteration.

The ratio of the iteration time to the timeout time needs to be controlled. If the function takes 1ms and the timeout adds 1ms you double the time to do the function. If however the function takes 100ms to do a single iteration then the timeout adds only 1% to the time to complete.

As you suggest doing the await every so many iterations will improve the ratio. But without knowing the time per iteration you can not be sure of a reasonable count.

A suggested improvement.

The await can be improved using a object to handle the promises based on time rather than iteration count.

Inside the loop you use await as follows

await idler.idle(); 

Which returns a promise or not depending on the idler settings. If the time since last idle context is greater than idler.interval a promise is returned and execution context is set to idle (allowing events). If the time is less than idler.interval then undefined is returned and execution continues without interrupt.

The following is a general purpose interrupt to allow for pending events to be processed. Ill call it idler for want of a better name.

// idler.interval is min time between idle execution context in ms
// idler.start() sets the timer to now (not really needed for small values 
// of idler.interval. Should not be called inside a process
// idler.idle() Request idle promise. Returns promise if time since last idle
// is greater than idler.interval. Else returns undefined.
export const idler = (() => 
 var lastIdle = 0;
 var interval = 3; // ms
 function timeout(ready) setTimeout(ready, 0) 
 return 
 idle() 
 var now = performance.now();
 if (now - lastIdle > interval) 
 lastIdle = now;
 return new Promise(timeout);
 
 ,
 start() lastIdle = performance.now() ,
 set interval(val) interval = val >= 1 ? val : 1 ,
 get interval() return interval ,
 ;
)();

The interface provides a way to control how often (time) to idle the execution context to allow for pending events to execute. The idle time can not be directly controlled (you can create 0 latency timeout using the message event but this has some problems)

Using the above is simple and can work over concurrent iterators. See example code below on how it can be used.

This is only a suggestion as your specific circumstance are unknown to me.

Testing concurrency

The following example uses a controllable load workLoad(time) that blocks execution context for a fixed time.

It runs several concurrent iterators that call workLoad a fixed number of times. The iterators run concurrently (time share) and illustrate how idler can manage execution context switching giving you some stats regarding the overhead.

There is also a ticker that provides additional events.

The purpose is to play with the settings to match your expected environment and workloads. You can then find the optimal settings for your needs.

const iterations = 100; // process iteration count
const load = 10; // in ms Blocking load per iteration
const processes = 6; // Number of iteration concurrent processes
const tickTime = 1000; // ms between tick calls
const idlerInterval = 20;// ms Min interval between idle execution context 
const ids = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";

/* for stats */
var tickerStart;
var totalTime = 0;
var processCount = 0;
var expectedTime = 0;
var tickCpuTime = 0;



const idler = (()=>
 var lastIdle = 0;
 var interval = 3; // ms
 function timeout(ready) setTimeout(ready,0) 
 const API = 
 idle()
 var now = performance.now();
 if(now - lastIdle > interval)
 lastIdle = now;
 return new Promise(timeout);
 
 ,
 start() lastIdle = performance.now() ,
 set interval(val) interval = val >= 1 ? val : 1 ,
 get interval() return interval ,
 
 return API;
)();

// The work function
async function longFunction(cycles = 100, load = 10, id)
 processCount += 1;
 expectedTime += cycles * load;
 const now = performance.now();
 while(cycles--)
 workLoad(load);
 await idler.idle();
 
 const time = performance.now() - now;
 processCount -= 1; 
 results(time, id);


function ticker()
 const now = performance.now();
 if(processCount > 0)
 log("Tick " + (performance.now()-tickStart).toFixed(0) + " ms ");
 setTimeout(ticker, tickTime)
 
 tickCpuTime += performance.now() - now;



log(`Starting $processes iterators with $iterations*loadms workload each.`);
idler.interval = idlerInterval;
log(`Idler interval $idlerIntervalms.`);
idler.start();
var tickStart = performance.now();
for(let i = 0; i < processes; i++)
 longFunction(iterations, load, ids[i]);

ticker();




/*========================================================================
 helper functions not related to answer
 ========================================================================*/
 
function log(data)
 datalog.innerHTML = `<div>$data</div>`+datalog.innerHTML;

function results(time,id)
 log("Id '" + id + "' took : "+(time.toFixed(3))+"ms to complete.");
 if(processCount === 0) 
 var totalTime = (performance.now() - tickStart) - tickCpuTime;
 log("===================================================");
 log("Expected time : " + (expectedTime).toFixed(0) + "ms");
 log("Actual time : " + (totalTime).toFixed(0) + "ms");
 log("Idler total overhead : " + (totalTime - expectedTime).toFixed(3) + "ms")
 const overhead = (totalTime - expectedTime)/(processes * iterations);
 log("Idler per iteration overhead : " + (overhead).toFixed(3) + "ms " + (overhead / load * 100).toFixed(2) + "% of iteration time");
 log("Ticker took : " + tickCpuTime.toFixed(3) +"ms");
 log("===================================================");
 

function workLoad(time = 1) // blocks for time in ms
 if(!isNaN(time))
 var till = performance.now() + Number(time);
 while(performance.now() < till);
 

#datalog 
 font-family : consola;
 font-size : 12px;
 color : #0F0;

body 
 background : black;

<div id="datalog"></div>

Notes:

• You should cut and past into a clean environment. Running on this page adds unknown additional executing context's.




• The log function is slow (uses markup insertion) and its time has been mostly ignored. The results will therefor be slightly higher than actual.

I fear this might not be the best approach depending on the scale your server application is aiming at:

• You have to clutter your code with these instructions. Especially the more time granularity you would like to have, the more statements you have to insert.




• The contract that each computing entity regularly gives away its computation time is not enforced at a single place, instead, it is up to every entity itself. If one entity hangs, your complete system hangs. (Actually, that's one of the reasons why general-purpose operating systems are preemptive.)

Several people and I explored this back then in a Flow-Based-Programming implementation for JS: https://github.com/jpaulm/jsfbp.

I'd consider the approach valid for small environments, but would prefer a different way on the global layer, e.g.

• real threads
  
  There are some extensions/plugins for Node.js for this. Maybe someone can comment on how clean and easy this is.


• process spawning

Then again, inside a thread or a process, you can employ the green threads/fibers/JS-esque multithreading approach again - if you feel the need for it, e.g. because of easy reasoning about concurrency issues.