Pushing webSocket and POST performance to the limit

Clash Royale CLAN TAG#URR8PPP
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;
up vote
0
down vote
favorite
I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).
Short overview:
Connects to the Site's websocket
Messages are arrays of objects - loop through those
If one object's "id" property is "12345", send POST to the same website
I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.
Things I thought about:
Use a more powerful machine than a raspberryPI or a low budget VPS
- Would this even make a difference for such a small task?
Get a VPS that is as close to the website's server as possible
- How could I find the best one? (The website is running cloudflare)
Change something about the code to speed up the post request
- Is there ANY way to make this faster than using request.post()?
Change something to receive the websocket message earlier
- When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?
Loop ... faster?!
- Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.
Use something else besides NodeJS?
Here is the code:
const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);
ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])
);
function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);
Any help is appreciated - I am really trying to push this to the absolute limit here!
Have a nice evening, thank you!
javascript performance node.js websocket
 |Â
show 2 more comments
up vote
0
down vote
favorite
I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).
Short overview:
Connects to the Site's websocket
Messages are arrays of objects - loop through those
If one object's "id" property is "12345", send POST to the same website
I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.
Things I thought about:
Use a more powerful machine than a raspberryPI or a low budget VPS
- Would this even make a difference for such a small task?
Get a VPS that is as close to the website's server as possible
- How could I find the best one? (The website is running cloudflare)
Change something about the code to speed up the post request
- Is there ANY way to make this faster than using request.post()?
Change something to receive the websocket message earlier
- When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?
Loop ... faster?!
- Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.
Use something else besides NodeJS?
Here is the code:
const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);
ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])
);
function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);
Any help is appreciated - I am really trying to push this to the absolute limit here!
Have a nice evening, thank you!
javascript performance node.js websocket
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13
 |Â
show 2 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).
Short overview:
Connects to the Site's websocket
Messages are arrays of objects - loop through those
If one object's "id" property is "12345", send POST to the same website
I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.
Things I thought about:
Use a more powerful machine than a raspberryPI or a low budget VPS
- Would this even make a difference for such a small task?
Get a VPS that is as close to the website's server as possible
- How could I find the best one? (The website is running cloudflare)
Change something about the code to speed up the post request
- Is there ANY way to make this faster than using request.post()?
Change something to receive the websocket message earlier
- When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?
Loop ... faster?!
- Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.
Use something else besides NodeJS?
Here is the code:
const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);
ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])
);
function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);
Any help is appreciated - I am really trying to push this to the absolute limit here!
Have a nice evening, thank you!
javascript performance node.js websocket
I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).
Short overview:
Connects to the Site's websocket
Messages are arrays of objects - loop through those
If one object's "id" property is "12345", send POST to the same website
I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.
Things I thought about:
Use a more powerful machine than a raspberryPI or a low budget VPS
- Would this even make a difference for such a small task?
Get a VPS that is as close to the website's server as possible
- How could I find the best one? (The website is running cloudflare)
Change something about the code to speed up the post request
- Is there ANY way to make this faster than using request.post()?
Change something to receive the websocket message earlier
- When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?
Loop ... faster?!
- Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.
Use something else besides NodeJS?
Here is the code:
const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);
ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])
);
function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);
Any help is appreciated - I am really trying to push this to the absolute limit here!
Have a nice evening, thank you!
javascript performance node.js websocket
asked Jan 29 at 16:44
Thomas Weiss
1
1
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13
 |Â
show 2 more comments
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13
 |Â
show 2 more comments
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f186267%2fpushing-websocket-and-post-performance-to-the-limit%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
â Jorge Bellón
Jan 30 at 9:37
Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
â Thomas Weiss
Jan 30 at 13:25
If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
â Jorge Bellón
Jan 30 at 14:12
Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
â Thomas Weiss
Jan 30 at 21:11
You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
â Jorge Bellón
Jan 31 at 9:13