Pushing webSocket and POST performance to the limit

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0;







up vote
0
down vote

favorite












I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).



Short overview:



  1. Connects to the Site's websocket


  2. Messages are arrays of objects - loop through those


  3. If one object's "id" property is "12345", send POST to the same website


I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.



Things I thought about:




  • Use a more powerful machine than a raspberryPI or a low budget VPS



    • Would this even make a difference for such a small task?



  • Get a VPS that is as close to the website's server as possible



    • How could I find the best one? (The website is running cloudflare)



  • Change something about the code to speed up the post request



    • Is there ANY way to make this faster than using request.post()?



  • Change something to receive the websocket message earlier



    • When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?



  • Loop ... faster?!



    • Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.


  • Use something else besides NodeJS?


Here is the code:



const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);

ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])


);

function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);



Any help is appreciated - I am really trying to push this to the absolute limit here!



Have a nice evening, thank you!







share|improve this question



















  • Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
    – Jorge Bellón
    Jan 30 at 9:37










  • Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
    – Thomas Weiss
    Jan 30 at 13:25










  • If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
    – Jorge Bellón
    Jan 30 at 14:12










  • Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
    – Thomas Weiss
    Jan 30 at 21:11










  • You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
    – Jorge Bellón
    Jan 31 at 9:13
















up vote
0
down vote

favorite












I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).



Short overview:



  1. Connects to the Site's websocket


  2. Messages are arrays of objects - loop through those


  3. If one object's "id" property is "12345", send POST to the same website


I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.



Things I thought about:




  • Use a more powerful machine than a raspberryPI or a low budget VPS



    • Would this even make a difference for such a small task?



  • Get a VPS that is as close to the website's server as possible



    • How could I find the best one? (The website is running cloudflare)



  • Change something about the code to speed up the post request



    • Is there ANY way to make this faster than using request.post()?



  • Change something to receive the websocket message earlier



    • When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?



  • Loop ... faster?!



    • Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.


  • Use something else besides NodeJS?


Here is the code:



const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);

ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])


);

function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);



Any help is appreciated - I am really trying to push this to the absolute limit here!



Have a nice evening, thank you!







share|improve this question



















  • Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
    – Jorge Bellón
    Jan 30 at 9:37










  • Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
    – Thomas Weiss
    Jan 30 at 13:25










  • If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
    – Jorge Bellón
    Jan 30 at 14:12










  • Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
    – Thomas Weiss
    Jan 30 at 21:11










  • You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
    – Jorge Bellón
    Jan 31 at 9:13












up vote
0
down vote

favorite









up vote
0
down vote

favorite











I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).



Short overview:



  1. Connects to the Site's websocket


  2. Messages are arrays of objects - loop through those


  3. If one object's "id" property is "12345", send POST to the same website


I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.



Things I thought about:




  • Use a more powerful machine than a raspberryPI or a low budget VPS



    • Would this even make a difference for such a small task?



  • Get a VPS that is as close to the website's server as possible



    • How could I find the best one? (The website is running cloudflare)



  • Change something about the code to speed up the post request



    • Is there ANY way to make this faster than using request.post()?



  • Change something to receive the websocket message earlier



    • When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?



  • Loop ... faster?!



    • Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.


  • Use something else besides NodeJS?


Here is the code:



const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);

ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])


);

function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);



Any help is appreciated - I am really trying to push this to the absolute limit here!



Have a nice evening, thank you!







share|improve this question











I made a short NodeJS script to interact with a 3rd party website (their TOS officially allows automated/scripted interaction as long as it's not spamming).



Short overview:



  1. Connects to the Site's websocket


  2. Messages are arrays of objects - loop through those


  3. If one object's "id" property is "12345", send POST to the same website


I am now trying to cut the time between receiving the socket event and the POST request reaching their server by as much as possible. Counting milliseconds here.



Things I thought about:




  • Use a more powerful machine than a raspberryPI or a low budget VPS



    • Would this even make a difference for such a small task?



  • Get a VPS that is as close to the website's server as possible



    • How could I find the best one? (The website is running cloudflare)



  • Change something about the code to speed up the post request



    • Is there ANY way to make this faster than using request.post()?



  • Change something to receive the websocket message earlier



    • When i have multiple connections to the same socket opened, I can see that socket A sometimes receives two messages with 5 objects each while Socket B might receive the same 10 objects later but in a single array. Any way I can use this?



  • Loop ... faster?!



    • Array length varies between ~5 and ~100 - they come in every few seconds. Post is only being triggered once an hour or so.


  • Use something else besides NodeJS?


Here is the code:



const WebSocket = require('ws');
const ws = new WebSocket(webSocketPath);

ws.on('message', function incoming(data)
var socketData = JSON.parse(data).data;
for(var i = 0; i < socketData.length; i++)
if(socketData[i].id == "12345")
sendPost(socketData[i])


);

function sendPost(socketObject)
request.post(url:postURL, json: socketObject, function callback(error, response, body)
console.log(body);
);



Any help is appreciated - I am really trying to push this to the absolute limit here!



Have a nice evening, thank you!









share|improve this question










share|improve this question




share|improve this question









asked Jan 29 at 16:44









Thomas Weiss

1




1











  • Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
    – Jorge Bellón
    Jan 30 at 9:37










  • Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
    – Thomas Weiss
    Jan 30 at 13:25










  • If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
    – Jorge Bellón
    Jan 30 at 14:12










  • Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
    – Thomas Weiss
    Jan 30 at 21:11










  • You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
    – Jorge Bellón
    Jan 31 at 9:13
















  • Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
    – Jorge Bellón
    Jan 30 at 9:37










  • Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
    – Thomas Weiss
    Jan 30 at 13:25










  • If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
    – Jorge Bellón
    Jan 30 at 14:12










  • Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
    – Thomas Weiss
    Jan 30 at 21:11










  • You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
    – Jorge Bellón
    Jan 31 at 9:13















Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
– Jorge Bellón
Jan 30 at 9:37




Does the web service support HTTP/2.0 protocol? This allows pipelining of requests, so you can achieve a higher throughput and network efficiency (it is a binary protocol).
– Jorge Bellón
Jan 30 at 9:37












Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
– Thomas Weiss
Jan 30 at 13:25




Hey, yes it does. But does pipelining make difference? I am not sending two requests at once and I basically don't need to have the response faster (which is what pipelining does as far as I understoof). My aim is that my initial request reaches the webserver as fast as possible to trigger actions there. @JorgeBellón
– Thomas Weiss
Jan 30 at 13:25












If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
– Jorge Bellón
Jan 30 at 14:12




If every request you make is fully dependent on the previous response's content (there is a true-dependency), then pipelining does not make any difference. If your concern is your first request, then you care about latency and this would only provide you higher throughput. Have you measured the base TCP latency your client-server connection has? This can set a lower bound that could help you see how far you can get.
– Jorge Bellón
Jan 30 at 14:12












Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
– Thomas Weiss
Jan 30 at 21:11




Wow. I am not that experienced. I will try and see if I can find out how to measure TCP latency and report back. And yes, my POST request is directly triggered by a specific event on the website's websocket. If a condition there is met, it will hand this specific object to a function that will send a part of the object via POST to the website. So basically, I need to get the message fast and send the following post super fast. I tried 3 things: Running on my own PC, a raspberryPI 3 with Ubuntu and a small VPS at DigitalOcean. Thanks already! Will come back to you! @JorgeBellón
– Thomas Weiss
Jan 30 at 21:11












You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
– Jorge Bellón
Jan 31 at 9:13




You can try simple http ping or other http benchmarking tools. Just try not to flood the webserver or your IP will appear on the blacklist.
– Jorge Bellón
Jan 31 at 9:13















active

oldest

votes











Your Answer




StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
);
);
, "mathjax-editing");

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "196"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f186267%2fpushing-websocket-and-post-performance-to-the-limit%23new-answer', 'question_page');

);

Post as a guest



































active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes










 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f186267%2fpushing-websocket-and-post-performance-to-the-limit%23new-answer', 'question_page');

);

Post as a guest













































































Popular posts from this blog

Python Lists

Aion

JavaScript Array Iteration Methods