c# - Parallel async HttpWebRequests with different proxies. How to maximize performance? -

i'm writing app needs lot of parallel httpwebrequests bookmaker site different proxies. reason why i'm using different proxies bookmaker can ban ip if there many requests it. goal freshest site content fast possible. here code settings have:

        servicepointmanager.defaultconnectionlimit = 1000;         servicepointmanager.expect100continue = false;         servicepointmanager.usenaglealgorithm = false;          (var = 0; < proxycollection.count; i++)         {             var proxylocal = proxycollection[i];             var ilocal = i;             task.run(async () =>                 {                     var httpwebrequest = (httpwebrequest) webrequest.create(string.format("https://bookmaker.com/page{0}", ilocal));                     httpwebrequest.proxy = proxylocal;                     httpwebrequest.preauthenticate = true;                     httpwebrequest.automaticdecompression = decompressionmethods.gzip | decompressionmethods.deflate;                      using (var httpwebresponse = (httpwebresponse) await httpwebrequest.getresponseasync())                     using (var responsestream = httpwebresponse.getresponsestream())                     using (var streamreader = new streamreader(responsestream))                     {                         var stringcontent = await streamreader.readtoendasync();                         //here i'm processing new data. works pretty fast, isn't problem                         processstringcontent(stringcontent);                     }                 });                      } 

and code works not fast expected.

first problem in strange reason requests don't start @ 1 time. can see in task manager loading has 2 or more maximums. more on have 1 realy fast proxy , proxycollection contains it. if use await task.delay(5000) after code above, in cases moment when 5000ms have passed request fast proxy doesn't start!

second problem total time of task execution is slow. expected if 1 request needs 200-300ms execute, 100 requests in parallel , async needs little more time. times "more" 10-20 times. have suspicion, wrong.

third problem when i'm running this code freezes ui (not full freeze, ui lags). read webrequest.create processing synchronously , can time (dns lookup, proxy settings e.t.c) , if i'm making lot of requests can fill threads (ui thread too) creating webrequests. tried create requests direct ip address (webrequest.create(string.format("https://10.10.10.1/page{0}", ilocal)) - nothing changed, , i'm setting proxy (so auto detect proxy isn't needed), don't understand why can creating take time (and problem creating or maybe smth else?).

please can point me i'm doing wrong? i'm lost in servicepointmanager settings (all tried didn't help). can .net deal such type of task? or maybe need use nodejs example best performance?

p.s: collection of proxies not such big (50-100 different proxies).

back (parallel) basics: don't block threads, make async i/o work you

example:

parallel.for(0, 10, delegate(int i) {     httpwebrequest request = (httpwebrequest)webrequest.create(         new uri("http://www.mysi.com"));      string datatosend = "data";     byte[] buffer = system.text.encoding.getencoding(1252).         getbytes(datatosend);     request.method = "post";     request.contenttype = "application/x-www-form-urlencoded";     request.contentlength = buffer.length;      request.host = "www.mysite.com";      stream requeststream = request.getrequeststream();     requeststream.write(buffer, 0, buffer.length);     requeststream.close();      httpwebresponse response = (httpwebresponse)request.getresponse(); }); 

send multiple webrequest in parallel.for