Optimize nutch performance on hadoop cluster -


i'm trying optimize nutch performance crawling sites. test performance on small hadoop cluster, 2 nodes 32gb ram, cpu intel xeon e3 1245v2 4c/8t. config nutch http://pastebin.com/bbrhpfuq

so, problem: fetching jobs works not optimal. reduce task has 4k pages fetching, 1kk pages. example see screenshot https://docs.google.com/file/d/0b98dgnxoqkmvt1doovvpuu1pnxm/edit reduce task finished in 10 minutes, 1 task work 11 hours , still continue working, it's bottle neck when have 24 reduce task, works one.

may can give usefull advices or links can read problem.

it problem in nutch, takes 50 000 000 1 site , 500 000 other. when creating queue host see 1 extremely big queue , other small.


Comments

Post a Comment

Popular posts from this blog

Payment information shows nothing in one page checkout page magento -

i using magento ver. 1.9.1.0 want remove shipping information , shipping method step 1 page checkout page. changed code in core file like in app\code\core\mage\checkout\block\onepage.php public function getsteps() { $steps = array(); $stepcodes = $this->_getstepcodes(); /* remove shipping address , shipping method */ $excludestep=array_diff($stepcodes,array('shipping','shipping_method')); //$excludestep=$stepcodes; if ($this->iscustomerloggedin()) { $stepcodes = array_diff($excludestep, array('login')); } foreach ($excludestep $step) { $steps[$step] = $this->getcheckout()->getstepdata($step); } return $steps; } and in app\code\core\mage\checkout\controllers\onepagecontroller.php public function savebillingaction() { if ($this->_expireajax()) { return; } if ($this->getrequest()-...
Read more

tcpdump - How to check if server received packet (acknowledged) -

Image
i running tcpdump on server capture traffic between server (servera) , remote server (serverb). not know how read output tcpdump. point out if serverb acknowledging packets being sent servera being received serverb? here tcpdump log: 2015-03-31 11:32:18.038284 ip (tos 0x0, ttl 64, id 30781, offset 0, flags [df], proto tcp (6), length 60) servera.14033 > serverb.8004: flags [s], cksum 0x02bd (correct), seq 337482636, win 14600, options [mss 1460,nop,nop,ts val 506588081 ecr 0,nop,wscale 7], length 0 0x0000: 4500 003c 783d 4000 4006 bae4 0a04 7a0e e.. servera.14033: flags [s.], cksum 0x3b2a (correct), seq 1742982091, ack 337482637, win 4140, options [mss 1380,nop,wscale 0], length 0 0x0000: 4500 0030 5ff2 0000 f606 5d3b cefd b48a e..0_.....];.... 0x0010: 0a04 7a0e 1f44 36d1 67e3 cbcb 141d 938d ..z..d6.g....... 0x0020: 7012 102c 3b2a 0000 0204 0564 0103 0300 p..,;*.....d.... 2015-03-31 11:32:18.078622 ip (tos 0x0, ttl 64, id 30782, offset 0, flags [...
Read more