Optimize nutch performance on hadoop cluster -


i'm trying optimize nutch performance crawling sites. test performance on small hadoop cluster, 2 nodes 32gb ram, cpu intel xeon e3 1245v2 4c/8t. config nutch http://pastebin.com/bbrhpfuq

so, problem: fetching jobs works not optimal. reduce task has 4k pages fetching, 1kk pages. example see screenshot https://docs.google.com/file/d/0b98dgnxoqkmvt1doovvpuu1pnxm/edit reduce task finished in 10 minutes, 1 task work 11 hours , still continue working, it's bottle neck when have 24 reduce task, works one.

may can give usefull advices or links can read problem.

it problem in nutch, takes 50 000 000 1 site , 500 000 other. when creating queue host see 1 extremely big queue , other small.


Comments

Post a Comment

Popular posts from this blog

javascript - AngularJS custom datepicker directive -

i make custom datepicker directive custom template. but have no idea how start building it... how include date data directive? i appreciate guide or give me advice work on more precisely. it might you! follow below steps html code sample: <label>birth date</label> <input type="text" ng-model="birthdate" date-options="dateoptions" custom-datepicker/> <hr/> <pre>birthdate = {{birthdate}}</pre> <script type="text/ng-template" id="custom-datepicker.html"> <div class="enhanced-datepicker"> <div class="proxied-field-wrap"> <input type="text" ui-date-format="yy-mm-dd" ng-model="ngmodel" ui-date="dateoptions"/> </div> <label> <button class="btn" type="button"><i class="icon-calendar"></i>
Read more

javascript - jQuery date picker - Disable dates after the selection from the first date picker -

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>jquery ui datepicker - default functionality</title> <link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css"> <script src="//code.jquery.com/jquery-1.10.2.js"></script> <script src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script> <link rel="stylesheet" href="/resources/demos/style.css"> <script> $(function () { $("#datepicker1").datepicker(); }); </script> <script> $(function () { $("#datepicker2").datepicker(); }); </script> </head> <body> <p>date 1: <input type="text" id="datepicker1"></p> <p>date 2: <input type="t
Read more