javascript - Cannot screen scrape site -


i trying screen scrape contents of following page (lists medical specialties) , contents of links contained in page (all universities offer training in given specialty). here main page specialties: https://services.aamc.org/eras/erasstats/par/index.cfm

the first link on page specialty adolescent medicine (pediatrics) , url https://services.aamc.org/eras/erasstats/par/display8.cfm?nav_row=par&spec_cd=321

when try directly navigate above url, server redirects me main page. however, clicking on link adolescent medicine takes me page want access.

i confused why clicking on link takes me correct page, navigating same url not. here thoughts:

1) javascript issues- disabled js , still keep getting redirected main page. moreover, clicking link in table still takes me correct page.

2) cookies- disabled cookies still ran same problem when trying access adolescent medicine page using direct url. interestingly, navigating using table links still works cookies disabled.

3) https- not sure how check if causing problem.

i have spent quite lot of time looking @ developer's console on chrome , have not been able find reason why cannot access specialty pages using direct url. or guidance appreciated!

the key making sure appropriate headers set. server apparently requires valid user agent , correct referrer set. once set header shown below, worked!

headers = { 'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8', 'accept-encoding':'gzip, deflate, sdch', 'accept-language':'en-us,en;q=0.8', 'connection':'keep-alive', 'user-agent': 'mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, gecko) chrome/39.0.2171.95 safari/537.36', 'referer' : 'https://services.aamc.org/eras/erasstats/par/index.cfm' }


Comments

Post a Comment

Popular posts from this blog

javascript - AngularJS custom datepicker directive -

i make custom datepicker directive custom template. but have no idea how start building it... how include date data directive? i appreciate guide or give me advice work on more precisely. it might you! follow below steps html code sample: <label>birth date</label> <input type="text" ng-model="birthdate" date-options="dateoptions" custom-datepicker/> <hr/> <pre>birthdate = {{birthdate}}</pre> <script type="text/ng-template" id="custom-datepicker.html"> <div class="enhanced-datepicker"> <div class="proxied-field-wrap"> <input type="text" ui-date-format="yy-mm-dd" ng-model="ngmodel" ui-date="dateoptions"/> </div> <label> <button class="btn" type="button"><i class="icon-calendar"></i>
Read more

javascript - jQuery date picker - Disable dates after the selection from the first date picker -

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>jquery ui datepicker - default functionality</title> <link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css"> <script src="//code.jquery.com/jquery-1.10.2.js"></script> <script src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script> <link rel="stylesheet" href="/resources/demos/style.css"> <script> $(function () { $("#datepicker1").datepicker(); }); </script> <script> $(function () { $("#datepicker2").datepicker(); }); </script> </head> <body> <p>date 1: <input type="text" id="datepicker1"></p> <p>date 2: <input type="t
Read more