python - Amazon web scraping -


i'm trying scrape amazon prices phantomjs , python. want parse beautiful soup, new , used prices books, problem is: when pass source of request phantomjs prices 0,00, code simple test.

i'm new in web scraping don't understand if amazon have measures avoid scraping prices or i'm doing wrong because trying other more simple pages , can data want.

pd i'm in country not supported use amazon api, that's why scraper necesary

import re import urlparse  selenium import webdriver bs4 import beautifulsoup time import sleep  link = 'http://www.amazon.com/gp/offer-listing/1119998956/ref=dp_olp_new?ie=utf8&condition=new'#'http://www.amazon.com/gp/product/1119998956'  class amzonscraper(object):     def __init__(self):         self.driver = webdriver.phantomjs()         self.driver.set_window_size(1120, 550)      def scrape_prices(self):         self.driver.get(link)         s = beautifulsoup(self.driver.page_source)         return s      def scrape(self):         source = self.scrape_prices()         print source         self.driver.quit()  if __name__ == '__main__':     scraper = taleojobscraper()     scraper.scrape()

first of all, follow @nick bailey's comment, study terms of use , make sure there no violations on side.

to solve it, need tweak phantomjs desired capabilities:

caps = webdriver.desiredcapabilities.phantomjs caps["phantomjs.page.settings.useragent"] = "mozilla/5.0 (x11; linux x86_64) applewebkit/53 (khtml, gecko) chrome/15.0.87"  self.driver = webdriver.phantomjs(desired_capabilities=caps) self.driver.maximize_window()

and, make bullet-proof, can make custom expected condition , wait price become non-zero:

from selenium.common.exceptions import staleelementreferenceexception selenium.webdriver.common.by import selenium.webdriver.support.ui import webdriverwait selenium.webdriver.support import expected_conditions ec  class wait_for_price(object):     def __init__(self, locator):         self.locator = locator      def __call__(self, driver):         try :             element_text = ec._find_element(driver, self.locator).text.strip()             return element_text != "0,00"         except staleelementreferenceexception:             return false

usage:

def scrape_prices(self):     self.driver.get(link)      webdriverwait(self.driver, 200).until(wait_for_price((by.class_name, "olpofferprice")))     s = beautifulsoup(self.driver.page_source)      return s

Comments

Post a Comment

Popular posts from this blog

cakephp - simple blog with croogo -

i not problem, made simple blog in croogo, problem is not adding or modifying or delete, because arabic language ?? function add in controller `public function admin_add() { $this->set('title_for_layout', __('add part')); if (!empty($this->request->data)) { $this->part->create(); if ($this->part->save($this->request->data)) { $this->session->setflash(__('the part has been saved'), 'default', array('class' => 'success')); $this->redirect(array('action' => 'index'));//, $this->part->id)); } else { $this->session->setflash(__('the part not saved. please, try again.'), 'default', array('class' => 'error')); } } }` => model `class part extends appmodel { public $name = 'part'; var $belongst...
Read more

How to group boxplot outliers in gnuplot -

i have large set of data points. try plot them boxplot, of outliers exact same value , represented on line beside each other. found how set horizontal distance between outliers in gnuplot boxplot , doesn't much, apparently not possible. is possible group outliers together, print 1 point , print number in brackets beside indicate how many points there are? think make more readable in graph. for information, have 3 boxplots 1 x value , times 6 in 1 graph. using gnuplot 5 , played around pointsize, doesn't reduce distance anymore. hope can help! edit: set terminal pdf set output 'dat.pdf' file0 = 'dat1.dat' file1 = 'dat2.dat' file2 = 'dat3.dat' set pointsize 0.2 set notitle set xlabel 'x' set ylabel 'y' header = system('head -1 '.file0); n = words(header) set xtics ('' 1) set [i=1:n] xtics add (word(header, i) i) set style data boxplot plot file0 using (1-0.25):1:(0.2) boxplot lw 2 lc rgb '#8b0000...
Read more

bash - Performing variable substitution in a string -

i have string contains use of variable, , i'd substitute variable's value string. right best have is #!/bin/bash foo='$name; echo bar' name="the name" expanded="$(eval echo "$foo")" echo "$expanded" which has obvious defects: prints the name bar while i'd print the name; echo bar instead of eval can bash's regex matching bash's string replacement: foo='$name; echo bar' name="the name" [[ "$foo" =~ \$([[:alnum:]]+) ]] && s="${!bash_rematch[1]}" && expanded="${foo/\$${bash_rematch[1]}/$s}" echo "$expanded" name; echo bar
Read more