indexing - How to crawl / index pages behind a login? -


is possible (are there tools out there) crawl pages (not content, url) that's behind login? looking creating new site, , need index each page on old site in order capture content, content types, map urls new site, etc... have login , i'm not looking add google or anything.

screaming frog won't it. , can't involve dev guys of current site - putting script on server won't work either. other way this?

yes can,integrate crawler "selenium".provide login credentials , can work done. few links may you:-

how use selenium python?

http://www.quora.com/is-it-possible-to-write-a-python-script-for-opening-a-browser-and-logging-into-a-website-how-could-you-do-it

https://selenium-python.readthedocs.org/en/latest/getting-started.html

it may take time , research yes done, take care of logout page while crawling.


Comments

Post a Comment

Popular posts from this blog

r - Trouble relying on third party package imports in my package -

my package uses ggplot2 extensively make custom plots. import ggplot2 in namespace file. can build, install, , load package r without errors. when call function package, complains can't find function in imported package. function called had absolute nothing plotting , didn't make use of ggplot2 . how can make function calls package work without these types of errors? error in theme_beata() (from plot.r#14) : not find function "theme_classic" the first function erroring is: #' custom ggplot2 theme beata #' @import ggplot2 #' @family themes #' inheritparams ggplot2::theme_classic #' @export theme_beata <- function(base_size = 14, base_family="serif", ticks = true) { ret <- theme_classic() + theme(axis.line = element_line(size=0.6), axis.text = element_text(size = base_size), axis.title = element_text(size= base_size + 2), title = element_text(size=base_size + 4)) if (...
Read more

Payment information shows nothing in one page checkout page magento -

i using magento ver. 1.9.1.0 want remove shipping information , shipping method step 1 page checkout page. changed code in core file like in app\code\core\mage\checkout\block\onepage.php public function getsteps() { $steps = array(); $stepcodes = $this->_getstepcodes(); /* remove shipping address , shipping method */ $excludestep=array_diff($stepcodes,array('shipping','shipping_method')); //$excludestep=$stepcodes; if ($this->iscustomerloggedin()) { $stepcodes = array_diff($excludestep, array('login')); } foreach ($excludestep $step) { $steps[$step] = $this->getcheckout()->getstepdata($step); } return $steps; } and in app\code\core\mage\checkout\controllers\onepagecontroller.php public function savebillingaction() { if ($this->_expireajax()) { return; } if ($this->getrequest()-...
Read more