python 2.7 - Variable time between cron jobs (or similar implementation) -


here's idea -- there's website want scrape. updates every 10 minutes, get's out of sync. it's important information scrape before updates. each time check site, can scrape 'time remaining' until next update.

is there way make cron job -- after each iteration -- can set time wait before running the time (t+1) iteration based on variable time (t) iteration?

i'm not particularly familiar cron jobs -- current super rough implementation uses -sleep-. not ideal.

i had same problem , implementet basic checks in program scraping website. let crontab call program , program performs checks. if data in, program exits doing nothing. wastes little processing resources proved reliable solution.

entry in crontab :

    */10 * * * * cd /var/www/vhosts/website/scripts && /usr/bin/python scraper.py 

you can make interval shorter make fit window of updates on site.

    */5 * * * * cd /var/www/vhosts/website/scripts && /usr/bin/python scraper.py 

*/5 executes script every 5 minutes, not have "sleep" program.

i advise against manipulating crontab program/script.


Comments

Popular posts from this blog

Payment information shows nothing in one page checkout page magento -

tcpdump - How to check if server received packet (acknowledged) -