Scraping A Website Using Scrapy And Selenium

I am going to scrape html contents on http://ntry.com/#/scores/named_ladder/main.php with Scrapy. But, because of the site's Javascript use and # , I guess I have to use Seleniu

Solution 1:

I installed Selenium and then loaded PhantomJS module and it worked perfectly.

Here is what you can try

from selenium import webdriver 
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

classFormSpider(Spider):
    name = "form"def__init__(self):

        dcap = dict(DesiredCapabilities.PHANTOMJS)
        dcap["phantomjs.page.settings.userAgent"] = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36")

        self.driver = webdriver.PhantomJS(desired_capabilities=dcap, service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any', '--web-security=false'])
        self.driver.set_window_size(1366,768)


    defparse_page(self, response):
            self.driver.get(response.url)
            cookies_list = self.driver.get_cookies()