Scrape And Clicking Form With Selenium

I want to scrape the simulation 'Richiedi il tuo prestito online' of a form of this website: I tried this: driver = webdriver.PhantomJS() driver.get('https://www.findomestic.it/')

Solution 1:

print(driver) returns the string representation of WebDriver instance (driver.__str__()) and it's normal behavior

print(driver.find_element_by_tag_name('body').text) returns nothing as after you submit the form page body is empty - it contains only scripts that are not displayed on page and so text property returns empty string as expected

You need to wait for results to appear on page:

from selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get("https://www.findomestic.it/")

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.select.bh-option"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, 'AUTO NUOVA'))).click()
driver.find_element_by_id("findomestic_simulatore_javascript_importo").send_keys("2000")
driver.find_element_by_id('findomestic_simulatore_javascript_calcola').click()for item in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'ul.fd-siff-element > li')))[1:]:
    print(item.text.split('\n')[:-1])

The output should be

['56,20 € PER', '42 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']['64,10 € PER', '36 MESI', '9,53 % TAN FISSO', '9,96 % TAEG FISSO']['75,20 € PER', '30 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']['91,80 € PER', '24 MESI', '9,46 % TAN FISSO', '9,89 % TAEG FISSO']['119,70 € PER', '18 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']

Solution 2:

To scrape the simulation "Richiedi il tuo prestito online" of a form of the website you can use the following solution:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.PhantomJS(executable_path=r'C:\\Utility\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')
driver.get("https://www.findomestic.it/ ")
driver.maximize_window()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.select.bh-option"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//ul[@class='select-list bh-options_list']//li/a[text()='Auto nuova']"))).click()
driver.find_element_by_css_selector("input#findomestic_simulatore_javascript_importo").send_keys("2000")
driver.find_element_by_css_selector("input#findomestic_simulatore_javascript_calcola").submit()