Selenium.common.exceptions.nosuchelementexception: Message: Web Element Reference Not Seen Before Using Geckodriver Firefox And Selenium With Python

I'm trying to get the data for several different tests from a test prep site. There are different subjects, each of which has a specialization, each of which has a practice-test,

Solution 1:

This error message...

selenium.common.exceptions.NoSuchElementException: Message: Web element reference not seen before: 980e5c29-e3af-4b13-979f-0f2bb58b3480

...implies that the GeckoDriver was unable identify the WebElement.

This error is coming out from get(webEl, win) within the Marionette source code:

get(webEl, win) {
  if (!(webEl instanceofWebElement)) {
    thrownewTypeError(pprint`Expected web element, got: ${webEl}`);
  }
  if (!this.has(webEl)) {
    thrownewNoSuchElementError(
      "Web element reference not seen before: " + webEl.uuid
    );
  }

@fc's comment in the discussion 'Element reference not seen before: undefined' using geckodriver, waitForElementVisible fails explains the actual issue:

---

However, the core issue was discussed in Intermittent test_navigation.py TestRefresh.test_basic | NoSuchElementException: Failed to trigger opening a new tab: Web element reference not seen before and was subsequently and was addressed through the changeset

---

Solution

Using the latest version of the binaries will solve the issue in terms of:

• Selenium v3.141.59
• GeckoDriver v0.26.0
• Firefox v70.0

Solution 2:

The problem is the iteration over practices. It holds WebElements, but thier reference is lost when you are navigating to a new page, even if it actually the same page Selenium treats it as a new one.

You can solve it by iterating by index. To do it with zip you can do something like this

practices_len = len(list(get_practices(subject_name, specialization_name)))
for i in range(practices_len):
    practices_list = list(get_practices(subject_name, specialization_name))
    practice = {}
    practice['name'] = practices_list[i][0]
    practices_list[i][1].click()

Solution 3:

Guy is right. The next time you load the specialization_url it is a new page with new elements but practices contains the web elements of the old page as web elements.

To only change the part where it happens the code below first creates a list of the practices and practice_clickables. It then searches for a fresh clickable whenever it returns to the new specialization_url page and prints the ID of the old and the current practice_clickable. With that it is clearly visible that the element in the same row now is a different one than it was the first time the page was loaded.

In addition the map or zip function seems to create a generator so that even the iteration over the practices fails because in that step webdriver code is executed on old objects. That is why I first create lists and iterate over a list.

Changed snippet:

practices = get_practices(subject_name, specialization_name)
practice_clickable = [item[1] for item in practices]
practices = get_practices(subject_name, specialization_name)
practices = [item[0] for item in practices]
forindex, practice_name in enumerate(practices):
    practice={}
    practice['name'] = practice_name
    practice_row = driver.find_element_by_xpath(f'//*[text()="{practice_name}"]/..')
    practice_clickable_n = practice_row.find_element_by_link_text('Begin')
    print('old:', practice_clickable[index])
    print('new:', practice_clickable_n)
    practice_clickable_n.click()
    questions=get_questions(subject_name, specialization_name, practice_name)

Complete scrape function:

def scrape():
    setup()
    subjects=get_subjects()
    forsubject_name, subject_clickable in subjects:
        subject={}
        subject['name']=subject_name
        subject['specializations']=[]
        subject_clickable.click()
        if ('http://') in driver.current_url:
            subject_url=driver.current_url.replace('http://', 'https://')else:
            subject_url=driver.current_url
        specializations=get_specializations(subject_name)
        forspecialization_name, specialization_clickable in specializations:
            specialization={}
            specialization['name']=specialization_name
            specialization['practices']=[]
            specialization_clickable.click()
            if'http://' in driver.current_url:
                specialization_url=driver.current_url.replace('http://', 'https://')else:
                specialization_url=driver.current_url
            practices = get_practices(subject_name, specialization_name)
            practice_clickable = [item[1] foritemin practices]
            practices = get_practices(subject_name, specialization_name)
            practices = [item[0] foritemin practices]
            forindex, practice_name inenumerate(practices):
                practice={}
                practice['name'] = practice_name
                practice_row = driver.find_element_by_xpath(f'//*[text()="{practice_name}"]/..')
                practice_clickable_n = practice_row.find_element_by_link_text('Begin')
                print('old:', practice_clickable[index])
                print('new:', practice_clickable_n)
                practice_clickable_n.click()
                questions=get_questions(subject_name, specialization_name, practice_name)
                practice['questions']=questions
                driver.get(specialization_url)
            driver.get(subject_url)
        data.append(subject)
    print(data)