How To Scrape Multiple Pages From One Site

I want to scrap multiple pages from one site.the pattern like this： https://www.example.com/S1-3-1.html https://www.example.com/S1-3-2.html https://www.example.com/S1-3-3.html ht

Solution 1:

Your indentation is out of order.

try(Method 1)

from bs4 import BeautifulSoup 
import requests

for i inrange(1, 6):      # Number of pages plus one 
    url = "https://www.example.com/S1-3-{}.html".format(i)
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')  
    results = soup.find_all('div', attrs={'class':'product-item item-template-0 alternative'})

Solution 2:

Your page analysis should be inside the loop, like this, otherwise, it will only use one page:

.......
    for i inrange(5):      # Number of pages plus one 
        url = "https://www.example.com/S1-3-{}.html".format(i)
        r = requests.get(url)
        from bs4 import BeautifulSoup  
        soup = BeautifulSoup(r.text, 'html.parser')  
        results = soup.find_all('div', attrs={'class':'product-item item-template-0 alternative'})
........

Solution 3:

Firstly, you have to introduce all orders inside of the loop, otherwise, only will work with the last iteration.

Second, You could try closing the requests session at the end of each iteration:

import requests  
    for i inrange(5):      # Number of pages plus one 
    url = "https://www.example.com/S1-3-{}.html".format(i)
    r = requests.get(url)
    from bs4 import BeautifulSoup  
    soup = BeautifulSoup(r.text, 'html.parser')  
    results = soup.find_all('div', attrs={'class':'product-item item-template-0 alternative'})
    r.close()