Script That Uses Parameters And Reads Results

I am trying to write a script that takes in a URL with certain parameters, reads from the resulting web page a list of new URLs, and downloads them locally. I am very new to progra

Solution 1:

It looks like you are trying to build something similar to a web crawler, unless you want to render the content. You should explore the source code from scrapy this will help in understanding how others wrote the similar logic. I would suggest using requests library instead of urllib since it's easier. python library has builtin html, Json and XML parsers.

You should inspect the content-type header to understand what kind of content you are trying to download if the page type is unknown. There can be alternative strategies, scrapy should give you more ideas.

Hope this helps.

Solution 2:

I would recommend checking out BeautifulSoup for parsing the returned page. With it, you can loop through the links and extract the link address fairly easy and append them to a list of the links.