How Can I Get Match Information With Python

I have a problem with extracting data using the request library. I want to get the time of the match, the name of the teams, the odds for the result from the nesine.com Example: Ma

Solution 1:

It's first worth understanding why you can't use requests to retrieve this data in the way you are doing, but also how you may be able to do it.

requests workflow is rather quite simple - the .get() function sends an HTTP request to the server requesting relevant resources about the page. The server then replies with the relevant HTML, CSS, and Javascript that composes the page. This server, like many, does not reply with the entirety of the page, and instead, using the Javascript in which the server responds back to the client it further loads more information into the page. This is visually obvious when you load the website and you see a "loading" symbol inside the frame in which you are trying to request information from, but it becomes more apparent when you do a quick analysis in the Network tab of your Inspect Element.

At around 1,500ms to 2,500ms my browser sends a new request to the an API endpoint that seems to pull the relevant information that you seek (I only knew this after further digging around - not some crazy intuition). Compare this with http://example.com, you can see how a simple HTML & CSS website only replies with the initial GET data:

To extract the information you are looking for, then, you must look through the Network tab and see how your browser is pulling the data from the server. Doing a quick "Find" for the word Boluspor we get the following results:

Now that we know where the API request is, then we can query it with requests to get the relevant information. I recommend first clicking the url of where the actual results are coming in from and select "Copy as fetch", which you will get the following:

fetch("https://bulten.nesine.com/api/bulten/getprebultenfull", {
  "headers": {
    "accept": "application/json, text/javascript, */*; q=0.01",
    "accept-language": "en-US,en;q=0.9",
    "authorization": "Basic RDQ3MDc4RDMtNjcwQi00OUJBLTgxNUYtM0IyMjI2MTM1MTZCOkI4MzJCQjZGLTQwMjgtNDIwNS05NjFELTg1N0QxRTZEOTk0OA==",
    "cache-control": "no-cache",
    "content-type": "application/json; charset=utf-8",
    "pragma": "no-cache",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-site"
  },
  "referrer": "https://www.nesine.com/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": null,
  "method": "GET",
  "mode": "cors",
  "credentials": "include"
});

The above alone doesn't seem to indicate that there is anything special going on in the request (e.g. headers, params, etc.), so accessing the endpoint alone should work -- and that is the case:

https://bulten.nesine.com/api/bulten/getprebultenfull

You can then use requests to grab that page, load it via the json module, and begin organizing the data.