Calling An Api Concurrently In Python

I need to talk to an api to get information about teams. Each team has a unique id. I call the api with that id, and I get a list of players on each team (list of dicts). One of th

Solution 1:

One solution would be to:

• prepare a list of tasks to perform, in your case list of teams IDs to be processed,
• create fixed pool of N thread workers,
• each worker thread pops a task from the list and processes the task (downloads team data), after completion it pops another task,
• when task list is empty, the worker thread stops.

This solution could safe you from the case when processing of a particular team takes e.g. 100 time units, when other teams are processed in 1 time unit (on an average).

You can tune number of thread workers depending on number of teams, average team processing time, number of CPU cores etc.

Extended answer

This can be achieved with the Python multiprocessing.Pool:

from multiprocessing import Pool

defapi_call(id):
    pass# call API for given idif __name__ == '__main__':
    p = Pool(5)
    p.map(api_call, [1, 2, 3])