How To Speed Up For Loop Execution Using Multiprocessing In Python

I have two lists. List A contains 500 words. List B contains 10000 words. I am trying to find similar words for List A with respect to B.I am using Spacy's similarity function. Th

Solution 1:

You can use multiprocessing package. This I hope will reduce your time significantly. See here for a sample code.

Solution 2:

Have you tried nlp.pipe()?

You could do something like this:

from operator import itemgetter
import spacy

nlp = spacy.load("en_core_web_lg")
ListA = ['Apples', 'Monkey']  # 500 words lists
ListB = ['Grapefruit', 'Ape', 'Oranges', 'Banana']  # 10000 words lists
s_words = []
docs_a = nlp.pipe(ListA)
docs_b = list(nlp.pipe(ListB))
for token1 in docs_a:
    list_to_sort = []
    for token2 in docs_b:
        list_to_sort.append((token1.text, token2.text, token1.similarity(token2)))
        sorted_list = sorted(list_to_sort, key=itemgetter(2), reverse=True)[0][:2]
        s_words.append(sorted_list)
print(s_words)

That should already speed things up for you. The function nlp.pipe() also has the parameter n_process which might be what you're looking for.