What Are Some Viable Strategies To Detecting Duplicates In A Large Json File When You Need To Store The Duplicates?

I have an extremely large set of data stored in json that is too large to load in memory. The json fields contain data about users and some metadata - however, there are certainly

Solution 1:

You can partition the records by hash value into smaller sets that fit into memory, remove duplicates in each set, and then reassemble them back into one file.