Pickled File Won't Load On Mac/linux

I have an application that imports data from a pickled file. It works just fine in Windows but Mac and Linux behaviour is odd. In OS X, the pickled file (file extension '.char') is

Solution 1:

Probably you didn't open the file in binary mode when writing and/or reading the pickled data. In this case newline format conversion will occur, which can break the binary data.

To open a file in binary mode you have to provide "b" as part of the mode string:

char_file = open('pickle.char', 'rb')

Solution 2:

As mentioned by Adam, the problem is likely to be the newline format of the pickle file.

Unfortunately, the real problem is actually caused on save rather than load. This may be recoverable if you're using text mode pickles, rather than binary. Try opening the file in universal newline mode, which will cause python to guess what the right line-endings are ie:

char_file=open('filename.char','rU')

However, if you're using a binary format (cPickle.dump(file, 1)) you may have an unrecoverably corrupted pickle (even when loading in Windows) - if you're lucky and no \r\n characters show up then it may work, but as soon as this occurs you could end up with corrupted data, as there's no way to distinguish between a "real" \r\n code and one windows has inserted on seeing just \n.

The best way to handle things to be loaded in multiple platforms is to always save in binary mode. On your windows machine, when saving the pickle use:

char_file = open('filename.char','wb')
cPickle.dumps(data, char_file)

Solution 3:

Another way to get this error is to forget to close the output file after pickling. This can leave an incomplete file that fails in random ways during subsequent unpickling.

Solution 4:

self.file = open(self.file_name, 'w')

Should be:

self.file = open(self.file_name, 'wb')

In your createSaveFile function, to save the file in binary mode (rather than text mode). You should also make sure you open the file in binary mode as well (rb).

If you don't use binary mode then Windows will convert all new-lines to \r\n and will effectively corrupt the file (at least as far as other OS's are concerned).

Solution 5:

Use dos2unix tool

dos2unix pickle.char