`pickle`: Yet Another `importerror: No Module Named My_module`

I have a class MyClass defined in my_module. MyClass has a method pickle_myself which pickles the instance of the class in question: def pickle_myself(self, pkl_file_path): wit

Solution 1:

I can tell from your question that you are probably doing something like this, with a class method that is attempting to pickle the instance of the class. It's ill-advised to do that, if you are doing that… it's much more sane to use pkl.dump external to the class instead (where pkl is pickle or dill etc). However, it can still work with this design, see below:

>>>classThing(object):...defpickle_myself(self, pkl_file_path):...withopen(pkl_file_path, 'w+') as f:...      pkl.dump(self, f, protocol=2)...>>>import dill as pkl>>>>>>t = Thing()>>>t.pickle_myself('foo.pkl')

Then restarting...

Python 2.7.10 (default, Sep  22015, 17:36:25) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type"help", "copyright", "credits"or"license"for more information.
>>> import dill
>>> f = open('foo.pkl', 'r')
>>> t = dill.load(f)
>>> t
<__main__.Thing object at 0x1060ff410>

If you have a much more complicated class, which I'm sure you do, then you are likely to run into trouble, especially if that class uses another file that is sitting in the same directory.

>>>import dill>>>from bar import Zap>>>print dill.source.getsource(Zap)
class Zap(object):
    x = 1
    def __init__(self, y):
        self.y = y

>>>>>>classThing2(Zap):   ...defpickle_myself(self, pkl_file_path):...withopen(pkl_file_path, 'w+') as f:...      dill.dump(self, f, protocol=2)...>>>t = Thing2(2)>>>t.pickle_myself('foo2.pkl')

Then restarting…

Python 2.7.10 (default, Sep  22015, 17:36:25) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type"help", "copyright", "credits"or"license"for more information.
>>> import dill
>>> f = open('foo2.pkl', 'r')
>>> t = dill.load(f)
>>> t
<__main__.Thing2 object at 0x10eca8090>
>>> t.y
2>>> 

Well… shoot, that works too. You'll have to post your code, so we can see what pattern you are using that dill (and pickle) fails for. I know having one module import another that is not "installed" (i.e. in some local directory) and expecting the serialization to "just work" doesn't for all cases.

See dill issues: https://github.com/uqfoundation/dill/issues/128https://github.com/uqfoundation/dill/issues/129 and this SO question: Why dill dumps external classes by reference, no matter what? for some examples of failure and potential workarounds.

EDIT with regard to updated question:

I don't see your issue. Running from the command line, importing from the interpreter (import test_serialization), and running the script in the interpreter (as below, and indicated in your steps 3-5) all work. That leads me to think you might be using an older version of dill?

>>>import os>>>import make_persist_load>>>>>>MAKE_AND_PERSIST = False#True>>>LOAD = (not MAKE_AND_PERSIST)>>>>>>cwd = os.getcwd()>>>store_path = os.path.join(cwd, "test_stored_env.pkl")>>>>>>if MAKE_AND_PERSIST == True:...    make_persist_load.make_env_and_persist()...>>>if LOAD == True:...    loaded_env = make_persist_load.load_env(store_path)...>>>

EDIT based on discussion in comments:

Looks like it's probably an issue with Windows, as that seems to be the only OS the error appears.

EDIT after some work (see: https://github.com/uqfoundation/dill/issues/140):

Using this minimal example, I can reproduce the same error on Windows, while on MacOSX it still works…

# test.pyclassEnvironment():
    def__init__(self):
        pass

and

# doit.pyimport test
import dill

env = test.Environment()
path = "test.pkl"withopen(path, 'w+') as f:
    dill.dump(env, f)

withopen(path, 'rb') as _f:
    _env = dill.load(_f)
    print _env

However, if you use open(path, 'r') as _f, it works on both Windows and MacOSX. So it looks like the __import__ on Windows is more sensitive to file type than on non-Windows systems. Still, throwing an ImportError is weird… but this one small change should make it work.

Solution 2:

In case someone is having same problem, I had the same problem running Python 2.7 and the problem was the pickle file created on windows while I am running Linux, what I had to do is running dos2unix which has to be downloaded first using

sudo yum install dos2unix

And then you need to convert the pickle file example

dos2unix data.p