How to import python pickle *.pkl?

Question

I find some ELMo-models on github.

I want to use the Simplified-Chinese version: http://pbmpb9h15.bkt.gdipper.com/zhs.model.tar.xz

But these files were saved as python pickles.

I want to import them as Mathematica List or Association.

Is there any extension to do this?

Answer 1

I found some tools which can translate python objects to wolfram expressions.

https://github.com/WolframResearch/WolframClientForPython

import pickle as pkl
import wolframclient.serializers as wxf
def pkl2wxf(path):
    file = open(path, 'rb')
    objs = []
    while True:
        try : objs.append(pkl.load(file))
        except EOFError : break
    file.close()
    print(objs)
    wxf.export(objs, path + '.wxf', target_format='wxf')

The function pkl2wxf can finish these works. Let's do some test:

f = open('objs.pkl', 'wb')
# Test basic types
testDict = {
    0: None,
    1: [1, 2, 3, 4],
    2: ('true', 'false'),
    3: {'yes': True, 'no': False}
}
pkl.dump(testDict, f)
f.close()
pkl2wxf('objs.pkl')

Now just use Import["objs.pkl.wxf"], all these python types are translated.

{<|
   0->None,
   1->{1,2,3,4},
   2->{"true","false"},
   3-><|"yes"->True,"no"->False|>
|>}

Answer 2

This pickle format is almost by definition, meant for Python and Python only. Your best bet is to load it in Python, and the export to some format that Mathematica can handle. For lists and dictionaries with only primitive datatypes in them, I'd use JSON. It is not clear that the data you linked is such.

If the data in this pickle is very simple, you could use the ExternalEvaluate to evaluate the Python data and transfer it to Mathematica. Automatic translation will only work for simple data types.

---

From the Python documentation page you linked:

The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.