Python lzma unable to load joblib

Question

I have a scikit learn pipeline that I serialize using:

with lzma.open('outputs/baseModel_LR.joblib',"wb") as f:
    dill.dump(pipeline, f)

When I try to open the file and load the pipeline using:

with lzma.open('outputs/baseModel_LR.joblib',"rb") as f:
    model = dill.load(f)

it gives error:

---------------------------------------------------------------------------
EOFError                                  Traceback (most recent call last)
somePath/notebooks/test.ipynb Cell 5 in <cell line: 1>()
      1 with lzma.open('outputs/baseModel_LR.joblib',"rb") as f:
----> 2         model = dill.load(f)
      3 model

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dill/_dill.py:373, in load(file, ignore, **kwds)
    367 def load(file, ignore=None, **kwds):
    368     """
    369     Unpickle an object from a file.
    370 
    371     See :func:`loads` for keyword arguments.
    372     """
--> 373     return Unpickler(file, ignore=ignore, **kwds).load()

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dill/_dill.py:646, in Unpickler.load(self)
    645 def load(self): #NOTE: if settings change, need to update attributes
--> 646     obj = StockUnpickler.load(self)
    647     if type(obj).__module__ == getattr(_main_module, '__name__', '__main__'):
    648         if not self._ignore:
    649             # point obj class to main

File /anaconda/envs/azureml_py38/lib/python3.8/lzma.py:200, in LZMAFile.read(self, size)
    194 """Read up to size uncompressed bytes from the file.
...
    100                        "end-of-stream marker was reached")
    101 else:
    102     rawblock = b""

**EOFError: Compressed file ended before the end-of-stream marker was reached**

Has anyone faced this problem and solved it? I use lzma because otherwise the joblib size is 27GB and with lzma its just 20MB