How to unzip large zip file

[raf1hh]

Hi, does anybody know of a lib or extension that can handle large zip files without running out of memory?

I tried ZipFile and I tried to backport ZipFile from 2.6 as it has improvements in handling large files but run into issue of missing basestring.

Thanks

Raf

[mikezs]

An optimised zip file handler could be created in Symbian C++ as an extension. This would have the size limit imposed by your phone, not by Python.

[raf1hh]

I looked at the zipfile module and think that the issue is that it attempts to read each individual file contained in the zip file into memory. Thus, when it encounters a file that is larger then avail. memory, it throws out of memory error.

In my opinion, the only correct way to unzip a file would be by using a stream and unzip the file in chunks. But I havent found a module hat would do this yet.

Raf

[raf1hh]

A "backported" ZipFile module from 2.6 worked after a minor edit. In case anybody runs into the same issue (it unzipped a 30MB file in ~30 seconds on N95-3).

Module is too large to post in this forum so email or msg me if you want it.

Raf

[eswenson1]

Hi Raf,

I'm interested in your backported-to-2.5 2.6 zipfile implementation. How might I get it from you? -- Eric

[raf1hh]

Send me a private message with your email I will email it to you.

[jethro.fn]

Can you create a diff against the unmodified source and post that here?

[raf1hh]

Here is a UNIX diff of original 2.6 zipfile.py and one that works with pys60:

655c655
<     def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=False):
---
>     def __init__(self, mfile, mode="r", compression=ZIP_STORED, allowZip64=False):
680c680
<         if isinstance(file, basestring):
---
>         if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):
682c682
<             self.filename = file
---
>             self.filename = mfile
685c685
<                 self.fp = open(file, modeDict[mode])
---
>                 self.fp = open(mfile, modeDict[mode])
689c689
<                     self.fp = open(file, modeDict[mode])
---
>                     self.fp = open(mfile, modeDict[mode])
695c695
<             self.filename = getattr(file, 'name', None)
---
>             self.filename = getattr(mfile, 'name', None)

Since I hate UNIX diff, here is the modified __init__ method of the ZipFile class that needs to be replaced in the 2.6 module:

    def __init__(self, mfile, mode="r", compression=ZIP_STORED, allowZip64=False):
        """Open the ZIP file with mode read "r", write "w" or append "a"."""
        if mode not in ("r", "w", "a"):
            raise RuntimeError('ZipFile() requires mode "r", "w", or "a"')

        if compression == ZIP_STORED:
            pass
        elif compression == ZIP_DEFLATED:
            if not zlib:
                raise RuntimeError,\
                      "Compression requires the (missing) zlib module"
        else:
            raise RuntimeError, "That compression method is not supported"

        self._allowZip64 = allowZip64
        self._didModify = False
        self.debug = 0  # Level of printing: 0 through 3
        self.NameToInfo = {}    # Find file info given name
        self.filelist = []      # List of ZipInfo instances for archive
        self.compression = compression  # Method of compression
        self.mode = key = mode.replace('b', '')[0]
        self.pwd = None
        self.comment = ''

        # Check if we were passed a file-like object
        if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):
            self._filePassed = 0
            self.filename = mfile
            modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'}
            try:
                self.fp = open(mfile, modeDict[mode])
            except IOError:
                if mode == 'a':
                    mode = key = 'w'
                    self.fp = open(mfile, modeDict[mode])
                else:
                    raise
        else:
            self._filePassed = 1
            self.fp = file
            self.filename = getattr(mfile, 'name', None)

        if key == 'r':
            self._GetContents()
        elif key == 'w':
            pass
        elif key == 'a':
            try:                        # See if file is a zip file
                self._RealGetContents()
                # seek to start of directory and overwrite
                self.fp.seek(self.start_dir, 0)
            except BadZipfile:          # file is not a zip file, just append
                self.fp.seek(0, 2)
        else:
            if not self._filePassed:
                self.fp.close()
                self.fp = None
            raise RuntimeError, 'Mode must be "r", "w" or "a"'

Hope it helps.

Raf

[jethro.fn]

Thanks!

One observation:

if (type(mfile) == type("")) or (type(mfile) == type(u"")) or (type(mfile) == type('')):

"" and '' are both non-Unicode strings, since Python allows either kind of quotes and makes no distinction between them. So this works equally well:

if type(mfile) == type("") or type(mfile) == type(u""):

[rahul.shalik.makode]

hi,

i am using compression and decompression on client side using zlib

on server side reverse process is doing in java

i am unable decompress data on client side

error code: -11502

KEZlibErrStream -11501
KEZlibErrData -11502

i am not getting where i am wrong

[croozeus]

hi,

on server side reverse process is doing in java

i am unable decompress data on client side

Check your server implementation. We can only help to create or extract a zip archive using zipfile in Python

[JOM]

i am using compression and decompression on client side using zlib
on server side reverse process is doing in java
i am unable decompress data on client side

error code: -11502

KEZlibErrStream -11501
KEZlibErrData -11502

1) Try to decompress on server side, where you did compression.
2) Try to decompress with 7-Zip

...in other words, verify that your compression is working correctly before debugging why client side cannot decompress

Cheers,

--jouni