Chapter 11

Compressed Files

A zip file is a single file that holds many others, usually squeezed smaller. You meet them constantly: a download, an email attachment, an export from some web service. Python reads and writes them with the zipfile module, and once it's open the pattern is close to a regular file.

Opening a zip

Here's building one so we have something to work with, then reading it back:

import zipfile
from pathlib import Path

with zipfile.ZipFile("notes.zip", "w") as z:
    for note in sorted(Path("notes").glob("*.txt")):
        z.write(note)

with zipfile.ZipFile("notes.zip") as z:
    for name in z.namelist():
        print(name)

notes/klara-and-the-sun.txt
notes/one-hundred-years-of-solitude.txt
notes/piranesi.txt
notes/project-hail-mary.txt

ZipFile(path) opens for reading, ZipFile(path, "w") for writing. .namelist() gives you the names of the archive members, the files stored inside.

z.write(note) stored each file under the path you passed it. Path("notes").glob yields notes/piranesi.txt, so that's the name inside the zip, folder and all. Pass arcname= to store it under a shorter name.

Reading a member without unpacking

You don't have to extract a zip to disk to use what's in it. .read(name) hands you one member's bytes:

import zipfile

with zipfile.ZipFile("notes.zip") as z:
    data = z.read("notes/piranesi.txt")

print(data.decode("utf-8").splitlines()[0])

Piranesi

.read returns bytes, from Chapter 10, so decode it if the member is text. There's also .open(name), which gives you a file object you can stream through in chunks, for a member too big to hold in memory.

Extracting

import zipfile
from pathlib import Path

with zipfile.ZipFile("notes.zip") as z:
    z.extractall("unpacked")

names = sorted(p.name for p in Path("unpacked/notes").glob("*.txt"))
print(names)

['klara-and-the-sun.txt', 'one-hundred-years-of-solitude.txt', 'piranesi.txt', 'project-hail-mary.txt']

.extractall(dest) writes every member under dest, recreating any folders in the member names. .extract(name, dest) pulls a single one.

One safety note. A hostile zip can name a member something like ../../secrets to write outside the folder you asked for, an attack called "zip slip." Python 3.11.4 and later block that inside extractall, so on a current version you're covered. If you extract members one at a time, or support older Python, check each name yourself: reject anything absolute or containing ...

Writing a compressed zip

By default zipfile stores files without shrinking them. Ask for compression:

import zipfile
from pathlib import Path

content = Path("notes/one-hundred-years-of-solitude.txt").read_bytes()

sizes = {}
for mode, label in [(zipfile.ZIP_STORED, "stored"),
                    (zipfile.ZIP_DEFLATED, "deflated")]:
    with zipfile.ZipFile(f"{label}.zip", "w", compression=mode) as z:
        z.writestr("note.txt", content)
    sizes[label] = Path(f"{label}.zip").stat().st_size

print("deflated is smaller:", sizes["deflated"] < sizes["stored"])

deflated is smaller: True

z.write(path) adds a file from disk. z.writestr(name, data) adds content you already have in memory, with no file involved. ZIP_DEFLATED is the compression everyone means when they say "zip."

tarfile and shutil.make_archive

tarfile is the same idea for .tar, .tar.gz, and .tar.bz2, the archive formats you see more often on Linux. tarfile.open("x.tar.gz") reads one, tarfile.open("x.tar.gz", "w:gz") writes a compressed one. When you extract a tar from anywhere you don't fully trust, pass filter="data" to block its version of the zip-slip trick.

For the everyday case, "make a zip of this folder," shutil has a one-liner:

import shutil
from pathlib import Path

shutil.make_archive("notes-backup", "zip", "notes")
print(Path("notes-backup.zip").exists())

True

make_archive(base_name, format, root_dir) zips root_dir into base_name.zip, with the members stored relative to root_dir. The formats include "zip", "tar", and "gztar".

Common mistakes

Forgetting that a zip stores paths. z.write("notes/x.txt") stores the member as notes/x.txt, and extractall rebuilds that folder. Use arcname= if you want a flat archive.

Treating z.read output as text. It's bytes. Decode it.

Opening a .tar.gz with zipfile. Wrong module. zipfile for .zip, tarfile for anything .tar. If you're not sure what you have, check the magic number from Chapter 10.

Extracting an archive from anywhere without thinking about where the members land.

Practice

Try each of these before you read the solution under it.

1. Write zip_folder(folder, zip_path) that puts every file under folder into a zip, each stored under its path relative to folder (no leading folder name).
2. Write read_from_zip(zip_path, member) that returns a member's text, or None if that member isn't in the archive.
3. Write zip_report(zip_path) that prints each member's name and its uncompressed size.

Solutions

1. rglob finds the files; relative_to strips the folder prefix for arcname.

import zipfile
from pathlib import Path

def zip_folder(folder, zip_path):
    folder = Path(folder)
    with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as z:
        for file in sorted(folder.rglob("*")):
            if file.is_file():
                z.write(file, arcname=file.relative_to(folder))

zip_folder("notes", "flat.zip")
with zipfile.ZipFile("flat.zip") as z:
    for name in z.namelist():
        print(name)

archive/the-left-hand-of-darkness.txt
klara-and-the-sun.txt
one-hundred-years-of-solitude.txt
piranesi.txt
project-hail-mary.txt

2. Check namelist() first so a missing member is None, not a KeyError.

import zipfile

def read_from_zip(zip_path, member):
    with zipfile.ZipFile(zip_path) as z:
        if member not in z.namelist():
            return None
        return z.read(member).decode("utf-8")

print(read_from_zip("flat.zip", "piranesi.txt").splitlines()[0])
print(read_from_zip("flat.zip", "dune.txt"))

Piranesi
None

3. .infolist() gives a ZipInfo per member, with .filename and .file_size.

import zipfile

def zip_report(zip_path):
    with zipfile.ZipFile(zip_path) as z:
        for info in z.infolist():
            print(f"{info.file_size:>5}  {info.filename}")

zip_report("flat.zip")

  307  archive/the-left-hand-of-darkness.txt
  351  klara-and-the-sun.txt
  421  one-hundred-years-of-solitude.txt
  363  piranesi.txt
  298  project-hail-mary.txt

Where this leaves you

You can open a zip, list and read its members without unpacking, extract it, create a compressed one, and reach for tarfile or shutil.make_archive when those fit better. Chapter 12 is the last one: putting the whole book together into a single tool that reads the log, writes summaries, tidies the notes, and backs everything up.