# Archivey > Read, stream, and safely extract archives behind one uniform interface. Archivey is a Python library (`pip install archivey`, `import archivey`) that reads ZIP, TAR, 7z, RAR, ISO and single compressed files through one API: `archivey.open_archive(path)` returns a reader with `members()`, `get()`, `read()`, `open()`, `stream_members()` and `extract_all()`. It is not a wrapper around zipfile or tarfile, and their method names do not apply. Start with the cheat sheet; the API reference has every signature. # Start here # Archivey Archivey reads ZIP, TAR, 7z, RAR, ISO and single compressed files from Python, with one simple API. Untrusted archives extract safely by default. Archive formats hide some slow and error-prone ways of reading, and archivey's defaults steer around them whenever possible. The [cheat sheet](https://archivey-dev.github.io/archivey/cheat-sheet/index.md) has the main calls on one page, with the zipfile and tarfile equivalents. Coding agents can read these docs as Markdown, starting from [llms.txt](https://archivey-dev.github.io/archivey/llms.txt), or all in one file from [llms-full.txt](https://archivey-dev.github.io/archivey/llms-full.txt). ## Install Install the `archivey` package from PyPI with your usual tool (e.g. `uv add "archivey[recommended]"` or `pip install "archivey[recommended]"`). Without `recommended`, archivey has no dependencies and reads ZIP, TAR, 7z with its common compression methods, and the single compressed files the standard library handles, such as gzip, bzip2 and xz. The extra adds ISO, lz4, Brotli, zstd on Python 3.13 and older, the rarer compression methods in 7z and ZIP, and AES decryption, which most encrypted 7z and ZIP archives need. RAR archives can be listed, and files stored in them without compression read, with nothing else installed. To read compressed files, you also need RARLAB's `unrar` or `rar`, 6.0 or later, or `unar` 1.10 or later, which handles fewer RAR archives. [Install](https://archivey-dev.github.io/archivey/install/index.md) has the details. ## Open and list ```python import archivey with archivey.open_archive("photos.zip") as archive: for member in archive: print(member.name, member.size) ``` `open_archive` identifies the format from the file's contents, so the same code opens a `.7z` or a `.tar.gz`. Each entry in the archive, whether a file, a directory or a link, is a *member*, as in `zipfile` and `tarfile`. ArchiveMember lists everything a member carries. [Opening an archive](https://archivey-dev.github.io/archivey/opening/index.md) covers passwords, formats and name encodings. ## Read a member ```python with archivey.open_archive("photos.zip") as archive: data = archive.read("holiday/beach.jpg") ``` `read` returns the whole member as bytes. For a large member, `open` gives you a file object to read in pieces: ```python import shutil with archive.open("holiday/video.mp4") as stream, open("video.mp4", "wb") as out: shutil.copyfileobj(stream, out) ``` `archive.get(name)` gives you the member with that name, or `None`. `open` and `read` follow a link to another member of the archive and give you that member's contents. They never follow it outside the archive: a link whose target isn't a member, such as one pointing to `/etc/passwd` or to a file the archive doesn't hold, raises `LinkTargetNotFoundError`. A directory has no contents to read, so they raise `ArchiveyUsageError`. On some archives, reading a member means first decompressing the members stored before it, so reading them out of order gets slow. [Solid archives](https://archivey-dev.github.io/archivey/reading/index.md#solid-archives) explains when this happens. The next section shows how to avoid it by reading them all in one pass. ## Read everything in one pass ```python with archivey.open_archive("backup.tar.gz") as archive: for member, stream in archive.stream_members(): if not member.is_file or stream is None: continue # a directory or a link process(member.name, stream.read()) ``` `stream_members` goes through the archive in the order it is stored and gives you each member with a stream of its contents. It reads each member once, so it is never slower than reading them by name, and on some archives it is much faster. Directories and links come with `None` in place of a stream, and each stream works only until the loop moves on. A directory entry that hides data in the archive is the exception: it can come with a stream of those bytes, which is why the loop tests `member.is_file`. ## Extract ```python with archivey.open_archive("download.zip") as archive: archive.extract_all("out/") ``` `extract_all` writes every member under `out/`. By default it refuses anything that would land outside that folder, such as `../` paths, absolute paths or links pointing out of it, and it stops archives that expand to far more data than they hold. [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md) lists every protection and how to relax them for archives you trust, and also shows how to extract only some files. [Security](https://archivey-dev.github.io/archivey/security/index.md) covers what archivey relies on and where its guarantees stop. ## When something goes wrong Problems with the archive raise a subclass of `ArchiveyError`: `CorruptionError` for damaged data, `EncryptionError` for a missing or wrong password, `PackageNotInstalledError` when a format needs the extra. Archivey raises these the same way for every format, including where the underlying library would stop quietly and hand back short data. [Errors](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md) has the full list. If archivey reads, lists or extracts an archive wrongly, please [open a bug report](https://github.com/archivey-dev/archivey/issues/new/choose). # Cheat sheet The calls most code needs, what each returns and raises, and how they map from `zipfile` and `tarfile`. Each section links to the page with the details. ## The main calls ```python import archivey # Open any supported format. The format comes from the file's contents. with archivey.open_archive("photos.zip") as archive: names = [member.name for member in archive] # list, in archive order member = archive.get("notes.txt") # ArchiveMember, or None data = archive.read("holiday/beach.jpg") # bytes with archive.open("holiday/video.mp4") as stream: # binary file object first_kb = stream.read(1024) # Read every member in one pass, in archive order. with archivey.open_archive("backup.tar.gz") as archive: for member, stream in archive.stream_members(): if member.is_file: print(member.name, len(stream.read())) # Extract safely. with archivey.open_archive("download.zip") as archive: report = archive.extract_all("out/") # Decompress a single file, such as a .gz. with archivey.open_stream("notes.txt.gz") as stream: data = stream.read() # bytes # Identify the format without opening the archive. info = archivey.detect_format("backup.tar.gz") print(info.format) # ArchiveFormat.TAR_GZ ``` What each call returns, and the errors you're most likely to meet. The [next section](#exceptions) has all of them. - **`open_archive(source)`** returns an ArchiveReader, to use in a `with` block. `source` is a path, a binary file object or a folder. A missing file raises `FileNotFoundError`, and a file in no format archivey reads raises `FormatDetectionError`. A file object that can't seek, such as a pipe, raises `StreamNotSeekableError` unless you pass `streaming=True`, which reads TAR, ZIP and single compressed files from one. [Opening an archive](https://archivey-dev.github.io/archivey/opening/index.md) - **`for member in archive`** and **`archive.members()`** give ArchiveMember objects. A directory's name ends with `/` in every format. - **`archive.get(name)`** returns the member with that name, or `None`. - **`archive.read(name)`** returns the member's contents as `bytes`, and **`archive.open(name)`** a file object to read them in pieces. Both take a name or an `ArchiveMember`. A name the archive doesn't have raises `KeyError`, a directory raises `ArchiveyUsageError` unless the archive stored data in it, and a link whose target isn't in the archive raises `LinkTargetNotFoundError`. Only one stream from `open` can be open at a time, and it can't seek. [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md) has the options that change both. - **`archive.stream_members()`** yields `(member, stream)` pairs in archive order. The stream is `None` for links and most directories, and works only until the loop moves on. [Read everything in one pass](https://archivey-dev.github.io/archivey/index.md#read-everything-in-one-pass) - **`archive.extract_all(dest)`** returns an ExtractionReport with one result per member. A refused member, such as `../evil.txt`, is skipped and recorded in the report with the status `BLOCKED`, without raising. By default the first member that fails raises and ends the extraction: `ExtractionError` for a file already at the destination, `CorruptionError` for damaged data. [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md) - **`open_stream(source)`** returns a file object with the decompressed contents. On a `.tar.gz` you get the TAR's bytes. A ZIP, 7z, RAR, ISO or plain TAR raises `FormatDetectionError`. [Single compressed files](https://archivey-dev.github.io/archivey/opening/index.md#single-compressed-files) - **`detect_format(source)`** returns a FormatInfo with the `format` and how sure archivey is about it. A file in no format archivey reads raises `FormatDetectionError`. [Format](https://archivey-dev.github.io/archivey/opening/index.md#format) `open_archive(source, streaming=True)` reads the archive once, from start to end, which also works on a pipe. Its reader has no `members()`, `get()`, `read()` or `open()`: they raise `ArchiveyUsageError`. [Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once) ## Exceptions ```text ArchiveyError problems with the archive or what reading it needs ├── OpenError reading could not start │ ├── FormatDetectionError not a format archivey reads │ └── StreamNotSeekableError ├── ReadError │ ├── CorruptionError damaged data │ │ └── TruncatedError the data seems to end early │ ├── EncryptionError password missing or wrong │ └── LinkTargetNotFoundError ├── ExtractionError a member could not be written │ ├── FilterRejectionError a safety check refused a member │ ├── NameCollisionError │ └── NameRewrittenError ├── ResourceLimitError a limit was reached, such as an archive bomb ├── UnsupportedFeatureError a recognized format, with something archivey can't read ├── PackageNotInstalledError an optional package or program is missing └── DiagnosticRaisedError a diagnostic turned into an error by DiagnosticPolicy ArchiveyUsageError a mistake in the calling code, not an ArchiveyError ``` An argument with a type or value the call can't use raises `ArchiveyUsageError` that is also a `TypeError` or a `ValueError`. Errors that aren't about the archive pass through as themselves: `FileNotFoundError`, `OSError` from the disk, and `KeyError` for a missing name. [Errors and diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md) ```python try: with archivey.open_archive("download.zip") as archive: archive.extract_all("checked/") except archivey.EncryptionError: print("Wrong or missing password") except archivey.ArchiveyError as e: print("Could not extract:", e) ``` ## Safe defaults ```python with archivey.open_archive("download.zip") as archive: archive.extract_all( "defaults/", policy="strict", # portable names; no setuid bits or group and other write overwrite="error", # an existing file fails the member on_error="stop", # the first failed member raises limits=archivey.ExtractionLimits( max_extracted_bytes=2 * 2**30, # 2 GiB written in total max_ratio=1000.0, # how much a member, or the archive, may expand max_entries=262_144, # files, folders and links created ), ) ``` These are the defaults. Under every policy, `extract_all` refuses `../` paths, links pointing outside the destination, and device files. Going over a limit raises `ResourceLimitError`. [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md#options) lists every policy, and [Archives you trust](https://archivey-dev.github.io/archivey/extracting/index.md#archives-you-trust) shows how to relax them. Reading has limits too, with the same `ResourceLimitError`: a listing stops at 262,144 members, and an archive can ask a decoder for at most 2 GiB of memory. [Hardening](https://archivey-dev.github.io/archivey/security/index.md#hardening) lists them, and `config=archivey.ArchiveyConfig(...)` changes them. ## From zipfile and tarfile Each item gives the archivey call first, then the zipfile and tarfile calls it replaces. - `open_archive(p)`: `ZipFile(p)`, `tarfile.open(p)`. - `[m.name for m in archive]`: `zf.namelist()`, `tf.getnames()`. - `archive.members()`: `zf.infolist()`, `tf.getmembers()`. - `archive.get(n)`, which returns `None` for a missing name: `zf.getinfo(n)`, `tf.getmember(n)`. - `archive.read(n)`: `zf.read(n)`, `tf.extractfile(n).read()`. - `archive.open(n)`: `zf.open(n)`, `tf.extractfile(n)`. - `archive.stream_members()`: a loop over `zf.open(info)`, or `for info in tf:` with `tf.extractfile(info)`. - `archive.extract_all(d)`: `zf.extractall(d)`, `tf.extractall(d)`. - `open_archive(p, password=pw)`: `zf.setpassword(pw)`. - `detect_format(p)`: `is_zipfile(p)`, `is_tarfile(p)`. - `CorruptionError` or `FormatDetectionError`: `BadZipFile`, `tarfile.ReadError`. - `member.name`: `info.filename`, `info.name`. - `member.size`: `info.file_size`, `info.size`. - `member.is_dir`: `info.is_dir()`, `info.isdir()`. - `member.modified`, a `datetime`: `info.date_time`, `info.mtime`. For a single compressed file, `open_stream(p)` replaces `gzip.open(p)`, `bz2.open(p)` and `lzma.open(p)`. [Migrating from other libraries](https://archivey-dev.github.io/archivey/migrating/index.md) covers py7zr and rarfile, and what behaves differently. # Install Archivey is on PyPI as `archivey`. It needs Python 3.11 or later and has no dependencies of its own. Optional packages, grouped as extras, add the rest: ```bash pip install archivey # no dependencies pip install "archivey[recommended]" # every format and compression method that installs everywhere pip install "archivey[seekable]" # faster seeking and reading in large gzip, bzip2 and DEFLATE data pip install "archivey[all]" # recommended and seekable together ``` If you only want the `archivey` command, `uvx archivey-cli` runs it with everything `[recommended]` adds, without installing anything for good, and `pipx install archivey-cli` installs it. `archivey-cli` is a separate package that holds no code: it depends on `archivey[recommended]` at the same version. `uvx archivey` works too, but it runs the base package only, because an extra must be named to be installed. If you use a free-threaded Python build, [Free-threaded Python](#free-threaded-python) has the extra to use instead. There's no extra per format, because ZIP, 7z and TAR share their compression methods. | You install | It reads | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `archivey` | ZIP, TAR and 7z with their common compression methods, and directories. gzip, bzip2, xz, LZMA, zlib, lzip and `.Z` files, alone or inside a TAR, and Zstandard too on Python 3.14 and later. RAR archives can be listed, and their uncompressed, unencrypted files read. | | `[recommended]` adds | ISO images, and LZ4, Brotli and Zstandard files (Zstandard only before Python 3.14). PPMd, Deflate64 and Zstandard inside 7z and ZIP, and Brotli and LZ4 inside 7z. AES decryption for 7z, RAR and WinZip AES ZIP files. Progress bars in the command-line tool. | | `[seekable]` adds | No new format. Faster seeking inside large gzip, zlib and bzip2 data, and large DEFLATE members in ZIP and 7z, when you open with `seekable_members=True`. On Linux and macOS, also faster reading of large DEFLATE data that compresses 3:1 or less, without that option. | Compressed RAR files, and a few encrypted ones, also need an outside program, which no extra can install. The next section covers it. ## RAR needs an outside program RAR is a proprietary format, and there's no self-contained Python decoder for it, so archivey reads compressed RAR members with an outside program. It lists RAR archives and reads stored (uncompressed) members itself, encrypted ones too when `cryptography` is installed and the member is in one part. The preferred program is RARLAB's `unrar` (or `rar`), version 6.0 or later. Archivey also supports `unar` 1.10 or later, from The Unarchiver, but it has some limitations. Archivey uses `unar` when it finds no usable `unrar`, and then passes an encrypted archive's password on `unar`'s command line, where other users on the same machine can see it while it runs. Some `unar` packages in Debian and Ubuntu have a bug that drops some files; archivey tests the `unar` it finds and won't use one that has it. If you want to choose the program yourself, set `rar_decompressor` in `ArchiveyConfig` to `"unrar"`, `"unar"`, `"none"` or `"auto"` (the default). With `"unrar"`, archivey never runs `unar`. ### Getting `unrar` On Debian and Ubuntu, `sudo apt install unrar` installs RARLAB's program. It's in the non-free (Debian) or multiverse (Ubuntu) section, and the `unrar-free` package is a different program that archivey doesn't use. Other Linux distributions have a package of RARLAB's UnRAR under a similar name. On Windows, download the command-line UnRAR from [RARLAB](https://www.rarlab.com/rar_add.htm) and put `UnRAR.exe` on your `PATH`. On macOS, Homebrew no longer ships `unrar`. You can install it from an unofficial Homebrew tap, which builds or downloads RARLAB's program: ```bash brew install gromgit/new-life/unrar ``` If you'd rather not trust that tap, download the UnRAR source from [RARLAB](https://www.rarlab.com/rar_add.htm) and build it with Xcode's command-line tools: ```bash tar xf unrarsrc-*.tar.gz cd unrar make cp unrar ~/.local/bin/ # or any folder on your PATH ``` Running `unrar` with no arguments prints its version, so you can check it's 6.0 or later. ### Getting `unar` On macOS, `brew install unar` installs a version archivey accepts. On Debian and Ubuntu, the packaged `unar` may have the bug described above; if archivey won't use it, either install `unrar` or build `unar` 1.10.8 from [The Unarchiver's source](https://github.com/MacPaw/XADMaster). ## Checking what an install can read If your program needs to know at run time which formats work with what's installed, [`format_availability()`](https://archivey-dev.github.io/archivey/api/index.md#archivey.format_availability) says whether a format is fully supported, partly supported (it opens, but some members need a missing package) or not available, and which packages would add the rest. RAR counts as partly supported without RARLAB's `unrar`, even when `unar` is installed. To find out, it runs each `unrar` or `rar` on your `PATH` once to read its version, whatever `rar_decompressor` says; pass `check_programs=False` to run nothing. `cryptography` isn't counted, because only encrypted members need it. ## Python versions and platforms Archivey needs Python 3.11 or later. Every change is tested on Linux with CPython 3.11 to 3.14 and PyPy 3.11, and on macOS and Windows with CPython 3.11 and 3.14. Other platforms aren't tested, and PyPy has some limits ([PyPy](#pypy) lists them). [How it is built](https://archivey-dev.github.io/archivey/how-it-is-built/index.md) has the full test setup. ## Free-threaded Python On a free-threaded build (3.13t and later), importing a compiled package that doesn't declare free-threading support turns the GIL back on for the rest of the process. If you use `pip install "archivey[free-threaded]"`, you get only the optional packages that keep it off: ISO, LZ4 and Zstandard support, and CLI progress bars, plus AES decryption from 3.14t. On 3.13t, `[recommended]` doesn't install at all, because `cryptography` can't be installed there yet, and neither does `archivey-cli`, which depends on it. Use `[free-threaded]` there instead. Other optional packages can still be installed. Archivey imports each one only when a file needs it, so the GIL comes back the first time you read PPMd, Deflate64 or Brotli data. Free-threaded builds are tested on Linux only. ## PyPy archivey works on PyPy 3.11, with the limits below. Every change to the main branch runs the whole test suite on PyPy 3.11, with and without the optional packages, and it has to pass there. Under PyPy, archivey doesn't use rapidgzip, lz4, pyppmd or inflate64, even when they are installed, because there they misread data or crash the interpreter. It reads as if they weren't installed: - LZ4 files, including `.tar.lz4`, and LZ4 members of 7z archives can't be read. - PPMd members of ZIP and 7z archives can't be read. - Deflate64 members of ZIP and 7z archives can't be read. - The accelerator behind `[seekable]` is off, so seeking backwards in gzip, zlib or bzip2 data, or in a DEFLATE member of a ZIP or 7z archive, decompresses more of it again. If you read data that needs lz4, pyppmd or inflate64, the error says why archivey doesn't use the package, in place of an install hint, and so do `format_availability()` and `archivey --version -v`. rapidgzip only speeds things up, so neither lists it, and you see its reason only if you ask for it outright by setting `use_rapidgzip` or `use_indexed_bzip2` to `AcceleratorMode.ON`. If you use PyPy, close member streams and passes yourself, with a `with` block or `close()`. On CPython, one that nothing refers to any more is closed at once. PyPy closes it only at a later garbage collection, and until then it still holds the archive. When a call finds the archive held, archivey runs a garbage collection and tries once more, so a stream or pass you dropped doesn't block it. One you still refer to does: unless you opened the archive with `concurrent_operations=True`, the next `open` raises `ArchiveyUsageError` because a member stream is already open, and an unfinished `stream_members` pass makes the archive's other calls raise. A RAR member read through `unrar` also keeps its `unrar` process running until the collection, and so does a decoder's child process where one is in use ([Decoders in a child process](https://archivey-dev.github.io/archivey/security/index.md#decoders-in-a-child-process)). Closing an XZ or LZMA stream also frees its decoder's dictionary at once. Dropped unclosed, the dictionary, as large as the data declares, stays allocated until that collection. [Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once) shows the `with` form for a pass. # Using archivey # Opening an archive ## What you can open `open_archive` takes a path, a binary file object or a folder. A folder opens like an archive of the files inside it. For a 7z or RAR archive split into several volumes, the path of any one of them is enough, and archivey finds the others. If it can't find them by name, such as volumes you hold as open file objects, you can pass a list of them, in order. A ZIP split by 7-Zip into `.zip.001`, `.zip.002` and so on opens the same way. If a part of a numbered set like that (`.7z.001`, `.zip.001`) is missing, opening raises `TruncatedError` naming it, rather than reading a shorter archive. A RAR set with a volume missing still opens, as `unrar` would, but `members()` raises `TruncatedError` naming the missing volume, and so does reading a member with data in it. A file object is read from its current position to its end, so an archive that starts partway through a larger file opens as is. Bytes after the archive don't stop it opening. Any that aren't zeros are recorded as `ARCHIVE_TRAILING_DATA`, which `DiagnosticPolicy.STRICT` turns into an exception. Behind more than 64 KiB of trailing bytes, zeros included, a ZIP's index is found by walking its members from the start, which takes longer. A file object that can't seek, such as a pipe or an HTTP response, needs `streaming=True` and works only for some formats, which [Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once) lists. Why not refuse bytes after the archive? They aren't part of any member, so every member still reads correctly. GNU gzip and bzip2 also decompress such a file and only warn, though `xz` refuses it. ## Format Archivey identifies the format from the file's contents, so a ZIP named `backup.bin` still opens as a ZIP. It falls back to the file name only when the contents don't settle it. To skip detection, pass `format=`, such as `format="zip"` or `format=archivey.ArchiveFormat.TAR_GZ`. If the file isn't in the specified format, it fails like a damaged one. `archivey.detect_format(path)` tells you the format without opening the archive. It returns a `FormatInfo` with the format and how sure archivey is about it: `CERTAIN` when the format's signature matched where the format puts it, `PROBABLE` when a test of its contents matched, and `GUESS` when the evidence is weak, such as only the file name. `payload_offset` is more than 0 for a self-extracting archive, where the archive starts after a program. A file in no format archivey knows raises `FormatDetectionError`. The confidence grade is for a person reading it and may change between releases, so decide what to do from the format, not the grade. ## Single compressed files A file that's compressed but not an archive, such as `notes.txt.gz`, opens as an archive of one member, named after the file without its extension (`notes.txt`). If you only want its contents, `open_stream` returns them as a file object: ```python with archivey.open_stream("notes.txt.gz") as stream: text = stream.read() ``` On a `.tar.gz` it removes only the compression, so you get the TAR's bytes. It refuses ZIP, 7z, RAR, ISO and plain TAR files, which aren't compressed streams. It takes `seekable=`, which works like `seekable_members` in [Seeking inside a member](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member). zlib, Brotli and `.lzma` files start with nothing that identifies them. By default `open_archive` recognizes them only by their extension (`.zz`, `.br` or `.lzma`), and a file without one raises `FormatDetectionError`. To open one anyway, pass `format=`, or set `always_probe_content=True` in the config, which makes `open_archive` try decoding the start of the file. `open_stream` always tries. Why doesn't `open_archive` try decoding? These formats are rarely saved as files of their own. They usually come inside something else, such as an HTTP response compressed with Brotli or zlib, and that's what `open_stream` is for. `open_archive` is mostly handed files, of any kind. For these formats, decoding the start and seeing whether it works is the only test, and some files that aren't compressed at all pass it, such as encrypted or media files. `open_archive` would then open one of them as a compressed file and return garbage or a confusing error. On the streams `open_stream` gets, the test is worth that risk. ## Passwords Pass the password as `password=`: one string, a list of candidates when you're not sure which one the archive uses, or a function that supplies passwords as they're needed. ```python archivey.open_archive("secret.7z", password="hunter2") archivey.open_archive("secret.zip", password=["likely", "fallback"]) ``` Archivey tries a list in order, and each wrong password costs some hashing work before it's rejected, so the most likely one is best placed first. An archive can use different passwords for different members, and archivey finds the right one from your list for each. How does archivey know which password is right? The archive can't simply say. A ZIP stores a check value of one byte, or two with AES, so about one wrong password in 256 (or in 65,536) passes it, and a 7z stores none. Archivey decrypts the start of the member and decompresses it: a wrong password gives random bytes, which most compression methods reject within a few hundred bytes. When more than one password gets through that, it reads the member to its end and checks its checksum. You can also pass a function, such as one that asks the user. Archivey calls it only when a password is needed and none tried so far works. It passes a `PasswordRequest` with the member being opened and the attempt number. When the password is needed to read an encrypted members list, which 7z and RAR support, `PasswordRequest.member` is `None`. Return a password to try, or `None` to give up, which raises `EncryptionError`. Archivey hands a RAR password to `unrar` only when the read needs one, and `unrar` would cut a password at a line break or a NUL character. So when the password that reaches `unrar` contains either, archivey raises `UnsupportedFeatureError` for the archive. It doesn't move on to the next password in your list, because the one handed over is the first candidate (or the one already found to be right). A password read from a file usually ends with a newline: remove it yourself, for example with `password.removesuffix("\n")`. The [RAR section](https://archivey-dev.github.io/archivey/formats/index.md#rar) has the details. A password or list of passwords given for a format with no encryption, such as TAR, is ignored and noted in the [diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md), which log a warning for each such archive by default. One list can then serve a whole batch of archives, and the diagnostics page shows how to quiet the warning. ## Names in other encodings Archives made by current tools either say which encoding their names use or store them as UTF-8, and archivey decodes them that way. It has to guess only for older formats, or archives made by older tools. Then it uses UTF-8 if the bytes are valid UTF-8, and otherwise the format's traditional encoding, such as cp437 for ZIP. Where there isn't one, it keeps each byte it can't decode as an escape, the way Python's `surrogateescape` does. An escape is a placeholder character between U+DC80 and U+DCFF. If you know the encoding, pass it as `encoding=`, and archivey uses it in place of that default for names that aren't valid UTF-8: a Latin-1 `café.txt` in a TAR lists as `'caf\udce9.txt'` without it and `'café.txt'` with `encoding="latin-1"`. Only ZIP, TAR, ISO and RAR read `encoding=`. The other formats decode names their own way, and a value passed to them is ignored and noted in the diagnostics. Why does valid UTF-8 win over `encoding=`? Text in another encoding almost never happens to be valid UTF-8. `café`, `Müller` and `Größe`, written in Latin-1, all fail to decode as UTF-8. So a name that decodes as UTF-8 almost certainly is UTF-8, even in an archive whose other names use your encoding. `member.name` is the decoded name, and `member.raw_name` is the bytes as stored. A name with an escaped byte, like `'caf\udce9.txt'`, can't be encoded as UTF-8, so printing or logging it can raise `UnicodeEncodeError`. `archivey.terminal.escape_control_chars(member.name)` gives a version that's safe to show. ## Members with the same name An archive can hold several members with the same name. Some tools update an archive by adding the new version of a file at its end and leaving the old one in place, as `tar --append` does, and RAR and ISO can keep old versions on purpose. All of them are listed. The last one is the current version: its `member.is_current` is `True`, and the earlier ones have `False`. `get`, `open` and `read` with a name use the current version, since that's the file the archive's own tool would give you. `extract_all` writes only the current version and reports the others as `ExtractionStatus.SUPERSEDED`. A name passed in `members=`, to `extract_all` or `stream_members`, matches every member with that name, so `stream_members(members=["a.txt"])` gives you each version. To pick one, pass the `ArchiveMember` from the listing. A `stream_members` loop can give you a member before archivey has seen the rest of the archive, so it can't yet tell whether a later copy replaces it. Each member then has `is_current` set to `True` until the loop ends, and the members you kept are updated then. `member.member_state_final` tells you which case you have: it's `False` while a member's fields can still change. ## Configuration Settings you'd keep the same across many archives live in `archivey.ArchiveyConfig`, passed as `config=`. These include the [limits](https://archivey-dev.github.io/archivey/security/index.md#hardening), which RAR program to use, the [accelerators](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member), and how [diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md) are reported. Without it, the defaults apply. The reference lists every setting. `ArchiveyConfig.formats` takes a FormatPolicy, which decides which formats a call opens. A format is a container and its compression, and both have to be allowed: ```python from archivey import ArchiveyConfig, ContainerFormat as C, FormatPolicy, StreamFormat as S # ZIP, and TAR with or without gzip or xz. Nothing else opens. UPLOADS = ArchiveyConfig(formats=FormatPolicy(only={C.ZIP, C.TAR, S.GZIP, S.XZ})) # Everything except RAR, and no plugin formats. NO_RAR = ArchiveyConfig(formats=FormatPolicy(exclude={C.RAR}, plugins=False)) ``` A file in a format the policy turns off raises `UnsupportedFeatureError`, and the message says which entry would let it open. `format=` doesn't get around it. `detect_format` still reports what a file is, and `config.formats.allows(info.format)` tells you whether it would open. A plugin format that's turned off isn't detected at all, so its code doesn't run. # Choosing how to read With no options, `open_archive` lets you read members in any order, with one member open at a time. That suits most programs. Three options change it for the cases where the default is slow or not enough: `streaming`, `seekable_members` and `concurrent_operations`. Opening an archive also applies a few limits, such as how many members it may list, which [Archives you trust](https://archivey-dev.github.io/archivey/extracting/index.md#archives-you-trust) explains how to raise. ## Solid archives A ZIP compresses each member on its own, so archivey can jump straight to any of them. A `.tar.gz` compresses the whole archive as one stream, so reaching a member in the middle means decompressing everything before it. An archive whose members are compressed together like this is called *solid*. Every compressed TAR is solid, and so are most 7z archives, since 7-Zip creates solid archives by default. RAR archives are solid only when created with that option. On a solid archive, reading members by name can decompress the same data again and again. `stream_members` decompresses it once. archivey tells you when this happens: on a solid 7z or RAR archive, the first open that decompresses data an earlier open already decompressed logs one INFO message on the `archivey.backends` logger. Reading the same member twice counts only the data before it in the archive. The message shows only if your application turns logging on. If your code needs to know whether an archive is solid, `archive.cost` tells you before you read anything. Its `access_cost` is `AccessCost.SOLID` when reaching a member means decompressing others first, and `AccessCost.DIRECT` when a member can be reached directly. Its `listing_cost` says what listing the members takes. It's `ListingCost.INDEXED` when the archive keeps a list of its members, as ZIP and 7z do when read from a file. It's `ListingCost.REQUIRES_SCANNING` when listing requires jumping from one member's header to the next, as in a plain `.tar`, a RAR, or a ZIP read from a pipe. It's `ListingCost.REQUIRES_DECOMPRESSION` when the headers are inside compressed data, as in a `.tar.gz`. Its `stream_capability` is `StreamCapability.SEEKABLE` when the source you opened can seek, and `StreamCapability.FORWARD_ONLY` when it can only be read forward, as a pipe. `archive.cost` is worked out once, when the archive opens. It describes what each kind of read costs, but it never allows or refuses one: the options below do that. ## Reading once ```python import sys with archivey.open_archive(sys.stdin.buffer, streaming=True) as archive: for member, stream in archive.stream_members(): ... ``` `streaming=True` is a promise to read the archive once, from start to end. When set, `open`, `read` and `members()` raise, so an out-of-order read fails right away instead of quietly costing time. If you plan to read an archive in one pass, setting it catches that mistake, even on an ordinary file. You get one pass, through `stream_members`, `for member in archive` or `extract_all`, and it is used up even if you leave the loop early. Inside a `stream_members` loop, `open`, `read` and `members()` raise even without `streaming=True`, since the pass is still using the archive. A pass you leave with `break` ends at once on CPython. One kept in a variable, or one an exception left, holds the archive until you close it, so use it in a `with` block when the loop can stop early: `with archive.stream_members() as members:` ends the pass at the end of the block, and `with iter(archive) as members:` does the same for a loop over members. A pass or member stream that nothing refers to any more never blocks the next call, on any Python: when a call finds the archive held, archivey runs a garbage collection and tries once more before it raises. With [`concurrent_operations=True`](#several-members-at-once) they work, from any thread, on every format, and so does `extract_all`. Streams from the pass never seek. A pipe, a socket or an HTTP response can only be read this way. TAR and ZIP archives and single compressed files can come from one. 7z, RAR and ISO archives need a file or another source that can seek. A ZIP keeps the full details of its members, such as permissions, symlinks and comments, in an index at its end. From a pipe, archivey finds each member by the short header in front of it, and gives you the member with what that header says: no permissions, and a symlink looks like a file. When the pass reaches the index, archivey fills in the rest on the members you already have. Until then `member.member_state_final` is `False`. From a pipe, the pass reaches a ZIP's index only when the sender closes the pipe, so the loop doesn't finish while the sender keeps it open. A compressed file or a compressed tar from a pipe that the sender keeps open is decompressed from what has arrived. A short one can still wait at open, and a read past its last byte waits for the sender to close the pipe ([Single-file compressors](https://archivey-dev.github.io/archivey/formats/index.md#single-file-compressors) says where). Why can't the others come from a pipe? 7z keeps its index at the end of the file, and reading an ISO means jumping between places in it. RAR stores a header before each member, as ZIP does, so archivey could read it from a pipe, copying what it reads to a temporary file for `unrar` as it goes. It doesn't yet. If you have a 7z, RAR or ISO archive in a pipe, copying it to a file or to an `io.BytesIO` first lets archivey read it. ## Seeking inside a member Some code needs to seek inside a member: a library that reads a ZIP stored inside the archive, a Parquet reader, an image decoder. By default a member stream only moves forward, and `seek` raises. With `seekable_members=True`, streams from `open` can seek, and an archive stored inside another one opens from such a stream like from a file. Moving backwards in compressed data can mean decompressing the member again from its start. If you install the `seekable` extra, archivey reads bzip2 data, and large DEFLATE data, through an [accelerator](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop) that marks places it can restart from as it reads, and every seek resumes from the nearest marked place before its target. DEFLATE is the compression in gzip files and in most ZIP members. Without the extra, a gzip file made of several members, such as one `bgzip` writes, resumes from the nearest member start. A zstd file made of several frames, such as one in the Seekable Zstandard format, always resumes from the nearest frame start. For a `bgzip` file or a Seekable Zstandard file, archivey finds those starts without decompressing anything. A forward seek keeps the checksum check when reaching the target means decoding the bytes in between, which is the usual case in compressed data. A seek that can jump over those bytes instead, as the accelerator can, gives the check up once you read on from there, until you seek back to a point you had already read up to. A seek to or past the member's end keeps it either way. [Damaged archives](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#damaged-archives) lists the other seeks that jump, and what is still checked after one. If you'll seek a lot, extracting the member to a file first is often faster. ## Several members at once By default only one member stream can be open at a time, and opening a second while the first is still open raises. A thread pool that reads different members at once needs `concurrent_operations=True`, which can't be combined with `streaming=True`. Each stream is opened the same way `open` always opens one, so on a solid archive it may first decompress the members stored before it. The streams share the source and take turns reading compressed data from it. On a ZIP, each member is then decompressed on its own, so several of those can run at once. On a TAR, and on an ISO, they share one file handle, so one read waits while another has it. Opening the archive once per worker still reads the index each time, which these streams do not. The exception is a source where each jump to a new position is costly, such as a member of another archive or a file read over the network. There, streams that take turns make the source jump back and forth, and each jump throws away what it had buffered. With `concurrent_operations=True`, threads can also share the rest of the reader. They can list members, iterate, run `stream_members()` passes and `extract_all()` at the same time, and if several need the member list at once, one builds it while the others wait. Each member stream still belongs to one thread. Opening the archive once per thread is also safe, as long as each thread opens it from its own path or file: one file object belongs to one reader. `close()` waits for an `open()` or `get()` running in another thread, then closes the streams still open on it. It does not wait for reads on those streams: a read running in another thread returns what it already had or raises `ValueError`, as a closed file does. Without `concurrent_operations=True` the reader belongs to one thread. If another thread is inside a reader call, such as `members()`, `open()`, or a `stream_members()` or `extract_all()` pass, `close()` raises `ArchiveyUsageError` and the reader stays open. Otherwise the close goes ahead, and the next `read()` on a member stream it closed raises `ValueError`, as a closed file does. Once a `close()` has returned, calling it again does nothing; after one that raised, the reader is still open and you still have to close it. The functions you give archivey, such as `on_diagnostic`, `on_progress`, a password provider, or a `members=` or `filter=` function, run on the thread whose call to archivey needs them, before that call returns. When several threads wait for the member list, the thread that builds it runs the functions that listing calls. One case falls outside your calls: a member stream or a `stream_members()` pass that you don't close is closed when Python frees it, on whatever thread Python frees it on, and a diagnostic that this close reports reaches `on_diagnostic` on that thread. That is usually the thread that drops the last reference to it, but for an object caught in a reference cycle it is the thread where the garbage collector happens to run. Close streams and passes yourself, or use `with`, to keep every call on your own threads. Archivey never calls your code from a thread it started. ## The same result everywhere With the `seekable` extra installed, archivey reads large DEFLATE data, and bzip2 data opened with `seekable_members=True`, through the rapidgzip [accelerator](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop), which is faster on large files. It returns the same data and errors as the standard library and tools such as 7-Zip and unzip, except for a few files built on purpose, which no known tool writes: a ZIP or 7z member holding two DEFLATE streams under one CRC ([Where the guarantees stop](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop)), and a cut `.gz` whose last bytes are made to look like its trailer ([Limits and accelerators](https://archivey-dev.github.io/archivey/formats/index.md#limits-and-accelerators)). For the first, the result can also change with how many cores the machine has and with the platform. If your code must give the same answer as other tools, on every machine, and you would rather it be slower than differ, turn the accelerator off: ```python config = archivey.ArchiveyConfig( use_rapidgzip=archivey.AcceleratorMode.OFF, use_indexed_bzip2=archivey.AcceleratorMode.OFF, ) with archivey.open_archive("data.zip", config=config) as archive: ... ``` The standard library then decodes everything. A backward seek in gzip or bzip2 data decompresses again from the start of the member, and a large gzip file reads on one core. ## Why these aren't on by default Seeking and reading several members at once are off by default, and `streaming=True` turns off out-of-order reads. Each of these can be slow in some cases, and nothing in the calling code shows it. Some seeks also skip the check for damage. With these defaults, the risky pattern raises instead of running. A ZIP can be read out of order at no cost, but a `.tar.gz` can't. If archivey raised only on the `.tar.gz`, code tested on ZIPs would first fail in production. So archivey raises on every format, and the mistake shows up during development. ## Summary | If you want to | Open with | What it costs | | ---------------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Read a few members by name, or list and then read | nothing | Out-of-order reads can be slow on [solid archives](#solid-archives) | | Read everything once | `streaming=True` | [One pass only](#reading-once); no reads by name | | Read from a pipe | `streaming=True` | [TAR, ZIP and single compressed files only](#reading-once) | | Seek inside a member | `seekable_members=True` | [A backward seek may decompress again](#seeking-inside-a-member), and a seek that [jumps over data](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#damaged-archives) stops the checksum check | | Read members from several threads | `concurrent_operations=True` | [Slow on sources where each jump is costly](#several-members-at-once) | | Get the same result as other tools, on every machine | `use_rapidgzip` and `use_indexed_bzip2` set to `OFF` in `ArchiveyConfig` | [Large gzip and bzip2 reads are slower](#the-same-result-everywhere) | # Extracting ## Extracting all or some members ```python with archivey.open_archive("download.zip") as archive: archive.extract_all("out/") ``` `extract_all` writes every member under the destination folder. It's safe by default: nothing lands outside `out/`, and an archive that expands far beyond its size is stopped. [What each policy does with unusual members](#what-each-policy-does-with-unusual-members) has the details. To extract from a pipe, open it with `streaming=True`. Extraction reads each member once, in order, so it needs no other option. A ZIP keeps its index at the end, so from a pipe archivey first writes each member's data into one `.archivey-tmp-` folder inside the destination, and puts the members in place only once it has read the index, the same way it does from a file. If the run stops before then, nothing of it is left in the destination. When `members` is a function it's called twice for each member, and sees `member.member_state_final` as `False` the first time. A `filter` is called once, on the member as the index describes it. Its `members` argument picks what to extract. It takes names, ArchiveMember objects from a listing, or a mix of both: ```python with archivey.open_archive("download.zip") as archive: archive.extract_all("out/", members=["README.md", "docs/guide.pdf"]) ``` Names must match exactly, and a directory's name ends in `/`. `members=["docs/"]` extracts only the directory entry, not what's in it, and `members=["docs"]` extracts nothing. A name that matches nothing is noted as the diagnostic `MEMBER_SELECTOR_UNMATCHED`, which logs a warning. `members` also takes a function, which gets each member and returns whether to extract it: ```python with archivey.open_archive("download.zip") as archive: archive.extract_all("out/", members=lambda member: member.name.endswith(".txt")) ``` `archivey.paths` builds such functions from wildcard patterns, with either of two sets of rules: ```python from archivey.paths import match_glob, match_tar_glob with archivey.open_archive("download.zip") as archive: archive.extract_all("out/", members=match_glob("docs/**")) archive.extract_all("out/", members=match_tar_glob("*.md")) ``` | Function | `*` | A folder's name | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `match_glob` | Matches within one folder level, and `**` matches any number of levels, so `"docs/**"` selects a folder with everything in it | Selects only the folder, unless you pass `with_contents=True` | | `match_tar_glob` | Matches across folders too, as in `tar`, so `"*.md"` selects every `.md` file at any depth | Selects the folder with everything in it | Another argument, `filter`, sees each member before it's checked and written. It can change the member by returning a changed copy, with a new name or new permissions, for example, or skip it by returning `None`. ## Options ```python with archivey.open_archive("download.zip") as archive: archive.extract_all( "out/", policy="strict", # how much to trust names and permissions overwrite="error", # what to do when a file is already there on_error="stop", # whether a damaged member stops the rest abort_on=[], # events that stop the whole extraction at once on_progress=None, # called as files are written limits=archivey.ExtractionLimits(max_extracted_bytes=2 * 2**30), # how much it may write dry_run=False, # run it without keeping anything ) ``` These are the defaults. `policy` decides how much of what the archive says about names and permissions gets written as it is: | `policy` | Names | Permissions | | -------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"strict"` (default) | Rewritten to a portable spelling, or refused when that isn't possible | Files keep their stored permissions without setuid, setgid, sticky or group and other write, so at most `rwxr-xr-x`; a file the owner can't execute is executable by nobody; folders always `rwxr-xr-x` | | `"standard"` | As in `"strict"`, but trailing dots and spaces are kept, and absolute paths are placed inside the destination | As stored, without setuid, setgid and sticky bits. Files are `rw-r--r--` and folders `rwxr-xr-x` when the archive stores no permissions | | `"trusted"` | As stored, but absolute paths are placed inside the destination | As stored, and the owner too when running as root. When the archive stores no permissions, whatever your system gives a new file | `overwrite` decides what happens when a file is already where a member would go: | `overwrite` | Effect | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `"error"` (default) | The member fails, and `on_error` decides whether extraction goes on | | `"skip"` | The existing file stays. This isn't a failure, so `on_error` doesn't apply | | `"replace"` | The existing file is deleted, and the member is written. An existing folder is replaced only if it's empty, and otherwise the member fails | | `"rename"` | The member is written next to it, with a number before the extension: `a.txt` becomes `a (1).txt` | A folder member merges into a folder that's already there. A file member whose path is an existing folder is a conflict like any other. `on_error` decides what a failed member does to the rest of the extraction: | `on_error` | Effect | | ------------------ | ------------------------------------------------------------- | | `"stop"` (default) | The first failure raises, and extraction stops there | | `"continue"` | The failure is recorded in the report, and extraction goes on | A refused member isn't a failure, so `"stop"` doesn't stop there. `abort_on=["blocked_member"]` does. Why doesn't a refused member stop the extraction? Refusing it is the protection doing its job, and the rest of the archive is still safe to write. Otherwise one stray name, such as an absolute path in an archive made on another system, would stop everything after it. `abort_on` lists events that stop the whole extraction immediately if they happen, even ones that aren't failures. It's empty by default: | `abort_on` value | Raises when | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"blocked_member"` | A member is refused | | `"name_collision"` | Two members of the archive would be written to the same path. This includes a file member landing on a folder that an earlier member created, as `x` does after `x/f`. This raises before `overwrite` is applied, so even with `"rename"` the second member isn't written. Not checked under `"trusted"` | | `"name_sanitized"` | A name is rewritten to its portable spelling, or an absolute name is placed inside the destination. Under `"strict"` an absolute name is refused instead, so `"blocked_member"` is the one that catches it | After an abort there's no report, and the files already written stay on disk. Each of these strings is also an enum value, so `policy=archivey.ExtractionPolicy.STRICT` works as well as `policy="strict"`. The enums are `ExtractionPolicy`, `OverwritePolicy`, `OnError` and `AbortOn`. The strings ignore case, and `-` works in place of `_`. `on_progress` gets a snapshot after each member and every so often during a large one: the member, the bytes and members done so far, and the expected totals when the archive records them. It's meant for progress bars. A ZIP read from a pipe reports each member as its data is read, before its index says whether the extraction keeps it, so the count of members done can include one that isn't extracted in the end. Its counts of members extracted and blocked stay at 0 until the members are in place, and one last snapshot gives them. `dry_run=True` runs the same extraction into a temporary folder, which is deleted before the call returns. Every member is decompressed and checked, but the files it creates stay empty, and nothing is written to `out/`. What's already in `out/` is looked at, though: the temporary folder gets an empty copy of each file, folder and symlink there that a member's path runs into. That costs a listing of each folder of `out/` the extraction goes into, not a copy of everything in it. So the report shows what extracting into `out/` as it is now would do, with the path each member would be written to. A member that would clash with a file already in `out/` fails under the default `overwrite="error"`, as it would for real, and is skipped, replaced or renamed under the other policies. ## What each policy does with unusual members These are refused under every policy: - `../evil.txt` or `a/../../evil.txt`. - A link to `../../outside` or `/etc/passwd`. - A device file or a named pipe. These are written as is under every policy, and noted: - `invoice\u200b.pdf`, with a zero-width space, word joiner or byte-order mark, is noted as `MEMBER_NAME_ZERO_WIDTH_CHARACTER`, which `DiagnosticPolicy.STRICT` turns into an exception. - A name with another invisible character, such as a zero-width joiner or a soft hyphen, is noted as `MEMBER_NAME_INVISIBLE_CHARACTER`. The rest depend on the policy: - **`/etc/evil.txt` or `C:/evil.txt`:** `"strict"` refuses it. `"standard"` and `"trusted"` write it as `etc/evil.txt` or `evil.txt`. - **`CON`, `aux.txt` or `file:ads`, which Windows can't create:** `"strict"` and `"standard"` refuse it. `"trusted"` writes it as is. - **A name with characters that reorder how it displays, making an `.exe` look like a `.png`:** `"strict"` and `"standard"` refuse it. `"trusted"` writes it as is. - **`notes.`, with a trailing dot and space:** `"strict"` writes it as `notes`. `"standard"` and `"trusted"` write it as is. - **`a/.../b`, with a folder named only dots or spaces:** `"strict"` and `"standard"` refuse it, because Windows would write it into `a`. `"trusted"` writes it as is. - **`caf\xe9.txt`, a name that isn't valid UTF-8:** `"strict"` and `"standard"` write it as `caf%E9.txt`. `"trusted"` writes it as is. - **`what?.txt`, with a character Windows doesn't allow:** `"strict"` and `"standard"` write it as `what%3F.txt`. `"trusted"` writes it as is. - **`back\slash` in a TAR:** `"strict"` and `"standard"` write it as `slash` in a folder `back`. `"trusted"` writes it as is. - **`README`, then `readme`; `café` written two ways in Unicode; or `notes`, then `notes.`:** under `"strict"` and `"standard"`, the second one counts as a file already there. `"trusted"` writes both, if the disk tells them apart. - **A file with mode `rwsr-xr-x`:** `"strict"` and `"standard"` write it as `rwxr-xr-x`. `"trusted"` writes it as is. - **A file with mode `rw-rw-rw-`:** `"strict"` writes it as `rw-r--r--`. `"standard"` and `"trusted"` write it as is. `"strict"` and `"standard"` treat `README` and `readme` as the same file on every system. An archive that writes more in total than `limits` allows stops the whole extraction, whatever the policy. Why the same result on every system? So code tested on one system behaves the same on another. macOS and Windows can't tell `README` from `readme` by default, and if `CON` were written on Linux and refused only on Windows, the first time you'd see the refusal is on a user's machine. `CON` and names with `:` are refused rather than rewritten because on Windows they're unsafe, not just unusual: `CON` opens the console, and `file:ads` writes data hidden inside `file`. A name that isn't valid UTF-8 is percent-escaped for the same reason: Linux writes it as is, but macOS refuses it. `%E9` is the byte's value, as in URLs, and a `%` in a rewritten name is written as `%25`, so the original can be worked out again. A name that already held `%E9` is written unchanged, so only a name archivey rewrote can be reversed: `result.presented_name` and `NameRewrite.PORTABLE_NAME` in `result.rewrites` say which ones those are. A `filter` sees each member before these checks run, so it can rename one they would refuse, such as `../evil.txt`, and that member is then written under its new name. A ZIP read from a pipe is the exception: there the checks also run on each member's name as it arrives, before its data is kept, so a member they refuse fails even if the filter renames it. Device files are refused whatever the filter does, and a link pointing outside is refused unless the filter also changes its target. `archivey.sanitize_names` is a ready-made filter for this, passed as `filter=` to `extract_all`. It drops a leading `/` or drive letter, resolves or drops `..`, removes those reordering characters, and adds `_` to names Windows reserves, such as `CON`. In a symlink's target it makes only the character and reserved-name fixes, and it leaves a hard link's target as stored. A hard link gets the contents of the file it points to in the archive, whatever happens to that file's name. If a filter renamed that file, the link points to the new name. If the file was left out, by `members=` or a filter, the link is written as a separate copy of its contents, except with `streaming=True`, where that data is gone by then and the link fails. If the file is refused, the link is refused too. Changing a hard link's target in a filter does nothing. A refused member isn't written, and the rest of the archive still extracts. The call returns a report with one result for each member, with the path it was written to in `result.path` and the member as the archive stored it in `result.member`. That shows what was refused: ```python with archivey.open_archive("download.zip") as archive: report = archive.extract_all("out/") for result in report: if result.status is archivey.ExtractionStatus.BLOCKED: print(result.member.name, result.error) ``` If you'd rather stop at the first refused member, `abort_on=["blocked_member"]` raises instead. ## Damaged members A member that fails partway leaves nothing behind. Archivey writes each file under a temporary name and renames it only once it's complete. The files written before the failure stay on disk, even with `on_error="stop"`. Going over the total size or entry count in `limits`, or the whole archive expanding too much, stops the extraction even with `on_error="continue"`, since it's a sign that the whole archive is hostile. A single member that expands too much fails like any other member. If the damage is in the list of members itself, such as a TAR, RAR or ZIP cut short, the members before it are written and the call then raises, with either `on_error`, and returns no report. If a ZIP's central directory is damaged, archivey also finds the members after the damage from the header in front of each one's data, and writes those too before it raises. A damaged 7z index fails `open_archive` before anything is written. A process killed during an extraction can leave a file or folder named `.archivey-tmp-` followed by random characters in the destination. Once no extraction is running there, it's safe to delete. ## On Windows Creating a symlink on Windows needs Developer Mode or an elevated process. Without either, each symlink member fails with an `ExtractionError` that says so. Windows also keeps file and directory symlinks apart. Archivey makes a directory link when the target is already a directory, or when the archive lists its members up front (ZIP, 7z, RAR and ISO do). Otherwise, as with a TAR archive or a pipe, a link starts as a file link, and when the run ends archivey remakes any link whose target turned out to be a directory. Windows normally refuses paths of 260 characters or more unless its `LongPathsEnabled` setting is on. Archivey gets around the limit by handing Windows long paths in its extended form (`\\?\C:\...`), so a deep archive still extracts in full, and results and errors still show the plain path. A single name longer than the filesystem allows, such as more than 255 characters on NTFS, still fails that member. Other programs may fail to open those long-path files if the setting is off. On a volume that keeps 8.3 short names, `LONGFI~1.TXT` is a second name for `longfilename.txt`. If you use `"strict"` or `"standard"`, a member with such a name collides with the file the run already wrote, the same way `README` and `readme` do. NTFS gives one file at most 1,024 names. In the rare case an archive holds more hard links to one file than that, archivey writes one copy of the file when it fills up and links the next names to the copy, so each 1,024 links cost one copy. The copies count toward `max_extracted_bytes` and the archive-wide `max_ratio`, since a small archive could otherwise write far more than its size. ## Archives you trust By default, an extraction stops after 2 GiB of output, after 262,144 entries, or when data expands more than 1000 times. When the archive records both a member's size and its compressed size, a member that would expand too much is refused before any of it is written. Other members, and the whole archive, are checked as data is written. The expansion check ignores outputs under 5 MiB, or under 256 MiB for a whole archive read from a pipe. To extract something bigger, pass higher `limits`, or `archivey.ExtractionLimits.UNLIMITED` to turn them off. Why ignore small outputs? Small files compress very well without being dangerous. 1 MiB of zeros compresses about 1,000 times with DEFLATE and about 23,000 times with bzip2, so a check on every member would stop ordinary archives. A real bomb writes far more than 5 MiB, so it's still caught. From a pipe, archivey doesn't know the archive's size, so it compares what it has written with what it has read so far. An honest archive that starts with a large blank file looks like a bomb until the rest arrives, so that check waits longer. A sparse file in a TAR, such as a disk image, is stored without its empty regions, so it can expand far more than 1000 times and stop the extraction. `member.is_sparse` tells you which members are sparse. Some limits apply before extraction starts, such as the number of members or how much memory a decoder may use. Those are set when you open the archive, through `config=archivey.ArchiveyConfig(...)`, and the reference lists them. [Security](https://archivey-dev.github.io/archivey/security/index.md) covers what archivey relies on and where its guarantees stop. # Security Archivey treats everything in an archive as untrusted: names, link targets, sizes and the compressed data itself. Whatever an archive contains, extraction writes only inside the destination folder, the limits cap how much an extraction writes and how much memory a decoder may ask for, and damaged data raises an `ArchiveyError`. This page covers what archivey assumes about everything else, and where those guarantees stop. ## What archivey relies on Archivey checks each path in the destination right before writing to it, so a folder that another program swaps for a link between two files is caught. A program that swaps a folder at exactly the right moment could still redirect a write outside the destination, so the guarantee holds fully only for a folder that untrusted programs can't write to. If the archive changes while it's open, reads may fail or return a mix of old and new data, but the guarantees at the top of this page still hold. A folder opened as a source gets extra checks, because its entries can be replaced by other kinds of file. A read never follows a link out of the folder or hangs on a pipe put in a file's place, and a file that was replaced or changed size since the listing is refused. On a filesystem that doesn't report file identities, such as some network and FUSE mounts, a file replaced by another of the same size isn't caught, and the replacement can be a hard link to a file elsewhere on that filesystem. Everywhere else, the most you get is content written after the folder was listed. Archivey also trusts the optional packages and programs it uses, such as `unrar` for RAR data, not to be malicious. It doesn't count on them to handle every input: when one fails, you get an `ArchiveyError` like any other damaged data, except in the cases below. ## Where the guarantees stop - **A crash in native code can take your process with it.** Decoders written in C, such as the standard library's zlib, bz2 and lzma, run inside your process. Archivey runs the decoders known to crash in a child process: rapidgzip, and pyppmd for every member compressed with PPMd, a method some 7z and ZIP archives use. A crash nobody has found yet in another decoder would still abort yours, unless you [run every decoder in a child process](#decoders-in-a-child-process). Why every PPMd member? pyppmd, the PPMd decoder, has a known issue that can crash it on damaged or crafted data, fixed in an upcoming pyppmd release. Until then, archivey decodes every PPMd member in a child process, where a crash becomes a `CorruptionError` for that member. Archivey reuses the processes, so each PPMd member costs about 1.4 to 2 ms more than decoding in your process. A solid PPMd folder is decoded as one, so it pays that once, plus about 0.1 to 0.35 ms for each 4 KiB member read from it. - **Running out of memory raises `MemoryError`, not an `ArchiveyError`,** so it's never mistaken for a damaged archive. `DecoderLimits` caps the memory an archive can ask one decoder for, 2 GiB by default. Everything else needs memory on top of that: the rest of your program, archivey itself, other archives open at the same time, and other members read at once, each with its own decoder. If memory is tight, such as in a container or on a server that opens many archives, a lower limit makes an archive that asks for too much raise `ResourceLimitError` instead. - **Nothing limits how long an operation takes.** The limits cap bytes, entries and password-hashing work, not time. Raising an exception from `on_progress` stops an extraction, but the callback only runs between chunks of written data, so a decompressor that's slow to produce the next chunk can't be interrupted. If you need a hard time limit, run archivey in a process you can stop. Archivey's exceptions can be pickled, so an error raised in that process comes back with its details. For a scan that has to return data, such as member names or hashes, [Stop a scan that takes too long](https://archivey-dev.github.io/archivey/recipes/index.md#stop-a-scan-that-takes-too-long) runs it in a worker process. When only the outcome matters, the simplest way is the command line in a subprocess with a timeout: ```python import subprocess import sys try: subprocess.run( [sys.executable, "-m", "archivey", "extract", "archive.7z", "-d", "out"], check=True, timeout=60, ) except subprocess.TimeoutExpired: ... # the subprocess was killed; "out" may hold some of the files except subprocess.CalledProcessError: ... # archivey refused the archive or stopped part-way; it said why on stderr ``` - **Reading a member has no size limit.** `read()` returns the whole decompressed member as bytes in memory. A stream from `open()` decodes only as much as you ask for, so `stream.read(n)` holds about `n` bytes however large the member is, but a `stream.read()` with no size returns the rest of the member. `DecoderLimits` is the main limit that applies to a read. It caps the decoder's own memory, so the data you read takes additional memory on top. `ExtractionLimits` doesn't apply, because it covers only what `extract_all` writes to disk. A RAR archive opened from a file object rather than a path is copied to a temporary file the first time a member needs `unrar` or `unar`. `spool_limits.max_bytes` caps that copy, along with the few other temporary files archivey makes for RAR archives, listed in the table below. When the archive records a member's size in `member.size`, archivey never decodes more than that, and a member holding more raises `CorruptionError`. - **A pass that reads members as it goes checks the listing limits as it goes.** On a TAR, `stream_members()` yields members up to the limit and then raises, and `extract_all` leaves the files it wrote before then on disk. 7z, RAR and ISO archives check the limits before they give you any member, with or without `streaming=True`. A ZIP read from a file checks the member count its end record declares before any member, and the rest as it reads its index, so an index holding more members than its end record says gives you members up to the limit before the error, as a TAR does. A ZIP read from a pipe is checked like a TAR as its members arrive, and again on its index at the end. - **A ZIP extracted from a pipe is checked on what it stages before its index arrives.** The pass writes each member's data into a private `.archivey-tmp-` folder in the destination, and only the index at the end says which members the extraction keeps. The extraction limits count what's staged while it's there, so a member that's then not extracted, one your `filter` drops included, can still stop the extraction under `max_entries` or `max_extracted_bytes`, where the same archive read from a file completes. The name checks and `abort_on` run on each member's name as the pass meets it, so a hostile archive is refused before the rest of it is decoded, and the extraction can stop for a member the index would have left out. A run that stops before the index leaves nothing of its own in the destination. - **Limits apply to each archive separately.** A small archive can hold archives that each expand up to the limit, and those can hold more, so the total grows with every level you extract. Be careful if you extract archives found inside other archives. - **Accelerators are on by default for large streams.** If you open an archive with `seekable_members=True` and the `seekable` extra is installed, archivey reads bzip2 data of 1 MiB or more compressed, and DEFLATE data of 16 MiB or more, through rapidgzip. On Linux and macOS it also reads DEFLATE data of 16 MiB or more through rapidgzip without that option, when the file can seek and the data compresses 3:1 or less, because rapidgzip decompresses in parallel. DEFLATE is the compression in gzip files and in most ZIP members. rapidgzip marks places in the stream it can restart from as it reads, so a seek jumps to the nearest one instead of decompressing from the start. rapidgzip can crash on some damaged data, so it runs in a separate process, for bzip2 too, and if it crashes, the standard library's decoder takes over and reports what it finds. To avoid it, pass `config=archivey.ArchiveyConfig(use_rapidgzip=archivey.AcceleratorMode.OFF, use_indexed_bzip2=archivey.AcceleratorMode.OFF)`. rapidgzip gives the same results as the standard library except for one crafted shape: a ZIP member or 7z coder holding two DEFLATE streams, with a size and CRC that cover both. With rapidgzip you may get both streams when the second is long. It depends on the data, on how many cores the machine has and on the platform, so the same member may read on one machine and be refused on another. Without rapidgzip you get `CorruptionError`, and 7-Zip and unzip read the first stream and report a bad CRC. No known tool writes such a member. Turn rapidgzip off if your code must refuse it. Why not check for it? rapidgzip doesn't say where a DEFLATE stream ends. Archivey checks the end of the data from the last place rapidgzip can restart from, which catches a cut stream and any bytes after the last one. Finding where an earlier stream ended would mean decompressing the member again from the start, which costs as much as reading it without rapidgzip. Why a separate process, and why those sizes? The crashes seen so far are in rapidgzip's DEFLATE decoder. Its bzip2 decoder hasn't been seen to crash, but it runs in a separate process too as a precaution, since it's part of the same library. Archivey's fuzz tests run both and compare their output with the standard decoders. Starting that process takes about 45 ms; on Linux, later members of the same archive file reuse it and skip that cost. On a full read, rapidgzip saves about 3.4 ms per MB of compressed DEFLATE data, so it pays for itself from about 13 MB, and 16 MiB keeps it safely past that point. The standard library decodes bzip2 much more slowly, so there the process pays for itself between about 0.6 and 2 MiB, depending on how many cores you have, and 1 MiB sits between the two. - **After a seek, a crafted `.xz` or `.lz` file can return different data at a position you've already read.** A seek restarts at the block the file's own index names, and a crafted index can name the wrong one, so reading again after a seek back can give different bytes from the first read, with no error. Why trust the index? Where a block really ends is known only by decompressing it, and not having to decompress is why the index exists. Checking the index first would make every seek decompress from the start, on every honest file too. - **On Windows, a folder source is less protected.** Windows can't open a path without following links, so a subfolder swapped for a link during the listing can list files from outside the folder. Reading a file still checks it's the one listed where the filesystem reports file identities. Where it doesn't, only the last part of the path is checked, after the file is opened, so a folder above it swapped for a link can serve a same-size file from outside. ## Decoders in a child process `ArchiveyConfig(process_isolation=...)` says which decoders run in a child process, where a crash costs only the member being read: - **`ProcessIsolation.KNOWN_CRASHES`**, the default, isolates the decoders that have crashed on some input: rapidgzip and pyppmd. The decoders it leaves in your process cost nothing extra; each PPMd member, or each solid PPMd folder, costs about 1.4 to 2 ms more, and each 4 KiB member read from a solid PPMd folder about 0.1 to 0.35 ms, with the copying of larger members on top ([why every PPMd member](#where-the-guarantees-stop)). Where no child process can start, such as in a frozen application, a PPMd member raises `ResourceUnavailableError`, and rapidgzip gives way to the standard library's decoder with one warning. - **`ProcessIsolation.NATIVE_DECODERS`** also isolates zlib, bz2, lzma, zstd, lz4, brotli and inflate64 (Deflate64), detection's trial decodes included. Reads take longer. Measured on one Linux machine, a single stream took about 1.05 to 1.5 times as long for bzip2, 1.1 to 1.6 times for xz, 1.4 to 2 times for gzip and deflate, and 2 to 6 times for zstd, mostly copying data between processes. Each member with a decoder of its own also costs about 0.6 to 1 ms (1.4 to 2 ms for PPMd), which is most of the cost for an archive of many small members; a solid 7z folder is one decoder, plus about 0.1 to 0.3 ms for each 4 KiB member read from it, with the copying of larger members on top. Each process holds about 9 MB while it waits to be reused. Where no child process can start, reading raises `ResourceUnavailableError`; nothing is decoded in your process instead. [`scripts/bench_process_isolation.py`](https://github.com/archivey-dev/archivey/blob/main/scripts/bench_process_isolation.py) in the repository measures these costs on your machine. - **`ProcessIsolation.NONE`** decodes everything in your process. It's for trusted files, tests, and places that can't start processes. A damaged or crafted PPMd member can crash your process, and rapidgzip isn't used. AES decryption and checksums always run in your process. A child that crashes raises `CorruptionError`, and one killed from outside raises `ResourceUnavailableError` (SIGKILL, most often the out-of-memory killer) or `ReadError`. No setting limits how long a decoder takes; for that, see the time-limit recipe [above](#where-the-guarantees-stop). Why are processes reused, and what if a member damages one? Starting a process costs 12 to 18 ms, which on an archive of thousands of small members would be minutes, so a process that finished a member cleanly decodes the next one. One that reported an error, or a PPMd member that didn't end cleanly, isn't reused. A crash in a reused process isn't blamed on the member being read, since an earlier member may have caused it: that member is decoded again in a new process, and only a crash there counts. A process decodes at most 64 members before it's replaced. ## Hardening RAR data is decompressed by an external program found on your `PATH`: `unrar` or `rar`, or `unar` when neither is installed. Like the optional packages, it's [trusted not to be malicious](#what-archivey-relies-on), and archivey checks only that it runs and that archivey works with its version, which can be several years old. Older versions can have security bugs that later ones fixed, so an up-to-date version is safer. `unar` receives the password on its command line, where other users on the same machine can see it while it runs. If you set `rar_decompressor=RarDecompressor.UNRAR` in `ArchiveyConfig`, archivey never runs `unar`. With `RarDecompressor.NONE` it runs no external program at all, so stored RAR members still read (archivey decrypts an encrypted one itself if `cryptography` is installed, and raises `PackageNotInstalledError` if it isn't), and the rest raise `UnsupportedFeatureError`. The same goes for the optional packages. The `[recommended]` extra sets minimum versions, such as brotli 1.2.1 for a decoder buffer overflow that version fixed, but archivey uses a package that's already installed whatever its version. If one came from somewhere else, such as a system package, upgrade it, for example with `pip install -U brotli`. Every limit has a default that lets large legitimate archives through. A higher limit lets a malicious archive use more memory, disk space or processing time before it's stopped, so if you know what your archives look like, lower the limits to fit them. They're set in `ArchiveyConfig`, and `extract_all` also takes `limits=` for one call. [`scripts/measure_limit_costs.py`](https://github.com/archivey-dev/archivey/blob/main/scripts/measure_limit_costs.py) in the repository measures what each limit costs in time and memory on your machine. - **`extraction_limits.max_extracted_bytes`**, default 2 GiB, caps the bytes one extraction writes. It costs disk space, plus 5-10 ms per MiB written, 10-20 s at the default. - **`extraction_limits.max_entries`**, default 262,144, caps the members one extraction writes. It costs about 0.6 ms per file written, under 3 minutes at the default. - **`extraction_limits.max_ratio`**, default 1000, caps how much a member, or the whole archive, expands. It's checked as data is written, and for a member whose size and compressed size the archive records, before any of it is written. Outputs under 5 MiB, or 256 MiB for a whole archive read from a pipe, aren't checked. It has no cost of its own. - **`listing_limits.max_members`**, default 262,144, caps the members an archive can list. It costs about 25-50 µs and 1.5 KiB of memory per member, under 15 s and 384 MiB at the default. - **`listing_limits.max_metadata_bytes`**, default 64 MiB, caps the text kept for names, comments and link targets. A ZIP read from a pipe also keeps up to this much of its small members' data, in case they are links, on top of the text. It costs about 1.7 MiB of memory per MiB counted, 110 MiB at the default, and up to 64 MiB more for a ZIP read from a pipe. - **`decoder_limits.max_decoder_memory`**, default 2 GiB, caps the memory an archive can ask a decoder for, and the memory of rapidgzip's process for gzip, zlib and DEFLATE. It costs the memory itself. - **`decoder_limits.max_key_derivation_rounds`**, default 2^25, caps the password-hashing work per open archive. It costs 0.2-0.7 µs per round, 7-25 s at the default. - **`decoder_limits.max_ppmd_held_input`**, default 16 MiB, caps the largest PPMd member that's held and handed to the decoder whole. A larger one is fed in pieces, which `process_isolation` keeps in a child process. It costs up to twice the limit in memory while a PPMd member decodes, 32 MiB at the default. - **`spool_limits.max_bytes`**, default 1 GiB, caps the temporary disk space for RAR archives. `unrar` and `unar` need the archive as a file, so a RAR opened from a file object is copied, and so are volumes archivey can't link into its temporary folder and, for `unar`, an archive with data before it. A pass over a solid RAR also keeps the members that later file copies repeat. It costs the disk space itself. The costs were measured on one 2.8 GHz core. Why 2^25 rounds? An archive made by the usual tools costs, per password, 2^19 rounds for 7z, 2^18 for RAR3 (a fixed cost the archive does not choose), and 2^16 to 2^17 for RAR5. RAR5 asks for 2^15, and archivey runs that derivation two or three times: once for the password check, and once each for the key checksums are verified with and the key archivey decrypts with, when it needs them. A crafted 7z or RAR5 archive can ask for up to 2^24 rounds and repeat that work for each member. Two such 7z members use the whole default and a third is refused; one such RAR5 member already goes over it. Each password you try costs the same work again. Going over a limit raises `ResourceLimitError`. If you know which formats you accept, a [`FormatPolicy`](https://archivey-dev.github.io/archivey/opening/index.md#configuration) with an `only=` list refuses the rest before any reader runs on them, including formats a later archivey release adds and plugin formats that another library registers. An `exclude=` list refuses only what it names, so a format added later opens. The policy decides which reader runs, not what detection reads: detection still looks for every built-in format, within `detection_budget`. For archives from strangers, the default `policy="strict"` refuses or rewrites every kind of unsafe member archivey knows of. The [policy table](https://archivey-dev.github.io/archivey/extracting/index.md#what-each-policy-does-with-unusual-members) on the Extracting page shows what each policy does. Untrusted archives should be extracted into an empty folder that nothing else uses, so you can check what came out before moving it anywhere else. ## Known attacks on other archive tools Archivey's test suite replays published vulnerabilities of other archive readers that could also apply to it: path traversal, link and hard link escapes, header parsing bugs, decompression bombs, structures built to make a reader hang, member names that drive the terminal, and permissions that leave extracted files writable by others. They come from GNU tar, libarchive, Info-ZIP, 7-Zip, unrar and WinRAR, py7zr, rarfile, Apache Commons Compress, and Python's `tarfile` and `zipfile`. Each test builds the archive the advisory describes and checks that archivey refuses it, keeps it inside the destination, ends it with a typed error rather than a hang, or reads it as the format defines without allocating more than the archive's size pays for. The decompression libraries archivey calls have published bugs too: in brotli, zlib, CPython's decompressors and pyppmd. Those bugs are the libraries' to fix, so the tests for them check archivey's side: how much output one call may produce, that a decoder that failed is not called again, which decoders run in a child process, and that archivey's own CRC-32 combining ends for any length. Each test names the advisory it replays: - [`tests/test_cve_regressions.py`](https://github.com/archivey-dev/archivey/blob/main/tests/test_cve_regressions.py) - [`tests/test_cve_regressions_7z_rar.py`](https://github.com/archivey-dev/archivey/blob/main/tests/test_cve_regressions_7z_rar.py) - [`tests/test_cve_regressions_libarchive.py`](https://github.com/archivey-dev/archivey/blob/main/tests/test_cve_regressions_libarchive.py) - [`tests/test_cve_regressions_codecs.py`](https://github.com/archivey-dev/archivey/blob/main/tests/test_cve_regressions_codecs.py) - [`tests/test_cve_parsing.py`](https://github.com/archivey-dev/archivey/blob/main/tests/test_cve_parsing.py) ## Reporting a vulnerability Please report security problems privately through [GitHub's private vulnerability reporting](https://github.com/archivey-dev/archivey/security/advisories/new), not in a public issue. [SECURITY.md](https://github.com/archivey-dev/archivey/blob/main/SECURITY.md) says what's in scope and what happens after you report. The limits on this page are known and accepted. If you find one that does more harm than this page describes, please report it. # Errors and diagnostics Archivey tells you about problems in two ways. An exception means it couldn't give you a correct answer, so it stopped. A diagnostic means the answer is correct, but something along the way is worth knowing, such as a password that wasn't needed or a timestamp that was invalid and is `None`. ## Exceptions ```python try: with archivey.open_archive("maybe.7z") as archive: archive.extract_all("out/") except archivey.ArchiveyError as e: print("Could not extract:", e) ``` Every problem with the archive, or with something archivey needs in order to read it, raises a subclass of `ArchiveyError`. These are the ones you're most likely to handle: | Exception | Raised when | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CorruptionError` | The archive's data is damaged. Its subclass `TruncatedError` means the data seems to end early, but that's a guess, so catch `CorruptionError` for both | | `EncryptionError` | A password is needed and none was given, or none of the ones given works | | `FormatDetectionError` | The file isn't in a format archivey recognizes | | `UnsupportedFeatureError` | The format is recognized, but this archive uses something archivey can't handle, such as an unknown compression method | | `PackageNotInstalledError` | The archive needs an optional package or program that isn't installed. The message names it | | `ResourceLimitError` | The archive went over one of the [limits](https://archivey-dev.github.io/archivey/security/index.md#hardening). Its subclass `ResourceUnavailableError` means the system ran short instead, for example the out-of-memory killer ended a decoder, so raising a limit won't help | The reference lists the rest, such as `ReadError`, the parent of the first two. Damage can look like something else, and then the exception names what it looks like. In an encrypted member, a format with no reliable password check can't tell damaged data from a wrong password, so it raises `EncryptionError`. That's the case for 7z, and for the old ZIP encryption, whose check is a single byte. WinZip AES and RAR 5 check the password on its own, so damage there raises `CorruptionError`. A damaged header field that nothing else checks, such as a ZIP entry's compression method, can look like a feature archivey doesn't support and raise `UnsupportedFeatureError`. Either way, you can't use the file. If reading fails and the format was only a guess, the exception has `format_unconfirmed=True`. That happens when nothing in the file confirmed its format, so archivey went by the file name, or when the format has no reliable signature and only a check of the contents suggested it, as for zlib or Brotli data. The file may not be in that format at all, such as a file of zeros named `backup.gz`, rather than a damaged one. Errors that aren't about the archive also pass through as themselves. A missing file raises `FileNotFoundError`, and a disk or permission error while reading or writing raises `OSError`, so catch `OSError` next to `ArchiveyError` if your program cares about both. Any other exception type from inside archivey is a bug, and we'd like to hear about it. Why not turn every error into an `ArchiveyError`? Archivey translates the library errors it knows about, such as those for damaged data, a wrong password or a source that can't seek, and lets any other error through unchanged. Converting every error would let a single `except ArchiveyError` handle everything, but it would hide the cases nobody has looked at yet. A library failing in a way we didn't know about, or a bug in archivey itself, would look like a damaged archive. You might discard a good file, and we would never hear about the problem. When the unknown error comes through as itself, it gets noticed and reported, and we can then translate it properly. Exception messages are safe to print. Control characters from the archive, such as a member name built to move the terminal cursor, are escaped in the message. Attributes like `e.member_name` keep the raw value. ### Mistakes in the calling code Mistakes in the calling code raise `ArchiveyUsageError`, which isn't an `ArchiveyError`, so a catch-all for archive problems never hides a bug. The ones you're most likely to meet are: - calling `members()`, `open()` or `read()` on a reader opened with `streaming=True`; - opening a second member stream while another is still open, without `concurrent_operations=True`; - calling `open()` or `read()` on a directory; - giving an option a value it doesn't know, such as `policy="strictt"`. The error names the accepted values. [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md) explains the first two. A source that isn't a path or a file object raises `TypeError`, and asking for a member name the archive doesn't have raises `KeyError`, as a dictionary would. ## Damaged archives `members()` raises if the archive is damaged partway through its list of members, and gives you nothing. Iterating over the reader, or over `stream_members()`, gives you the members before the damage, then raises. On a ZIP whose central directory is damaged partway, you also get the members after the damage, found from the header in front of each one's data, before the exception. `members_report()` gives you both at once: ```python with archivey.open_archive("damaged.tar") as archive: report = archive.members_report() for member in report: print(member.name) if report.error is not None: print("The list stops early:", report.error) ``` Why does `members()` raise if the archive is damaged? It returns a list, and a shorter list looks just like a complete one, so code that only looks at the list would carry on as if the archive ended there. Iterating hands you each member as it's found, so the exception at the end still tells you the rest is missing. A TAR archive cut between two members, or inside a member's header, has nothing left that looks wrong, so it lists as a complete, shorter archive with no error. Only the diagnostic `ARCHIVE_EOF_MARKER_MISSING` shows it, and [`DiagnosticPolicy.STRICT`](#raising-or-quieting-a-diagnostic) turns it into an exception. A ZIP read from a pipe gives you its members before its index, which comes last, so some damage shows only once the index has been read, and iterating raises it after the last member. A member the index doesn't list, such as an update appended to the archive, is dropped from the list then and logged as `ARCHIVE_UNLISTED_DATA`, since reading the same archive from a file never sees it. Where the format stores a checksum for each member, reading a member from start to end checks it, and the read that reaches the end raises if the data is damaged. If you stop early, nothing is checked. A loop that reads in chunks gets every byte that could be read before the error, but those bytes may be wrong too, since archivey can't tell where the damage starts. Some archives store nothing to check a single member against, so damage there can go unnoticed: - A TAR member has no checksum of its own, and the checksum of a `.tar.gz` covers the whole file rather than one member. - ISO images and Brotli, `.Z` and `.lzma` files have no checksums at all. - `.xz`, `.zst` and `.lz4` files have one only if the program that wrote them added it. When archivey can't check a member that you'd expect to be checked, such as a 7z member encrypted without a CRC, it records the diagnostic `DIGEST_UNVERIFIABLE`, which `STRICT` also turns into an exception. A damaged member doesn't always raise before its end. Depending on the compression method, damaged data can decode to wrong bytes with no error, and only the checksum at the end catches it. A member stored without compression never notices damage on its own. A cut-off member can also end early without an error. `stream.read(member.size)` then returns fewer bytes than the member should hold, and the next read raises `TruncatedError`. So code that reads a member in one call and then stops should check the length it got, or call `read()` once more. A member whose data is damaged doesn't end a `stream_members()` loop. Its own read raises, and the loop goes on to the next member, so one pass can find every damaged member in the archive. In a solid archive the members after the damage in the same solid block fail too, each from its own read, and with random access `open()` itself can raise for them. The loop raises and stops early only when the damage is to the archive's structure, such as a TAR cut off inside a member, so that archivey can't find the next member. No option makes archivey check a member before handing you its bytes. If you must never use unchecked data, read the whole member into memory or a temporary file first, and use it only once the read has finished without an error. A member stream tells you what has been checked so far. `stream.verified` is a `Verification` level, and `stream.verifiable` is the most a read from the start to the end can reach: | Level | What has been checked | | ----------- | -------------------------------------------------------------------------------------------------- | | `NONE` | Nothing yet, or the member's password passed only a check that a wrong password can pass by chance | | `KEY` | The member isn't encrypted, or its password passed a reliable check | | `STRUCTURE` | A read reached the end, and the length and the end of the data were as expected | | `CONTENT` | A checksum stored with the data matched every byte | The levels are ordered, so you can compare them: ```python with archive.open(member) as stream: data = stream.read() if stream.verified < stream.verifiable: ... # a check this member carries didn't run ``` A member with no checksum of its own, such as a TAR, ISO or `.lzma` member, stops at `STRUCTURE`. A gzip, bzip2, lzip or zlib file reaches `CONTENT` from its own checksum when it's read from the start with no seek in between. `.xz`, `.zst` and `.lz4` files stop at `STRUCTURE` for now, even when they carry a checksum. An encrypted member whose password was accepted only on a weak check, such as the one-byte check of old ZIP encryption, is in the same position. Its `stream.verified` stays `NONE` until its checksum matches. If you close its stream before the end, or seek past bytes without reading them, archivey logs `ENCRYPTED_MEMBER_UNVERIFIED`, since the bytes you read may come from a wrong password. WinZip AES members are the exception: their check holds across seeks. If you opened the archive with `seekable_members=True` (see [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md)), most seeks keep the checksum check: a seek back, a seek to or past the end, and a seek forward that the next read has to decode its way through anyway. Bytes you read again after a seek back aren't checked a second time. A seek forward that jumps over bytes without decoding them, and lands before the end, stops the checksum check once you read on from there. That happens in a member stored without compression, through the `seekable` accelerator when it has a restart point between what you've read and where you seek to, and in a `bgzip` file or a Seekable Zstandard file, which list their restart points up front. The blocks or frames jumped over are never decompressed, so their own checksums aren't checked either. Seeking back to a point you had already read up to brings it back. The length is still checked after a jump, and damage the decompressor itself notices still raises. Once a read has raised, a later read that reaches the end raises the same error, even after a seek, so seeking back can't hand you the damaged member as if it were complete. After seeks, `stream.verified` shows whether the checksum still ran. If it's below `stream.verifiable` once you've read to the end, a seek back to the start and a read to the end runs every check. [Damaged members](https://archivey-dev.github.io/archivey/extracting/index.md#damaged-members) covers what extraction does. ## Diagnostics Each diagnostic, such as one for a name with characters that make it display as a different one, has a `code` to check for, a `message` meant for people, and a `context` with details such as the member's name, kept raw as in `e.member_name`. Archivey keeps them in a few places, depending on what you called: | After | Read them from | | ----------------------------------- | -------------------------------------------------------------- | | Opening, and anything on the reader | `archive.diagnostics`: everything since the archive was opened | | `archive.open(member)` | `stream.diagnostics`: that one read | | `extract_all` | `report.diagnostics`: that one call | | `members_report()` | `report.diagnostics`: that one listing | | Any listing | `member.diagnostics`: the ones about that member | All but the last are summaries: `counts` has an exact count for each code, and `retained` keeps the records themselves. An archive keeps at most 256 records by default, counting those attached to members, and `max_retained_diagnostic_references` in `ArchiveyConfig` changes that. Why a cap? A crafted archive can have a problem in each of a million members, and a record for each would take memory that the listing limits don't count. `counts` stays exact either way. Each diagnostic is also logged as a warning on a logger under `archivey`, such as `archivey.normalization`, so a script that calls `logging.basicConfig()` prints a line for each. To handle them as they happen instead, pass a function as `on_diagnostic=` in `archivey.ArchiveyConfig`. Archivey calls it with each Diagnostic as it's recorded, the same record the summaries keep. An exception it raises stops the operation and reaches your code. One code is logged at INFO rather than as a warning: `STREAM_REWIND_REDECOMPRESSES`, which reports how your code reads a compressed stream (seeking back), not a problem with the archive. `logging.basicConfig()` and the `archivey` command without `-v` show only warnings, so it prints nothing there. It is still counted in `reader.diagnostics`, passed to `on_diagnostic`, and raised by a policy that names it. To see its log line, set the `archivey` logger to INFO. ## Raising or quieting a diagnostic ```python config = archivey.ArchiveyConfig(diagnostic_policy=archivey.DiagnosticPolicy.STRICT) ``` With `DiagnosticPolicy.STRICT`, the diagnostics that say the archive itself is unusual, such as a name that displays as a different one, raise `DiagnosticRaisedError` from the call that found them. The ones about your own arguments, such as an unused password, and a few others, such as an empty archive, are only recorded as before. The reference lists which codes raise. `DiagnosticPolicy.PEDANTIC` raises on every code, including codes a later release adds. ```python config = archivey.ArchiveyConfig( diagnostic_policy=archivey.DiagnosticPolicy( ignore={archivey.DiagnosticCode.PASSWORD_ARGUMENT_UNUSED}, ), ) ``` A policy is two sets of codes. A code in `ignore` is no longer logged, passed to the callback or kept, but `counts` still counts it. A code in `raise_on` raises. Every other code is recorded as before. To change a preset, build a policy from the same set: `STRICT` is `DiagnosticPolicy(raise_on=archivey.ARCHIVE_INTEGRITY_CODES)`, so `raise_on=archivey.ARCHIVE_INTEGRITY_CODES - {archivey.DiagnosticCode.ARCHIVE_TRAILING_DATA}` is `STRICT` without that one code. The reference lists every code. # Command line The `archivey` command lists, checks and extracts archives, and prints members to the terminal. It comes with the base package. Progress bars need `tqdm`, which `[recommended]` installs; without it the command works the same, just without the bars. To run the command without setting up a project, use `uvx archivey-cli` (for example, `uvx archivey-cli extract photos.zip`), or `pipx install archivey-cli` to keep it. The `archivey-cli` package brings `[recommended]` with it, so the command reads every format that installs everywhere. [Install](https://archivey-dev.github.io/archivey/install/index.md) has the details. ```bash archivey photos.zip # list the members (same as archivey list) archivey test photos.zip # read every member and check its checksum archivey extract photos.zip # extract safely, into ./photos/ archivey info photos.zip # the format, and what reading it costs archivey cat photos.zip notes.txt # write one member to stdout ``` `list`, `test`, `extract` and `info` also answer to `l`, `t`, `x` and `i`. `archivey --help` lists every option. ## Listing and inspecting `archivey list` prints one line per member: its type, permissions, size, modification time and name. Three options add more: - `--long` adds columns for the compressed size, the space saved, the compression method and the CRC-32 checksum. - `--details` prints everything archivey knows about each member, as `key: value` lines. - `--digests` adds the checksums the archive stores, such as `[crc32=363a3020]`, without reading the members' data. `--long` and `--details` can't be used together. `archivey info` describes the archive instead of its members: its format and how archivey recognised it, whether it's encrypted, solid or split into volumes, how many members it has, and what reading it costs. For a `.tar.gz`, it says that listing means decompressing the whole archive, and that reading one member may mean decoding the ones before it. `archivey --version -v` prints the version, then each format and whether this install can read it fully, partly or not at all. [Install](https://archivey-dev.github.io/archivey/install/index.md#checking-what-an-install-can-read) explains what each one needs. ## Checking an archive `archivey test` reads every member to the end and compares it with the checksum the archive stores, without writing anything. It prints how many members passed and failed, and `-v` adds a line for each member. It exits with `1` when a member fails, and also when a member couldn't be tested or its checksum couldn't be checked, even if nothing failed. That happens, for example, when the members after a damaged spot can't be reached, or when a gzip file's checksum sits past the point where archivey stops looking for it. The warning printed for each one says why. A member whose format stores no checksum at all, such as a Brotli or `.Z` file, passes once it reads to the end without an error. `--outer` checks only the compression layer, as `gzip -t` does. For a `.tar.gz`, archivey reads the whole decompressed tar and checks gzip's own checksum, without looking at the members inside. gzip, lzip, bzip2 and zlib always carry that checksum, xz does unless it was written with `--check=none`, zstd and LZ4 only when the program that wrote the file added one, and Brotli and `.Z` never do. A ZIP, 7z, RAR, ISO or plain tar has no compression layer, so `--outer` refuses it. ## Extracting `archivey extract photos.zip` puts the members in a new folder named after the archive, `photos/`, so an archive with many top-level entries doesn't spread them over the current directory. If the archive holds a single top-level folder, that folder lands in the current directory instead. The new folder is always new: if something named `photos` is already there, archivey uses `photos (1)`. To choose the folder yourself, pass `-d out/`, or `-d .` for the current directory. When does a single folder stay inside `photos/`? Without `-d`, archivey never replaces or deletes anything in the current directory. If a name it needs is already taken, the default `--overwrite rename` gives the new entry a free name, such as `notes (1).txt`; with `error`, `skip` or `replace`, the folder stays in `photos/` instead. For an archive that doesn't list its members up front, such as a TAR, archivey extracts into `photos/` first and moves the folder out at the end. It leaves the folder in `photos/`, and prints why, if the move could change what a link points to (the folder is a symlink, or holds one that leads outside it), or if part of it couldn't be listed. Extraction uses the `strict` policy by default: a member that would land outside the destination, or a link that points outside it, is refused and reported, and the rest are still extracted. `--policy standard` and `--policy trusted` relax it, as [What each policy does](https://archivey-dev.github.io/archivey/extracting/index.md#what-each-policy-does-with-unusual-members) describes. Two defaults differ from the library's `extract_all()`, because a command is expected to finish and report instead of stopping at the first problem: | When | `archivey extract` | `extract_all()` | | ---------------------------------------------- | ----------------------------------------------------- | ----------------------- | | A file with the member's name is already there | writes the member as `photo (1).jpg` | raises an error | | A member fails | reports it, carries on, and exits with `1` at the end | raises an error at once | `--overwrite error --stop-on-error` makes the command behave like the library. `--overwrite` also takes `skip` (keep the file that's there) and `replace` (write over it). `--stop-on-error` stops at the first member that fails, but refused members are still only reported. To stop on those too, pass `--abort-on blocked-member`, which is the safest choice for an archive you don't trust. `--abort-on name-collision` stops when two members land on the same name, and `--abort-on name-sanitized` when a name had to be changed to be written. `--abort-on` can be given more than once. Files written before the stop stay on disk. `--dry-run` runs every check and reads every member, but writes nothing. It prints the same report and exits with the same code as a real extraction into an empty folder. Before the summary it adds one line, `peak use of the reader limits:`. It gives the largest decoder allocation archivey checked and what it was for, the bytes spooled to a temporary file at their peak, and two totals: the key-derivation rounds and the listing's members and metadata bytes. These are the numbers to compare with [`DecoderLimits`, `SpoolLimits` and `ListingLimits`](https://archivey-dev.github.io/archivey/security/index.md) before you open an untrusted archive with tighter limits. A part that is zero is left out, and when every part is zero there is no line. A run that stops before its summary, such as a limit refusal or `--abort-on`, prints no line either. Decoder memory counts only the allocations archivey checks itself: xz, LZMA Alone and zstd leave that check to their library. `--outer` removes only the compression layer: `archivey extract data.tgz --outer` writes `data.tar` and keeps the original, as `gunzip -k` does. ## Writing members to stdout `archivey cat` writes the selected members to stdout, one after another in archive order, as `unzip -p` and `tar -xO` do. Directories and links are skipped, and with `-v` each member's name goes to stderr. ```bash archivey cat config.zip settings.ini # one member archivey cat logs.tar.zst '**/*.log' # every .log file, at any depth curl -s URL | archivey cat - '*.txt' # - reads the archive from stdin curl -s URL | archivey cat - --outer | tar t ``` With `--outer`, `cat` writes what's inside the compression layer unchanged: the tar inside a `.tar.gz`, the file inside a `.gz`. Data reaches stdout as it's decoded, so if a member's checksum fails at the end, its data is already written. `cat` then prints the error and exits with `1`, as `zcat` does, so check the exit code before you trust the output. ## Choosing members `list`, `test`, `extract` and `cat` take patterns after the archive name, and work on the members that match: - `*` matches within one folder: `'*.md'` matches `README.md` at the top level, not `docs/guide.md`. - `**` matches any number of folders: `'**/*.md'` matches every `.md` file. - A folder brings its contents: `docs` and `docs/` both match `docs/` and everything in it, but not `docs.txt`. `--exclude PATTERN` leaves out the members it matches, even ones a pattern selected, and can be given more than once. On Windows, `\` works in place of `/`. Quote patterns, so the shell doesn't expand them first. If a pattern matches nothing, archivey says so. `test`, `extract` and `cat` then exit with `1`, as they do when `--exclude` leaves nothing; `list` exits with `0`. From Python, the same matching is `members=match_glob(pattern, with_contents=True)` from `archivey.paths`. ## Passwords When an archive needs a password and stdin is a terminal, archivey asks for it without showing what you type. With no terminal, as in a pipe or a cron job, it doesn't ask, and the encrypted members fail as if no password had been given. `--password` passes it on the command line instead. If you use it, other users of the machine can see the password in the process list, and your shell may keep it in its history. ## Reading from a pipe `archivey cat -` reads the archive from stdin. The other commands take a path, and a pipe can still reach them as `/dev/stdin` on Linux and macOS: ```bash cat backup.tar.gz | archivey list /dev/stdin ``` A pipe can only be read once, from start to end, so archivey reads it in a single pass. That works for TAR (compressed or not), ZIP and single compressed files such as `.gz`. 7z, RAR and ISO archives need to jump around in the file, so from a pipe they fail with exit `1` and a message to copy the input to a file first. `extract --outer` and `test --outer` also need a file; for a pipe, use `archivey cat - --outer`. A redirect from a file, as in `archivey cat - < backup.7z`, isn't a pipe, so every format works that way. ## Exit codes | Code | Meaning | | ----- | ---------------------------------------------------------------------------------------------------------------------------------- | | `0` | Success. | | `1` | The command failed: a member failed, a pattern matched nothing, `test` couldn't check everything, or an extraction stopped early. | | `2` | The command line was wrong. | | `3` | `extract` finished, with every member written or refused, and at least one refused. | | `130` | Interrupted with Ctrl-C. | | `141` | The output was closed early, as with `archivey test -v big.zip 2>&1 \| head -1`. For `test`, that means you don't have the result. | ## File names and shared options The first word after `archivey` is read as a command when it is one, so `archivey x a.zip` extracts. A file named `x` is reached as `archivey ./x` or `archivey list x`. A file whose name starts with `-` goes after `--`: `archivey -- -weird.zip`. An empty name is an error, not the current directory; write `.` for that. Options shared by every command can go before or after the command; a command's own options, such as `-d` or `--exclude`, go after it. Besides `--password` and `-v`, the shared ones are: - `--hide-progress` turns off progress bars. - `--track-io` reports how many bytes archivey decompressed and read, and how often it seeked. - `--probe-content` recognises a raw LZMA, zlib or Brotli stream whatever the file is named. These have no signature at their start, so archivey normally knows them only by their extension (`.lzma`, `.zz`, `.br` and the like). With this option it tries decoding the start of the file instead. It's off by default because some unrelated files pass the trial. `--outer` always tries decoding, so it doesn't need this option. # Recipes Each recipe on this page is a complete script for one common job. The first group is for data pipelines that download and unpack datasets. The second is for servers that check archives uploaded by users. The pages they link to explain the options in full. ## Data pipelines ### Download a dataset and extract it This replaces `tarfile.extractall`, `shutil.unpack_archive` and helpers such as torchvision's `download_and_extract_archive`: ```python import hashlib import shutil import urllib.request import archivey URL = "https://example.com/dataset.tar.gz" # The checksum published with the dataset SHA256 = "0000000000000000000000000000000000000000000000000000000000000000" with urllib.request.urlopen(URL) as response, open("dataset.tar.gz", "wb") as f: shutil.copyfileobj(response, f) with open("dataset.tar.gz", "rb") as f: if hashlib.file_digest(f, "sha256").hexdigest() != SHA256: raise RuntimeError("dataset.tar.gz does not match its published checksum") with archivey.open_archive("dataset.tar.gz") as archive: report = archive.extract_all( "data/", overwrite="replace", abort_on=["blocked_member"], ) extracted = [r for r in report if r.status is archivey.ExtractionStatus.EXTRACTED] print(len(extracted), "members extracted") ``` The same code works for a ZIP, 7z or RAR download, because `open_archive` identifies the format from the file's contents. `overwrite="replace"` lets the script run again over an earlier run's files. Without it, the first file that's already there stops the extraction. With `abort_on=["blocked_member"]`, a member that would land outside `data/`, such as `../evil.txt`, stops the extraction with an exception. Without it, that member is skipped and the report lists it as `ExtractionStatus.BLOCKED`. [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md#options) explains both options. By default, an extraction raises `ResourceLimitError` once it has written 2 GiB or 262,144 entries, or when data expands more than 1000 times. A sparse TAR member, such as a disk image, can expand that much on its own. Listing an archive also raises `ResourceLimitError` past 262,144 members, or past 64 MiB of text kept for names, comments and link targets. A larger dataset from a source you trust needs higher limits: ```python import archivey config = archivey.ArchiveyConfig(listing_limits=archivey.ListingLimits.UNLIMITED) with archivey.open_archive("dataset.tar.gz", config=config) as archive: archive.extract_all("data/", limits=archivey.ExtractionLimits.UNLIMITED) ``` `ExtractionLimits.UNLIMITED` turns off all three extraction limits, the expansion check included, and `ListingLimits.UNLIMITED` turns off both listing limits. Listing still costs about 1.5 KiB of memory per member. To keep some protection, raise single fields instead, such as `ListingLimits(max_members=5_000_000, max_metadata_bytes=2**30)`. [Hardening](https://archivey-dev.github.io/archivey/security/index.md#hardening) lists each limit with its default and what it costs. ### Extract while downloading An HTTP response can't seek, so it opens with `streaming=True`. Nothing is saved apart from the extracted files: ```python import urllib.request import archivey URL = "https://example.com/dataset.tar.gz" with urllib.request.urlopen(URL) as response: with archivey.open_archive(response, streaming=True) as archive: archive.extract_all("data/", overwrite="replace") ``` This works for TAR and ZIP archives and single compressed files. 7z, RAR and ISO archives need a source that can seek, so for them `open_archive` raises `StreamNotSeekableError`, and the download needs to be saved first, as in the previous recipe. If the download stops partway, the call raises `TruncatedError` or another `CorruptionError`. From a TAR, the members written before then stay in `data/`. A ZIP keeps its index at the end, so its members are put in place only once the whole download has arrived, and a download that stops earlier leaves nothing in `data/`. [Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once) has the details. ### Read members without extracting A member stream is a binary file object, so `json`, `csv` and most other readers take it as is: ```python import csv import io import json import archivey with archivey.open_archive("dataset.zip") as archive: with archive.open("data/meta.json") as stream: meta = json.load(stream) with archive.open("data/train.csv") as stream: text = io.TextIOWrapper(stream, encoding="utf-8", newline="") rows = list(csv.DictReader(text)) print(meta, len(rows)) ``` If you use pandas, `read_csv` takes the stream the same way: ```python import archivey import pandas as pd with archivey.open_archive("dataset.zip") as archive: with archive.open("data/train.csv") as stream: df = pd.read_csv(stream) print(df.shape) ``` Member streams don't seek by default. A reader that needs to seek, such as one for Parquet files, needs `seekable_members=True` ([Seeking inside a member](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member)). On a `.tar.gz` or a solid 7z archive, opening members one by one decompresses the data before each of them again, and the next recipe avoids that. ### Read every member of a large archive once ```python import csv import io import archivey def is_csv(member): return member.name.endswith(".csv") row_counts = {} with archivey.open_archive("dataset.tar.gz", streaming=True) as archive: for member, stream in archive.stream_members(is_csv): if not member.is_file or stream is None: continue text = io.TextIOWrapper(stream, encoding="utf-8", newline="") row_counts[member.name] = sum(1 for _ in csv.reader(text)) print(row_counts) ``` `stream_members` decompresses the archive once, from start to end, and gives you each member with a stream of its contents. The function passed to it chooses the members, and the others are skipped without being handed to you. Each stream works only until the loop moves on. With `streaming=True`, the same loop also reads from an HTTP response, as in [Extract while downloading](#extract-while-downloading). [Read everything in one pass](https://archivey-dev.github.io/archivey/index.md#read-everything-in-one-pass) explains the loop. ## Checking uploaded archives An archive uploaded by a user can be built to harm the server that opens it. The recipes in this group read uploads with lower limits than the defaults, run decoders in child processes, and treat anything unusual as a reason to refuse the upload. [Security](https://archivey-dev.github.io/archivey/security/index.md) explains what archivey protects against and where that stops. ### List, hash and test an upload ```python import hashlib import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, ) MAX_TOTAL_BYTES = 1 * 2**30 class UploadTooLarge(Exception): pass def scan_upload(path): entries = [] total = 0 with archivey.open_archive(path, streaming=True, config=CONFIG) as archive: for member, stream in archive.stream_members(): sha256 = None if member.is_file and stream is not None: digest = hashlib.sha256() while chunk := stream.read(2**20): total += len(chunk) if total > MAX_TOTAL_BYTES: raise UploadTooLarge(path) digest.update(chunk) sha256 = digest.hexdigest() entries.append((member.name, member.type.value, member.size, sha256)) return entries for entry in scan_upload("upload.zip"): print(entry) ``` Reading each member to its end also tests it: where the format stores a checksum, the read that reaches the end raises if the data is damaged. No password is given, so an encrypted member raises `EncryptionError`. Each setting in `CONFIG` does one job: - `listing_limits` caps how many members the archive may list and how much text their names, comments and link targets may take. - `decoder_limits` caps the memory the archive can ask a decoder for. Lower it to what one upload may use on your server. - `ProcessIsolation.NATIVE_DECODERS` runs every native decoder, such as zlib, bz2 and lzma, in a child process, so a crash in one raises `CorruptionError` for that member instead of ending your server. AES decryption and checksums still run in your process. A large stream reads about 1.05 to 6 times slower depending on the compression, and each member with its own decoder adds about 1 ms (2 ms for PPMd), which is most of the cost for an archive of many small members ([Decoders in a child process](https://archivey-dev.github.io/archivey/security/index.md#decoders-in-a-child-process)). - `DiagnosticPolicy.STRICT` raises `DiagnosticRaisedError` when the archive itself is unusual, such as a name that displays as a different one, bytes after the end of the archive, or a TAR that ends without its end marker ([Raising or quieting a diagnostic](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#raising-or-quieting-a-diagnostic)). `ExtractionLimits` covers only what `extract_all` writes, so a loop that reads member data keeps its own count. [Where the guarantees stop](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop) says which limits apply to a read, and [Hardening](https://archivey-dev.github.io/archivey/security/index.md#hardening) lists every limit. ### Refuse an upload with a reason Every problem with the archive raises a subclass of `ArchiveyError`. A server can turn each one into a reason for the user. `CONFIG` and the byte count are the ones from the previous recipe: ```python import zipfile import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, ) MAX_TOTAL_BYTES = 1 * 2**30 class UploadTooLarge(Exception): pass def check_upload(path): total = 0 try: with archivey.open_archive(path, streaming=True, config=CONFIG) as archive: for member, stream in archive.stream_members(): if stream is not None: while chunk := stream.read(2**20): total += len(chunk) if total > MAX_TOTAL_BYTES: raise UploadTooLarge(path) except UploadTooLarge: return "the archive is too large" except archivey.FormatDetectionError: return "not an archive" except archivey.EncryptionError: return "encrypted archives are not accepted" except archivey.CorruptionError: return "the archive is damaged" except archivey.ResourceUnavailableError: return "the server is short of memory; try again later" except archivey.ResourceLimitError: return "the archive is too large" except archivey.DiagnosticRaisedError as e: return f"the archive looks unusual ({e.diagnostic.code.value})" except archivey.UnsupportedFeatureError: return "the archive uses a feature that is not supported" except archivey.ArchiveyError: return "the archive could not be read" return "accepted" with zipfile.ZipFile("upload.zip", "w") as zf: zf.writestr("report.txt", "hello") print(check_upload("upload.zip")) with open("damaged.zip", "wb") as f: f.write(open("upload.zip", "rb").read()[:-30]) print(check_upload("damaged.zip")) ``` What each exception means for an upload: - **`FormatDetectionError`**: the file isn't in a format archivey reads. - **`EncryptionError`**: a member needs a password. 7z archives and old ZIP encryption can't tell damage from a wrong password, so damage in an encrypted member raises this too. - **`CorruptionError`**, with its subclass `TruncatedError`: the data is damaged or ends early. - **`ResourceLimitError`**: the archive went over a limit. Its subclass `ResourceUnavailableError` means the machine ran short instead: a decoder's child process was killed from outside, most often by the out-of-memory killer, or no child process could start. Retrying with more memory may work; raising a limit won't. - **`DiagnosticRaisedError`**: `DiagnosticPolicy.STRICT` raised a diagnostic. `e.diagnostic.code` says which one. - **`UnsupportedFeatureError`**: the format is recognized, but the archive uses something archivey can't handle, such as an unknown compression method. A damaged header can look like this. - **`PackageNotInstalledError`**: reading the archive needs an optional package or program that your server doesn't have, such as `unrar`. That's worth logging as your problem, not the user's. Some errors aren't about the archive, so `except archivey.ArchiveyError` doesn't catch them. `ArchiveyUsageError` means a mistake in the calling code, `OSError` a problem with your own disk, and `MemoryError` that your process ran out of memory. Any other exception type from inside archivey is a bug. [Errors and diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#exceptions) has the full hierarchy. ### Stop a scan that takes too long The limits cap bytes, entries and password-hashing work, not time. For a hard time limit, run the scan in a worker process and kill it when the time runs out: ```python import multiprocessing import zipfile import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, ) def list_upload(path): with archivey.open_archive(path, streaming=True, config=CONFIG) as archive: return [member.name for member in archive] def list_with_timeout(path, seconds): with multiprocessing.get_context("spawn").Pool(1) as pool: return pool.apply_async(list_upload, (path,)).get(timeout=seconds) if __name__ == "__main__": with zipfile.ZipFile("upload.zip", "w") as zf: zf.writestr("report.txt", "hello") try: print(list_with_timeout("upload.zip", 30)) except multiprocessing.TimeoutError: print("the scan took too long") ``` Leaving the `with` block terminates the worker, also when the time ran out in the middle of a decoder. An `ArchiveyError` raised in the worker reaches the caller with its details, since archivey's exceptions can be pickled. The worker can be any function, such as `scan_upload` from the first recipe in this group, as long as it's defined at the top level of a module. If you only need to know whether the archive passed, and not data from it, the command line in a subprocess with a timeout is simpler. [Where the guarantees stop](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop) shows it. ### Extract an upload ```python import tempfile import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, ) with archivey.open_archive("upload.zip", config=CONFIG) as archive: dest = tempfile.mkdtemp(prefix="upload-") report = archive.extract_all( dest, abort_on=["blocked_member", "name_collision"], limits=archivey.ExtractionLimits( max_extracted_bytes=500 * 2**20, max_entries=10_000, ), ) print(dest, len(report)) ``` `CONFIG` is the one from [List, hash and test an upload](#list-hash-and-test-an-upload). A new, empty folder for each upload means nothing else can be in the way, and you can check what came out before moving it anywhere else. The default `policy="strict"` refuses or rewrites every kind of unsafe member archivey knows of, and `abort_on` turns a refused member, or two members that would be written to the same path, into an exception that ends the extraction. Files written before then stay in `dest`. [What each policy does with unusual members](https://archivey-dev.github.io/archivey/extracting/index.md#what-each-policy-does-with-unusual-members) lists the cases. ### Archives inside the upload The limits apply to each archive separately. A small upload can hold archives that each expand up to the limit, and those can hold more. The simplest choice is to refuse an upload that contains an archive. `detect_format` tells whether a member is one, when it's given the whole member, and `always_probe_content=True` in `CONFIG` makes it also try decoding the start of the member: ```python import io import tempfile import zipfile import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, always_probe_content=True, ) MAX_TOTAL_BYTES = 1 * 2**30 def contains_archive(path): total = 0 with archivey.open_archive(path, streaming=True, config=CONFIG) as archive: for member, stream in archive.stream_members(): if not member.is_file or stream is None: continue with tempfile.TemporaryFile() as copy: while chunk := stream.read(2**20): total += len(chunk) if total > MAX_TOTAL_BYTES: raise RuntimeError("the upload expands too much") copy.write(chunk) copy.seek(0) try: archivey.detect_format(copy, config=CONFIG) except archivey.FormatDetectionError: continue return True return False inner = io.BytesIO() with zipfile.ZipFile(inner, "w") as zf: zf.writestr("payload.txt", "x" * 1000) with zipfile.ZipFile("upload.zip", "w") as zf: zf.writestr("report.txt", "hello") zf.writestr("more.zip", inner.getvalue()) print(contains_archive("upload.zip")) ``` The member is copied to a temporary file because its first bytes aren't always enough. An ISO keeps its signature 32 KiB into the file, and a self-extracting archive starts with a program, so the archive may start far into the member. A check of the first few KiB alone is only a fast first filter, and a member it lets through still needs the full check. zlib, Brotli and `.lzma` data start with nothing that identifies them, and the copy has no file name to go by. Without `always_probe_content=True`, `detect_format` misses them, though `open_archive` reads the same bytes saved under a name ending in `.zz`, `.br` or `.lzma`. With it, the occasional file that only happens to decode as one of them also counts as an archive, which errs on the side of refusing. Even so, the check finds only what archivey recognizes. Other tools can find more: `unzip` and 7-Zip also find a ZIP behind arbitrary leading bytes, which archivey doesn't. A compressed file such as `notes.txt.gz` or `data.zz` counts as an archive here, since it can expand as much as one. So do Office documents, JAR files and wheels, which are ZIP archives. If you need to look inside, keep one budget for the bytes read at every level, and refuse the upload when archives are nested more deeply than you allow: ```python import io import tempfile import zipfile import archivey CONFIG = archivey.ArchiveyConfig( listing_limits=archivey.ListingLimits( max_members=10_000, max_metadata_bytes=4 * 2**20, ), decoder_limits=archivey.DecoderLimits(max_decoder_memory=256 * 2**20), process_isolation=archivey.ProcessIsolation.NATIVE_DECODERS, diagnostic_policy=archivey.DiagnosticPolicy.STRICT, always_probe_content=True, ) # Archives up to 2 levels inside the upload are scanned. A third level is refused. MAX_DEPTH = 2 class Budget: def __init__(self, max_bytes): self.left = max_bytes def take(self, n): self.left -= n if self.left < 0: raise RuntimeError("the upload expands too much") def scan(source, budget, depth=0): names = [] with archivey.open_archive(source, streaming=True, config=CONFIG) as archive: for member, stream in archive.stream_members(): names.append(" " * depth + member.name) if not member.is_file or stream is None: continue with tempfile.TemporaryFile() as copy: while chunk := stream.read(2**20): budget.take(len(chunk)) copy.write(chunk) copy.seek(0) try: archivey.detect_format(copy, config=CONFIG) except archivey.FormatDetectionError: continue if depth + 1 > MAX_DEPTH: raise RuntimeError("archives are nested too deeply") copy.seek(0) names += scan(copy, budget, depth + 1) return names inner = io.BytesIO() with zipfile.ZipFile(inner, "w") as zf: zf.writestr("payload.txt", "x" * 1000) with zipfile.ZipFile("upload.zip", "w") as zf: zf.writestr("report.txt", "hello") zf.writestr("more.zip", inner.getvalue()) print("\n".join(scan("upload.zip", Budget(100 * 2**20)))) ``` Each member is copied to a temporary file, which can seek, so an inner 7z, RAR or ISO archive opens too. The copy counts against the budget before anything is opened from it, so a nested archive can't use more than what's left. The temporary files take disk space up to the budget at each level. # Formats and extras What each format can do, what it needs installed, and what most often surprises callers. The first sections cover what holds across formats; then each format has a section of its own, in the same order: what it needs, how it lists and reads, and its quirks. The maintainer handbook has more depth on [7z](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/7z.md), [ISO](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/iso.md), [RAR](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/rar.md), [TAR](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/tar.md) and [ZIP](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/zip.md), and on the single-file compressors: what they share in [single-file.md](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/single-file.md), then [gzip](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/gzip.md), [bzip2](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/bzip2.md), [xz, lzip and LZMA Alone](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/xz.md), [zstd and LZ4](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/zstd-lz4.md), [Brotli](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/brotli.md) and [`.Z`](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/unix-compress.md), and on [disk images](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/dmg.md) and [directories](https://github.com/archivey-dev/archivey/blob/main/dev-docs/formats/directory.md). ## Quick matrix "Core" means a plain `pip install archivey`, with no dependencies. *Listing* is what `archive.cost.listing_cost` reports, and *members* is its `access_cost`: direct, or solid when reaching a member means decompressing others first ([Solid archives](https://archivey-dev.github.io/archivey/reading/index.md#solid-archives)). - **ZIP:** core; `[recommended]` adds Deflate64, PPMd, WinZip AES, and Zstandard before Python 3.14. Listing: from its index; from a pipe, header by header. Members: direct. Reads from a pipe. - **TAR:** core. Listing: header by header. Members: direct. Reads from a pipe. - **Compressed TAR** (`.tar.gz`, `.tar.bz2`, `.tar.xz`, `.tar.lz`, `.tar.lzma`, `.tar.zz`, `.tar.Z`, `.tar.zst`, `.tar.lz4`, `.tar.br`): needs what its compressor needs (below). Listing: needs decompression. Members: solid. Reads from a pipe. - **7z:** core for LZMA, LZMA2, Deflate, BZip2, Delta and the executable filters (BCJ, BCJ2, ARM64, RISC-V and the others); `[recommended]` adds PPMd, Deflate64, Brotli, LZ4, AES, and Zstandard before Python 3.14. Listing: from its index. Members: solid, unless written without solid blocks (`7z -ms=off`). Needs a source that can seek. - **RAR:** listing and uncompressed members in core; compressed members need the `unrar` program ([below](#the-program-for-rar)); `[recommended]` adds encrypted headers and uncompressed encrypted members in one part. Listing: header by header. Members: solid when the archive is solid, otherwise direct. Needs a source that can seek. - **ISO 9660:** `[recommended]` (`pycdlib`). Listing: from its index. Members: direct. Needs a source that can seek. - **Directory:** core. Listing: walks the tree. Members: direct. - **Single compressed files:** gzip, bzip2, xz, lzip, LZMA Alone, zlib and `.Z` in core; Zstandard in core on Python 3.14 and later, otherwise `[recommended]`; LZ4 and Brotli need `[recommended]`. One member. Members: direct. Reads from a pipe. Seeking inside the member needs `seekable_members=True` ([Seeking inside a member](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member)). - **UDIF disk images** (`.dmg`): recognised and refused ([Disk images](#disk-images)). [Install](https://archivey-dev.github.io/archivey/install/index.md) lists what each extra installs. `archivey[recommended]` is the usual choice, and `archivey[all]` adds the `[seekable]` accelerator. At run time, [`format_availability()`](https://archivey-dev.github.io/archivey/api/index.md#archivey.format_availability) says what the current install can read. The [library analysis](https://github.com/archivey-dev/archivey/blob/main/dev-docs/library-analysis.md) explains the choice of each compression library, and [Acknowledgements](https://archivey-dev.github.io/archivey/acknowledgements/index.md) credits them. ### The program for RAR No pip extra can read compressed RAR data. Archivey runs RARLAB's `unrar` or `rar`, version 6.0 or later, from `PATH`, or else The Unarchiver's `unar` 1.10 or later. Without either, RAR archives still list, uncompressed members read, and so does an uncompressed encrypted member in one part when `cryptography` is installed. `7z` and `unrar-free` are never used. `ArchiveyConfig.rar_decompressor` picks the program: ```python ArchiveyConfig(rar_decompressor="auto") # default: unrar when found, otherwise unar ArchiveyConfig(rar_decompressor="unrar") # only unrar; unar is never run ArchiveyConfig(rar_decompressor="unar") # only unar ArchiveyConfig(rar_decompressor="none") # run no program ``` `unar` reads less than `unrar`, and it takes the password on its command line, where other users on the machine can see it ([`unar` instead of `unrar`](#unar-instead-of-unrar)). `format_availability("rar")` reports `PARTIAL`, with `unrar` in `missing`, when no RARLAB program of 6.0 or later is found, even when a usable `unar` is installed. [Getting unrar](https://archivey-dev.github.io/archivey/install/index.md#getting-unrar) says how to install it. ## Member fields across formats ### The `extra` bags `ArchiveMember.extra` is a [`MemberExtra`](https://archivey-dev.github.io/archivey/api/index.md#members-and-archive-details) and `ArchiveInfo.extra` is an [`ArchiveInfoExtra`](https://archivey-dev.github.io/archivey/api/index.md#members-and-archive-details). Both are `dict[str, object]` subclasses, and both names are importable. Reading a known key (`extra["zip.compress_type"]`) gives that key's type to a type checker, and an unknown key (from a third party, or added later) stays legal and reads as `object`. `.get()` returns `object` for every key. The `EXTRA_*` constants in `archivey.types` name the keys that have constants. Writes are not type-checked: a value of the wrong type for a known key is accepted, as an unknown key is. To replace a bag, assign a `MemberExtra(...)` or `ArchiveInfoExtra(...)` rather than a bare dict. Changing the existing bag in place works as before. The format sections below mention a key only when it is part of that format's behaviour. The full list is on the two classes. ### Creation time and `ctime` `ArchiveMember.created` is a birth time or `None`, never Unix `st_ctime` (the time of the last inode change). Several writers store `st_ctime` where a reader might expect a creation time: a Rock Ridge ISO, a Unix RAR, a PAX TAR, a 7z written on Unix, and a ZIP written on Unix by 7-Zip or libarchive. That time is in `ArchiveMember.ctime` instead. RAR, 7z and ZIP have one creation slot, so a member has at most one of the two. Rock Ridge, and a PAX TAR from libarchive (`LIBARCHIVE.creationtime`), store both times and can have both. For ZIP the writer's host decides: a creation time is `created` only from a FAT, OS/2, NTFS or VFAT host, and `ctime` from any other host, an unknown one included. 7-Zip and libarchive on macOS store `st_ctime` too. A writer that marks itself Unix while storing a birth time (libarchive on Windows) gets `ctime`, so that `st_ctime` never lands in `created`. ### Fields that change after a member is returned `ArchiveMember.member_state_final` is `False` while archivey can still update a member you hold ([Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once)). It is always `False` during a `streaming=True` pass. The fields that change, by format: - **ZIP:** `link_target_member`. `is_current` from a pipe only: from a file, the pass reads the index first, so it is final when the member is yielded. The `size` of a large macOS member listed as `None` ([Archives from macOS](#archives-from-macos)), until a checked full read fills it in. From a pipe, `size`, `compressed_size` and the CRC-32 in `hashes` of a member whose sizes come after its data, until the pass gets past its data. Also from a pipe, until the pass reads the index, every field only the index holds: `type`, `name` (where a DOS host makes `\` a separator), `mode`, `create_system`, `comment`, and the times and owner ids from its extra fields. - **TAR, 7z, RAR, ISO:** `is_current` and `link_target_member`. - **Directory:** `link_target_member`. Names never repeat, so `is_current` never changes. - **Single compressed files:** `size`, when the format records none. A full read fills it in, and the flag stays `False` until then ([Single-file compressors](#single-file-compressors)). A symlink whose target archivey has not read yet (`read_link_targets=False`) can also change: - **ZIP and 7z:** `link_target`. A Windows reparse point whose data is not a link becomes a `FILE`. - **RAR:** `link_target`, in RAR 4 only. - **TAR, ISO, directory, single compressed files:** nothing. The header, Rock Ridge or the filesystem gives the target with the entry. `members()`, `members_report()` and `get()` return members that read `True`, except unread symlinks, and single-file or large macOS ZIP members whose `size` a full read has not filled in yet. Members read `False` until a call like those, or the end of a pass, has resolved the listing's links. A `streaming=True` pass yields every member before that, in every format. A `members_report_if_available()` peek resolves no links, so its members read `False` until a later call or pass does. A random-access pass with no listing call before it may or may not yield before that point, depending on how the format is listed, so read the flag rather than assume. A pass that goes past `ListingLimits` stops updating members from that point: a member yielded after it reads `True`, with `is_current` as yielded. ### Stored digests `member.hashes` holds the digests the archive already stores, keyed by HashAlgorithm. They are readable without decompressing anything, and they are not computed: a full read still checks the data the usual way. Values are always `bytes`. A CRC-32 is four big-endian bytes, so a CRC you computed compares as `zlib.crc32(data).to_bytes(4, "big")`. - **ZIP:** `crc32`, for files and symlinks, and for a directory that declares data with a non-zero CRC. It comes from the index, or from the local header when listed without one. A WinZip AE-2 member has none. - **7z:** `crc32`, for files. - **RAR5:** `crc32`, `blake2sp`, or both, for files that store them. - **`.lz`:** `crc32`, from a source that can seek. A file of several lzip members gets one value, combined from each member's CRC, equal to the CRC-32 of the whole payload. - **TAR, directory, and every other single compressed file:** none. When a format stores none, compute a digest while reading. A first-pass dedupe index can take the best one available: ```python import hashlib import archivey from archivey import HashAlgorithm def content_key(reader, member): """Best available digest for a first-pass dedupe index.""" if HashAlgorithm.BLAKE2SP in member.hashes: return ("stored", "blake2sp", member.hashes[HashAlgorithm.BLAKE2SP]) if HashAlgorithm.CRC32 in member.hashes: return ("stored", "crc32", member.hashes[HashAlgorithm.CRC32]) # No stored digest (e.g. tar, bzip2): compute while reading. h = hashlib.sha256() with reader.open(member) as stream: for chunk in iter(lambda: stream.read(1 << 20), b""): h.update(chunk) return ("computed", "sha256", h.digest()) with archivey.open_archive("backups.zip") as reader: for member in reader: if member.is_file and member.is_current: print(member.name, content_key(reader, member)) ``` A stored CRC-32 is weak, and each format stores its own kind; a computed digest is stronger but costs a full decode. Keeping `stored` and `computed` apart in the key lets your index decide how far to trust each. ## Behaviour shared by several formats ### Bytes after the end of an archive TAR, ZIP, 7z, RAR and ISO archives, and single compressed files, all check for bytes after the archive's end. A non-zero byte there is reported as `ARCHIVE_TRAILING_DATA`, with the format in `format`: a warning under the default policy, raised under `DiagnosticPolicy.STRICT`. Zero bytes are padding and pass. ZIP, TAR, 7z, RAR and ISO look at most 1 MiB past the end, so a byte further out goes unseen. Each format section says where its end is. Inside a ZIP or 7z member, bytes after the compressed stream are corruption, not trailing bytes ([ZIP](#zip)). ### Device files, FIFOs and sockets A ZIP, 7z, RAR or ISO entry whose mode is a device, FIFO or socket, or a TAR entry whose typeflag is a device or FIFO (TAR has no socket type), lists as `MemberType.OTHER` when it stores no data. Extraction does not create it: its result is `BLOCKED`, and the other members are extracted. `extra["special_file_type"]` names the kind. In ZIP, 7z, RAR and ISO, such an entry that does store data is a `FILE`, with `MEMBER_SPECIAL_FILE_HAS_DATA` reported. Each section for ZIP, 7z, RAR and ISO says when its mode is trusted; TAR reads the typeflag instead ([TAR](#tar-and-compressed-tar)). ### Names with an unpaired surrogate A UTF-16 name (in 7z, in RAR 1.5-4, and in Joliet on an ISO) can hold a surrogate without its partner, which NTFS allows. Archivey keeps that code unit in `member.name` (`'hi\ud800'`) and lists every member, as 7-Zip does. The CLI shows the name escaped, as `hi\ud800`. `member.raw_name` always holds the stored units. Under the `STRICT` and `STANDARD` extraction policies, extraction escapes the surrogate's UTF-8 bytes as it escapes any name that is not portable: `hi\ud800` is written `hi%ED%A0%80` on every OS, and `presented_name` holds the stored name. Under `TRUSTED` it writes what 7-Zip writes. On Linux and other POSIX systems that is the three-byte UTF-8 form, `hi` followed by `ed a0 80`; a filesystem that accepts only valid UTF-8, such as APFS, refuses those bytes, and the member fails with `ExtractionError`. On Windows it is the exact name. One exception: a unit from U+DC80 to U+DCFF looks the same as an undecodable byte (see [Names in other encodings](https://archivey-dev.github.io/archivey/opening/index.md#names-in-other-encodings)), so extraction writes it as that byte, where 7-Zip writes three bytes. ### macOS sidecar files A ZIP or TAR made on a Mac often holds AppleDouble sidecars: a `._name` file with the extended attributes and resource fork of `name`, next to it (macOS `tar`) or under `__MACOSX/` (Finder, `ditto`). Archivey lists them as ordinary files, as `unzip` does. To leave them out, skip them by name with `members=`: ```python from archivey import open_archive def is_macos_sidecar(member): base = member.name.rsplit("/", 1)[-1] return member.name.startswith("__MACOSX/") or base.startswith("._") with open_archive("photos.zip") as reader: reader.extract_all("out", members=lambda m: not is_macos_sidecar(m)) ``` The test goes by name only, so it also skips a file that was really called `._notes`, and anything in a real folder named `__MACOSX`. ## ZIP - **Needs:** nothing for stored, Deflate, BZip2, LZMA and XZ members, or for ZipCrypto. `[recommended]` adds Deflate64 and PPMd (`inflate64`, `pyppmd`, the packages the 7z reader uses too), WinZip AES (`cryptography`), and Zstandard before Python 3.14 (`backports.zstd`). A missing package raises `PackageNotInstalledError` when you read such a member. - **Listing:** archivey's own parser reads the central directory, the index at the end of the file. Stdlib `zipfile` is not used. - **From a pipe:** with `streaming=True`, header by header ([below](#zip-from-a-pipe)). ### ZIP from a pipe The pass finds each member by its local header, and reads the central directory after the last one. A member is yielded with what its local header says. Until the pass reaches the directory, a member's `mode`, `create_system`, `comment`, `uid` and `gid` are `None`, and a symlink is a `FILE`. Then every member is updated in place, and `member_state_final` turns `True`. A member whose sizes come after its data (general-purpose bit 3, which every writer to a pipe sets) is yielded with `size`, `compressed_size` and its CRC-32 unknown. Its data descriptor fills them in once the pass gets past its data. Extraction keeps each member's data in a private `.archivey-tmp-` folder inside the destination, and puts the members in place only once the pass has read the directory. It calls your `filter` once on each member as the directory describes it, so the files on disk match a read from a file. A member the directory does not list (an appended update, a planted entry) is still yielded when the pass meets it. When the pass reaches the directory, that member is dropped from the listing and reported as `ARCHIVE_UNLISTED_DATA`, and extraction does not keep it. What a pipe cannot read: - A bit-3 member compressed with LZMA without an end marker raises `UnsupportedFeatureError`, since nothing marks its end. - A bit-3 PPMd member is fed to the decoder in pieces, since its size isn't known in advance. - A directory written with a data descriptor gets no stream for the data it declares: its size comes after its data, so the pass cannot know the bytes are there, and the message says so. - With several candidate passwords, a member is confirmed on its first 1 MiB. When more than one candidate is still possible after that, the first one is used. Where a pipe and a file differ, on damaged archives or ones no writer produces: - A data descriptor that disagrees with the central directory, or a local header the directory contradicts, makes the pass raise `CorruptionError` for that member once it reaches the directory. A read from a file reads it. - Two members with the same name whose directory order reverses their order in the file list and stream as from a file. Extraction raises `CorruptionError` for that name, where extraction from a file writes the copy the directory lists last. - A member whose local header holds a name the safety checks refuse fails extraction, even when the directory or your `filter` gives it a safe name, since its data was not kept. From a pipe, the pass reads to the end of the stream after the last member, so a sender that keeps the connection open after the ZIP ends makes the pass wait for the end of the stream. The members are final only then. When the sizes come after each member's data, the wait also comes in the read that ends the last member's data. A ZIP under 258 bytes waits at `open_archive` too: a tar header starts at offset 257, and until that byte has arrived the file could still be a tar. Close the sending side when the ZIP is done. ### Damaged and unusual endings - **The end record is missing** (a download cut short): the members the local headers hold are listed, without the mode, host or comment that only the directory has. Then the listing raises `TruncatedError`, or `CorruptionError` when other bytes stand where the directory should be. - **The central directory is damaged partway:** the members before the damage are listed, then the ones the local headers hold after it, and then the listing raises `CorruptionError`. `members_report()` gives you both ([Damaged archives](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#damaged-archives)). - **A ZIP64 end record is damaged:** archivey reads past it when the plain end record holds every value itself and the directory's end can be found, and reports `ARCHIVE_EOF_MARKER_MISSING`. Otherwise it lists the members the local headers hold, then raises `CorruptionError`. - **The end record disagrees with the directory**, on the entry count or with an archive comment length past the end of the file: the members still list and read, and `ARCHIVE_EOF_MARKER_MISSING` is reported after the directory's last entry. From a file, the listing is complete before any member is handed out, so `DiagnosticPolicy.STRICT` raises before the first one. With `streaming=True`, the pass yields every member and then raises. - **Bytes after the end record and its comment** are `ARCHIVE_TRAILING_DATA` ([Bytes after the end of an archive](#bytes-after-the-end-of-an-archive)). - **The local header disagrees with the directory on the name, or stores a real CRC-32 where the directory stores 0:** the member is read, from a file and a pipe alike, as `unzip` reads it. The directory's name is used and `MEMBER_LOCAL_NAME_DIFFERS` is reported, which `STRICT` refuses. The data is checked against the local CRC-32 and `MEMBER_LOCAL_CRC_USED` is reported, which `STRICT` reads: Adobe AIR packages store `META-INF/AIR/hash` this way. - **The local header names another compression method than the directory:** the member is decoded by the local header's method, from a file and a pipe alike. The local header sits next to the data and is all a pipe has, so both reads give the same bytes, still checked against the CRC-32. `MEMBER_LOCAL_METHOD_DIFFERS` is reported, which `STRICT` refuses, since a tool that trusts the directory reads the member differently. The member's `compression` still names the directory's method, as the listing comes from the directory; the diagnostic gives both. WinZip AES in the directory only is refused with `CorruptionError` from a file and `EncryptionError` from a pipe, where the local header alone says how the data is encrypted. WinZip AES in the local header only is refused with `CorruptionError` from both, from a pipe once the pass reaches the directory. WinZip AES in both headers with AES records that differ (version, key strength or method), or with one header's record missing, is refused with `CorruptionError` from a file. A pipe decrypts by the local record and refuses it too: where that decryption fails, with the error it meets (`EncryptionError` for another key strength), and where it succeeds, once the pass reaches the directory. - **Both headers declare less than the data decodes to:** the member is refused with `CorruptionError`. - **Bytes after a member's compressed stream:** a member's compressed data must hold its codec's data and nothing else, as 7-Zip checks. For most methods that is one stream. For Zstandard and XZ it is what their own formats define as the data, as in a `.zst` or `.xz` file: one or more Zstandard frames, or one or more xz streams with stream padding (groups of four zero bytes) between and after them. Junk, other zero bytes, or a second DEFLATE or BZip2 stream raise `CorruptionError` once the data before them has been read, for every method, whatever the declared size and CRC cover. So does an LZMA member without an end marker, or a PPMd member, whose declared size stops short of its data. One zero byte after LZMA data without an end marker reads, because 7-Zip's encoder sometimes writes it. Under ZipCrypto these read as the password-or-damage `EncryptionError` instead, caused by that `CorruptionError`, because the bytes are encrypted. With rapidgzip, one crafted shape may read instead: a long second DEFLATE stream, with a size and CRC that cover both streams. See [Where the guarantees stop](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop). - `ListingLimits.max_members` is checked at open against the entry count the end record declares, and again as the directory is read, in `stream_members()` too. From a pipe it is checked as the pass meets each local header, and again on the directory's entries. ### Archives from macOS - **A ZIP over 4 GiB that macOS wrote without ZIP64** reads. Finder's Compress (Archive Utility, `ditto -c -k`) stores every offset past 4 GiB with its top bits cut off, so 7-Zip and Python's `zipfile` fail on such an archive. Archivey adds back the missing multiple of 4 GiB where an offset falls below the end of the members before it, but only when a local header with the member's name is at the corrected offset. The member's CRC-32 is still checked. Each correction reports `ARCHIVE_OFFSET_WRAPPED`, which `STRICT` does not raise on, since macOS writes such archives by default. - A single member over 4 GiB has its sizes cut too. For entries carrying Archive Utility's marks, archivey corrects them with `MEMBER_SIZE_WRAPPED`: the compressed size from where the data really ends, and the size from that where Deflate's ratio leaves only one value (members under about 4 MB compressed). - **Archive Utility stores each member's size modulo 4 GiB**, so for a deflated member over about 4 MB compressed, the listing cannot tell whether the size passed 4 GiB. This happens in a Mac ZIP of any length. Such a member lists `size=None`, with the stored value in `extra["size_mod_2_32"]` and `MEMBER_SIZE_WRAPPED` reported. A read to the end, checked against that value and the CRC-32, fills in `size`; the key stays. - Finder and `ditto` add `__MACOSX/` sidecar files ([macOS sidecar files](#macos-sidecar-files)). ### Split and spanned sets - A set split by 7-Zip's `-v` (`name.zip.001` to `name.zip.00N`) opens from any part, when every part is in the same directory. The parts are byte slices of one ordinary ZIP, and archivey joins them. A missing part raises `TruncatedError`. - A spanned set written by `zip -s` (`.z01` to `.zip`) addresses its entries by disk number, and is refused with `UnsupportedFeatureError`. Join it with the tool that made it. ### Compression methods - A method archivey cannot read lists normally, and reading the member raises `UnsupportedFeatureError`. So does an LZMA member with `lc + lp` over 4 (which 7-Zip writes with `-mm=LZMA:lc=8` and liblzma cannot decode), and a PPMd member with restore method 2. Under ZipCrypto both read as the password-or-damage `EncryptionError` instead, because those settings are encrypted. - "Version needed to extract" never refuses a member: only a method or encryption archivey cannot read does. ### Encryption - Encrypted members decode every compression method an unencrypted one does. - **ZipCrypto** checks a password against one byte, so a wrong one passes about one time in 256. Its data then fails the CRC or the decompressor, and archivey raises `EncryptionError` saying the password may be wrong or the member damaged, since a damaged member read with the right password fails the same way. Decryption is pure Python and slow (a few MiB/s). A ZipCrypto member can seek, but a backward seek decrypts it again from the start. Confirming one of several passwords can be expensive on a stored member. - **WinZip AES** (method 99, AE-1 and AE-2) needs `[recommended]`. Without it, an AES member is still listed as encrypted, and reading it raises `PackageNotInstalledError`. An AE-2 member has no CRC-32 in `hashes`: its HMAC checks the data. A failing HMAC raises `CorruptionError`, with one password or several: a wrong password gets past the two-byte check only once in 65 536 tries, so damage is by far the likelier cause. - **PKWARE Strong Encryption** is not supported. Such members list as encrypted, and opening one raises `UnsupportedFeatureError`. An archive whose central directory is itself encrypted this way cannot be listed: archivey raises `UnsupportedFeatureError` when it recognises the layout, and `CorruptionError` otherwise. - Both password checks admit some wrong passwords, so the CRC or HMAC at the end of the member is the real test. With several candidate passwords, each one that passes is checked further before it is used. Closing a member stream before its end reports `ENCRYPTED_MEMBER_UNVERIFIED` when only the short check accepted the password. ### Names and comments - **Names flagged UTF-8** (general-purpose bit 11) decode as UTF-8 when the stored bytes are valid UTF-8. - **Unflagged names** should be cp437, but many tools write UTF-8 without setting the flag. So archivey uses UTF-8 when the bytes are valid UTF-8, and otherwise the `encoding=` you passed to `open_archive`, or without one `ArchiveyConfig.zip_unflagged_fallback_encoding` (default `cp437`). When UTF-8 is inferred for an unflagged name, `member_name_encoding_inferred` records it, and `STRICT` refuses it. So `encoding=` decodes only the unflagged names that are not valid UTF-8, unlike `metadata_encoding` in Python's `zipfile` or `unzip -O`, which apply to every unflagged name. - **A wrongly set UTF-8 flag is ignored.** When bit 11 claims UTF-8 but the bytes are not valid UTF-8, the name decodes as an unflagged name that is not UTF-8 does. A `member_name_encoding_inferred` diagnostic names the encoding used, with `declared_encoding="utf-8"`, and `STRICT` refuses the archive. `raw_name` keeps the stored bytes. - **An Info-ZIP Unicode Path extra field** (`0x7075`) whose checksum matches the stored bytes outranks the guess and names the member in UTF-8. `raw_name` is then the field's UTF-8 bytes, and `extra["alternate_raw_name"]` holds the stored ones. A malformed field (too short, or matching with an empty, cut-short or non-UTF-8 name) is dropped with a `member_header_record_skipped` diagnostic, and the archive opens on every Python version. Python 3.12's `zipfile` refuses a too-short field or a non-UTF-8 name. - **Comments** decode the same way. A member comment follows its name's flag (a flag the name's bytes refute counts as unset for the comment too), and the archive comment, which has no flag, decodes as an unflagged name. A byte the chosen encoding does not define stays in the comment as a surrogate escape, as in a name. An Info-ZIP Unicode Comment field (`0x6375`) whose checksum matches the stored comment gives a member comment whose bit 11 is clear in UTF-8, by the Unicode Path field's rules, as 7-Zip reads it. ### Times, owners and extra fields - Times start from the DOS fields, then NTFS, then the Unix fields (`0x000d`, `0x5855`), then the Extended Timestamp field when present. The Unix fields are read as unzip 6.00 reads them. macOS `ditto` and Archive Utility write only `0x5855`, so their archives list `modified` and `accessed` in UTC. - Owner ids: Info-ZIP `zip` on Unix records `uid` and `gid`, read as unzip 6.00 reads them. `uname` and `gname` stay `None`, and Windows writers store no owner. - Extra fields archivey takes nothing from (a vendor's own, or a known one found malformed) stay readable as `extra["zip.extra_fields"]`, as `{tag: data}`. ### Member types - A Unix mode is read only when "version made by" says Unix. A device, FIFO or socket follows [the shared rule](#device-files-fifos-and-sockets); `zip -FI` stores a named pipe's content as data, which makes it a `FILE`. - **A directory entry that declares data** (APPNOTE gives a directory none) lists as a directory with that size, and `MEMBER_DIRECTORY_DATA_IGNORED` is reported; `STRICT` refuses the archive. Extraction creates the directory, as unzip and 7-Zip do. `read()` returns the bytes, checked against the stored CRC-32, and so does the stream `stream_members()` yields for it. A CRC of 0, the usual value for a directory, is no digest: when it was the entry's only check (unencrypted or ZipCrypto), `DIGEST_UNVERIFIABLE` says the bytes come back unchecked. A WinZip AES entry has its HMAC check them, so nothing is reported. A directory that declares size 0 over a body larger than an empty one of its method is reported too (`read()` has nothing to return for it). The Java `jar` tool's deflated empty directory body (compressed size 2, size 0) is not. ## TAR (and compressed TAR) - **Needs:** nothing for a plain `.tar`. A compressed TAR needs what its compressor needs ([Single-file compressors](#single-file-compressors)). - **Listing:** archivey parses the headers itself, the same way on every Python version. - **Old tars with no magic:** a v7 tar (`tar --format=v7`, old bsdtar or star) has no `ustar` signature. archivey recognises it by its first header's checksum, from a file, a pipe or a `BytesIO`, with or without a `.tar` name, and inside a compressor (a gzipped one is `TAR_GZ`). It is reported as `PROBABLE`, not `CERTAIN`. - **Reading members:** a plain TAR from a file reads each member straight from its place. A compressed TAR is solid: reaching a member means decompressing everything before it, so a single `stream_members()` pass is the fast way to read it. - **From a pipe:** yes. The listing reads up to 1 MiB past the end-of-archive blocks, so a sender that keeps the connection open after the tar ends makes the listing wait for more bytes or for the end of the stream. Close the sending side when the tar is done. A compressed TAR from a held pipe can also wait at open ([Single-file compressors](#from-a-pipe-held-open)). - With `concurrent_operations=True`, member reads share one file handle and take turns on it, so reads from several threads are safe but not parallel. ISO works the same way. - Hardlinks are extracted as hardlinks. An `extract_all` with no filter resolves them in one pass. - A tar from macOS `tar` can hold `._name` sidecar files ([macOS sidecar files](#macos-sidecar-files)). ### How the archive ends A tar has no member count, so a damaged header after the first one could pass for the end of the archive and shorten the listing without an error. Archivey checks how the archive ends: - **A header that does not parse** where the walk stops raises `CorruptionError`, whatever the diagnostic policy, because a well-formed tar never ends that way. This holds in random-access and streaming reads, whatever follows the bad header: more members, nothing, or a block of zeros. - **A clean end on a member boundary without the two zero blocks** is reported as `ARCHIVE_EOF_MARKER_MISSING`, not raised. A tar written without them, tars joined with `cat`, and a tar cut exactly at a member boundary look the same. When you need a listing proven complete, set that code to `RAISE` in the diagnostic policy (`DiagnosticPolicy.STRICT` does), which turns it into `DiagnosticRaisedError`. - **A first zero block followed by a block that is not zero** is reported the same way, with `context.expected_marker="second_zero_block"`. The zero block ends the members, so every member is listed and reads normally, as GNU tar ("A lone zero block") and 7-Zip list them; `STRICT` raises it. The trailing-bytes check then starts from the block after the damaged one. At least one member must come before the zero block: a file that is only a zero block and other bytes is not shown to be a TAR, and raises `CorruptionError`. - **Bytes after the end** are `ARCHIVE_TRAILING_DATA` (trailing junk, or a second archive joined on; [Bytes after the end of an archive](#bytes-after-the-end-of-an-archive)). Zero padding passes because `tar` writes 10 KiB records, so "nothing but zeros" is the strictest rule that does not flag what `tar` itself writes. On a compressed tar, the 1 MiB checked is decompressed to inspect it, and bytes after the compressed stream itself are reported the same way, with the codec's name as `format` ([Single-file compressors](#single-file-compressors)). - **Truncation inside a member's data** always raises `TruncatedError`, whatever the policy: in a listing or a pass when the walk reaches the cut, and from a member stream on the read that reaches it. A read with a size returns the bytes before the cut first; `read()` with no size raises and returns nothing. On a compressed tar, the stream's own checksum adds three cases: - A stream cut inside its footer (a missing gzip CRC-32, say) does not fail the listing, since every member was read whole, but its checksum was never checked. That is `DIGEST_UNVERIFIABLE`, which `STRICT` refuses. - A stream that fails to decompress past the end-of-archive blocks is `DIGEST_UNVERIFIABLE` too, worded as "whether the checksum passed cannot be told": the failing bytes may be another compressed member after the one holding the tar. - The check stops at trailing bytes, and says nothing about the stream's checksum past them. In a concatenated `.tar.gz` they are a second gzip member, and the first one's checksum was already checked. Junk inside the same gzip member can leave that checksum unchecked, so treat `ARCHIVE_TRAILING_DATA` on a compressed tar as "the members may not have been checked" too. ### Headers that declare data they should not have - **A device or FIFO header with data** (typeflag `3`, `4` or `6` with a non-zero size) raises `CorruptionError` at that header, before the end-of-archive checks run. Such an entry has no data, and GNU tar skips the header as damaged. Reading on from the next block, an all-zero payload would look like the end of the archive and drop every member after it. - **A directory header (typeflag `5`) with data** has no data area either, so the next header is read straight after it. When that header parses, the directory lists with no size, as GNU tar, bsdtar and 7-Zip list it. When the walk ends instead on a zero block inside the declared span, the directory lists with no size and `MEMBER_DIRECTORY_DATA_IGNORED` is reported (`STRICT` refuses the archive). The declared blocks, if they are there, were read as the end-of-archive marker, and the members behind them are not listed, as GNU tar does. Nothing in the file tells that from an end marker after blocks that were never written, so the report says what the header declares, and the end checks above report the rest. `open()` refuses the directory, and `stream_members()` yields no stream for it, since its bytes cannot be told from the end marker. A block that is neither a header nor zero where the data would begin raises `CorruptionError` naming the directory header. A walk that gets past the declared span (extended headers filled it) has read all it could hide, and nothing is reported. - **An old-style directory with data:** a regular-file header (typeflag NUL, `0` or contiguous `7`) whose name ends in `/` is a directory, as GNU tar reads it; 7-Zip agrees for NUL and `0`. When its size is not zero, the directory keeps that `size` and `MEMBER_DIRECTORY_DATA_IGNORED` is reported. Extraction creates the directory and skips the blocks, as GNU tar does, and `read()` on the directory returns them, as does the stream `stream_members()` yields for it, in both access modes. `STRICT` refuses the archive. - **A symlink or hard link header (typeflag `2` or `1`) with data** has no data area either. When a member header follows it at once, the size names no data, and the link lists with nothing reported, as GNU tar and `tarfile` read it. Otherwise the declared blocks are skipped, so a zero block among them does not end the archive, and `MEMBER_LINK_DATA_IGNORED` reports the link (`STRICT` refuses the archive). GNU tar and `tarfile` skip nothing and stop at such a zero block, dropping the later members. When the declared blocks are all zeros and no member header follows them, they are also the end marker of a link whose blocks were never written: the end checks then read them as the end marker, as after a file, so an intact one gets no warning, and the link is reported the same way. Extraction creates the link. ### Member types and fields - A device or FIFO follows [the shared rule](#device-files-fifos-and-sockets). A device keeps its numbers in `extra["tar.devmajor"]` and `extra["tar.devminor"]`. A device number field that is not a number is dropped and reported as `MEMBER_HEADER_RECORD_SKIPPED`, as GNU tar lists such a device too. - **A member of an unknown type is a file when it has data.** A header whose type byte archivey does not know (such as `9`) and that declares data is a `FILE` of that size, which reads and extracts like any file, as GNU tar extracts it. One that declares no data is `OTHER`, and its extraction result is `BLOCKED`. Both are reported with `MEMBER_TYPE_UNKNOWN`, which `STRICT` does not refuse, and `extra["tar.type"]` keeps the byte. - **Sparse members are extracted dense.** A GNU or PAX sparse member (`tar -S`) is written with its holes filled with zeros, so it takes its full size on disk. The zeros count as output for the [ratio limit](https://archivey-dev.github.io/archivey/security/index.md#hardening), which a TAR checks across the whole archive because a member has no compressed size. A 10 MiB sparse file packs into a 10 KiB tar, 1024:1, and `extract_all` raises `ResourceLimitError` against the default `max_ratio` of 1000. `member.is_sparse` tells you which members are sparse before you extract. For a sparse archive you trust, raise the limit with `reader.extract_all(dest, limits=ExtractionLimits(max_ratio=...))`, or `max_ratio=None`. - A PAX time record that does not parse leaves that time unset, and `MEMBER_TIMESTAMP_INVALID` reports it. ## 7z - **Needs:** nothing for LZMA, LZMA2, Deflate, BZip2, Delta, stored data and the executable filters. `[recommended]` adds PPMd, Deflate64, Brotli, LZ4 and AES, and Zstandard before Python 3.14 (3.14 has it in the standard library). - **Listing:** archivey's own parser reads the header. `py7zr` is used only to check archivey in tests. - **Reading members:** a solid block is decoded once by `stream_members()`. A random `open()` of a member in the middle of a block may decode again from the block's start. - **From a pipe:** no. The index is at the end of the file, so `open_archive` raises `StreamNotSeekableError`. ### Compression methods - **BCJ2** (which 7-Zip writes for x86 executables at `-mx9`) reads on a core install. An encrypted BCJ2 archive also needs AES, as any encrypted 7z does. Its decoder is pure Python, about half the speed of the same file under BCJ. It keeps no seek points, so a backward seek decodes again from the block start. - **ARM64** (which 7-Zip 23 writes for AArch64 executables) reads on a core install. Python's `lzma` cannot build that filter, so archivey decodes it in pure Python. On real AArch64 code a read runs at roughly four fifths of the speed of the same file with no filter. The filter slows with branch density: on data where every word is a branch, it runs about 15 times slower than on real code. - **RISC-V** (a filter 7-Zip 24 added for RISC-V executables) reads on a core install, decoded in pure Python like ARM64. On real RISC-V code a read runs at roughly half the speed of the same file with no filter. The filter slows with branch density: on data where every word is a branch, it runs about 15 times slower than on real code. - Bytes after a coder's stream are corruption, as in a ZIP member ([ZIP](#damaged-and-unusual-endings)), except that a Zstandard or LZ4 coder reads concatenated frames as one stream, so a further frame is content that counts against the declared size. ### Encryption - **AES with stored data, and neither a block digest nor a member CRC:** 7z has no password check value, so a wrong password can give garbage, as in 7-Zip. Archivey reports `DIGEST_UNVERIFIABLE` (`reason="no_integrity_anchor"`). Treat the data as unchecked. - **Confirming a password on an encrypted block** decodes only until it can decide: the first member CRC covering at least 4 bytes, or 64 KiB of output for a compressed block, whose codec rejects a wrong key within a few bytes. Peak memory is one 64 KiB chunk. Time grows with the block's size only for stored-plus-AES (or PPMd) blocks whose only CRC is at the end, with several candidates, since nothing rejects a wrong key before that CRC. Give a single known password there when you can. `ExtractionLimits` do not apply to this step. - **Partial reads after an unconfirmed password** report `ENCRYPTED_MEMBER_UNVERIFIED`. When confirmation stopped at its 64 KiB budget without reaching a CRC, the member's own CRC at its end is the only check, and a stream closed before the end skips it. - **Encrypted headers with a wrong password:** a decoded header with no file records is rejected as `EncryptionError`, never returned as an empty listing. The header's CRC is what catches a wrong password. 7-Zip stores that CRC; `py7zr` does not, and for its archives a wrong password can, rarely, decode to a header that parses with members in it, and that garbage listing is returned. - `NumCyclesPower` (the key derivation's cost) is capped at 24, or the `0x3F` no-hash value, as 7-Zip clamps it. Values from 25 to 62 raise `UnsupportedFeatureError`. ### Names and comments - **Member names** are UTF-16, so `encoding=` has no effect, and passing it reports `ENCODING_ARGUMENT_UNUSED`. A name can hold an [unpaired surrogate](#names-with-an-unpaired-surrogate). - **The archive comment** is UTF-16 too. An unpaired surrogate stays in `ArchiveInfo.comment`, so be ready for it if you print the comment. Only a comment with an odd byte count raises `CorruptionError`. ### Endings and member types - **Bytes after the archive's end** (its next header or its last packed stream, whichever is later) are `ARCHIVE_TRAILING_DATA` with `format="7z"`. Zero padding passes, although 7-Zip warns "There are data after the end of archive" for zeros too. A code-signed self-extracting `.exe` carries its signature after the archive, so it gets the warning, and `STRICT` refuses it. - A device, FIFO or socket that 7-Zip or p7zip stored on Unix follows [the shared rule](#device-files-fifos-and-sockets). The mode is trusted for this only when the attribute's `0x8000` Unix-extension bit is set. ## RAR - **Needs:** nothing to list, or to read an uncompressed member. `[recommended]` (`cryptography`) adds archives with encrypted headers, RAR 1.5-4 and RAR5 alike, and uncompressed encrypted members in one part (RAR 2.9 and later). Every other member needs `unrar` or `unar` ([The program for RAR](#the-program-for-rar)). BLAKE2sp checking needs no package. - **Listing:** archivey's own parser reads RAR 1.5 to RAR5 headers, header by header. The one exception is a compressed RAR 1.5-4 comment, which the selected program decodes; without it, or when the decoded text fails its CRC, `comment` is `None`. - **Reading members:** archivey reads uncompressed members itself. Everything else goes through the program. In a solid archive, `stream_members()` uses one `unrar p` run for the whole pass. A random `open()` out of order is a separate run that decodes from the start of the archive each time, so reading *n* members that way costs *n* full decodes. With `seekable_members=True`, a backward `seek()` on a compressed member costs the same: a new run from the start. `CostReceipt.access_cost` is `SOLID` to say so. - **From a pipe:** no; RAR needs a source that can seek. - Read only: archivey writes no RAR. ### Running `unrar` - `unrar` is preferred when both `unrar` and `rar` are on `PATH`. Archivey refuses a 5.x `unrar`. Debian 12 and Ubuntu 22.04 package `unrar` 6.x, and Ubuntu 24.04 packages 7.00, all recent enough; on an older release with 5.x, build from RARLAB's source. RARLAB's RAR for macOS download also contains `unrar`. It is not notarized by Apple, so macOS may block it until you allow it in System Settings. The "UnRAR for Mac OS X 64 bit" link on the RARLAB add-ons page is a 2018 Intel build someone else contributed. - `unrar` gets the password as a bare `-p`, with the secret on stdin, not in its arguments. `unrar` reads it as one line and would drop anything after a line break or a NUL character, so a password containing either raises `UnsupportedFeatureError` instead of being cut short ([Passwords](https://archivey-dev.github.io/archivey/opening/index.md#passwords)). - Archivey passes a password to `unrar` or `unar` only when the read needs it: the member is encrypted, the headers are, or an encrypted member comes before it in a solid archive. - An uncompressed encrypted member split across volumes, and any member encrypted with the RAR 2.0 cipher (archives from RAR 2.0 to 2.8), still need the program: each part of a split member has its own encryption record, and archivey does not implement the old cipher. - **A stream source is copied to disk for `unrar`.** `unrar` reads only files, so a RAR opened from a `BytesIO` or a file object is copied whole to a temporary file (a volume set, to a temporary directory) on the first member read that needs the program, and removed on close. Members archivey reads itself are read in place and need no copy. `ArchiveyConfig.spool_limits` (`SpoolLimits.max_bytes`, default 1 GiB) bounds the copy: over it, the read raises `ResourceLimitError` before anything is written. Opening from a path avoids the copy. - **A member read waits as long as `unrar` does.** `read()` on a RAR member stream has no time bound: archivey does not stop an `unrar` process that stalls. A solid archive can produce no bytes for a long time while `unrar` decodes the members ahead of the one you asked for, so no idle timeout would be safe. ### `unar` instead of `unrar` Under `"auto"`, the program is chosen once, when the archive is opened, and a read `unar` refuses is not retried with `unrar`. When `"auto"` picks `unar`, `archive.cost.notes` says so at open. With `unrar` chosen, `"auto"` still reads a member that `unrar` cannot be pointed at by name with an accepted `unar` ([Names unrar cannot select](#names-unrar-cannot-select)). With `"unrar"` or `"unar"`, archivey never switches programs; with `"unar"` and no usable `unar`, a read raises `PackageNotInstalledError`. Archivey runs each `unar` once on a small RAR5 archive and does not use one that decodes it wrong, as the Debian and Ubuntu packages before 1.10.8+ds1-10 do (Ubuntu 22.04 to 26.04 among them). `"auto"` then treats `unar` as absent. `brew install unar` on macOS installs a version archivey accepts. **The password is visible to other local users.** `unar` accepts a password only on its command line (`-p `), so while it runs, any user on the same machine can read it from the process list (`ps`, `/proc//cmdline`). On a shared machine, install `unrar`, or set `rar_decompressor="unrar"` so that `unar` is never used. A wrong password makes `unar` write nothing and report success; archivey reports that as `EncryptionError`, and a RAR5 password check usually rejects a wrong password before `unar` runs at all. Archivey refuses these reads with `UnsupportedFeatureError` before `unar` runs, because `unar` gets them wrong, sometimes with a success exit: - encrypted data in a RAR 2.x-4.x archive, except an uncompressed encrypted member in one part, which archivey decrypts itself. RAR5 encryption is read. - in a solid RAR 2.x-4.x archive, every compressed member after an encrypted one in the same solid stream, when opened on its own. A `stream_members()` pass over a solid RAR 2.x-4.x archive with encrypted headers or any encrypted member refuses every member it reaches, even those before the encrypted one; - a password that is not ASCII; - a multi-volume RAR5 set with encrypted headers (Homebrew's `unar` 1.10.8 returns nothing for it); - in a solid RAR5 archive, a member with data after an empty file, a directory or a link; - a member compressed with the RAR 1.5 algorithm; - an old-style volume set (`.rar`, `.r00`, ...) of more than 901 volumes: `unar` stops at `.z99`; - a multi-volume set with data before the first volume (an SFX stub). A single file with such a prefix is copied to a temporary file first; - in `stream_members()` over a solid archive that holds one of the members above, any readable member past the 4000th: that pass names each member it reads on the `unar` command line, which has a size limit. Such a member still opens on its own. With `"none"`, archivey runs no program. Listing still works, and so does reading an uncompressed member, also when it is split across volumes or sits in a solid archive, or when it is encrypted and in one part (that one needs `cryptography`, and raises `PackageNotInstalledError` without it). A compressed member, an encrypted one split across volumes, or a split one with a volume missing raises `UnsupportedFeatureError` before anything runs, and `archive.cost.notes` says so at open. ### Names unrar cannot select `unrar` is told which member to write with an include *mask* built from its name. Some names make that mask select the wrong members, or none: - **`*` or `?` in a name** also matches the member's siblings. Names like this are almost always constructed, so reading such a member raises `UnsupportedFeatureError`. On an archive that is not solid, the extra decode would also be unbounded: `ExtractionLimits` apply to extraction, not to `open()` or `read()`, and the error says how many bytes it would have cost. On a solid archive those bytes are already part of `AccessCost.SOLID`, so the error names the flag instead. Setting `ArchiveyConfig.rar_allow_glob_member_concatenation=True` reads it anyway. Two cases are not affected: a name that matches no other member (an accidental `report*.pdf` beside `report1.pdf`) reads with no flag, and a solid `stream_members()` pass reads everything, because it builds no mask at all. - **`*` or `?` in a directory part of the name, or a backslash in it:** `unrar` cannot read the member, and the flag does not change that. The one exception is a backslash in a RAR5 name written on Windows, which `unrar` on Linux and macOS reads as `_`. - **Several members under one name** (two members with the same name, or two names `unrar` reads the same way): `unrar` writes every member a name selects, in archive order, and archivey skips to the one you asked for, so each read returns that member's own bytes. A RAR5 name that is not valid UTF-8 is cut by `unrar` at its first invalid byte, and is read through that shorter name the same way. - **A name `unrar` reads as empty** raises `UnsupportedFeatureError`, and so does a name `unrar` reads but that cannot be handed back to it as a mask. - **A name ending in `.`, or with a `.` before a `\`:** `unrar` up to 7.30 beta 1 lets the mask `notes.` also select a member named `notes`, and 7.30 beta 2 does not. When that changes which members come before the one you asked for, archivey asks the `unrar` on `PATH` once, with a small test archive, which way it matches. If it gives no usable answer, the read raises `UnsupportedFeatureError`. - **A non-ASCII name in an older RAR (1.5-4) archive on a system with no UTF-8 locale:** `unrar` reads such a name through the locale, so archivey cannot tell which bytes `unrar` sends before any later member, and a read of a later member raises `UnsupportedFeatureError`. - **An unpaired surrogate in a RAR 1.5-4 name:** archivey selects the member with `?` in place of each such unit, and when that pattern also matches an earlier member, the read raises `UnsupportedFeatureError` unless you set `rar_allow_glob_member_concatenation`. `unar` selects members by position, so none of these apply to it. Under the default `"auto"`, archivey reads such a member with `unar` when one it accepts is installed. It does not use `unar` when the read needs a password (`unar` takes it on its command line), when the archive is a stream or has data before the RAR (`unar` would need its own copy of it), or when `unar` is known to misread the archive. `rar_decompressor="unar"` reads every member by position. ### Damage and endings - **A RAR5 archive cut exactly between two blocks** is reported, not raised. RAR5 ends each volume with an end-of-archive block, so archivey lists the members before the cut and then reports `ARCHIVE_EOF_MARKER_MISSING` (`expected_marker="end_of_archive_block"`), which `STRICT` raises. RAR 1.5-4 archives may lack the end block, so a cut between their blocks lists as complete. - **A cut inside a member's data or a header** lists the members before the cut, and the listing raises `TruncatedError`. - **A damaged end-of-archive block keeps the listing.** When the block after the last member fails its header checksum, every member is listed and reads normally, and archivey reports `ARCHIVE_EOF_MARKER_MISSING` with `observed_kind="nonzero"` after them, which `STRICT` raises. This is what `unrar t` does: each member tests OK, then it reports one error. A damaged header counts as the end block only if it has an end block's shape and nothing but zeros follows it within 1 MiB (`rar` pads a volume cut at exactly the `-v` size with zeros). The damaged block's next-volume flag is not trusted, so a volume set goes on to the next volume only when a member's own header says its data continues there. - **A damaged member header lists the members before it.** The members before it are listed and read normally, and then the listing raises `CorruptionError`. No later member of that volume is listed: the damaged header's size field cannot be trusted, so archivey does not know where the next header starts. `unrar` searches on and lists them. In a volume set, a member before the damage whose data continues is still followed into the next volume, whose members are listed before the error. A damaged main header raises `CorruptionError` at open. - **Bytes after the end-of-archive block** are `ARCHIVE_TRAILING_DATA` with `format="rar"`, once for each volume that has them. Zero padding passes, because `rar` pads volumes with zeros. `unrar` says nothing about such bytes; 7-Zip warns "There are data after the end of archive", for zeros too. - **A volume set with a volume missing lists what it has**, whether the missing volume is the first, one in the middle, or the last. The members whose headers are in the volumes present are listed, the set opens from any of them, and members wholly inside them read normally. A member with data in the missing volume raises `TruncatedError` when read, and so does every member past the gap in a solid archive. The listing then ends with `TruncatedError` naming the missing volumes, as for a cut file and as `unrar t` does. A later volume opened on its own by path is read the same way, as a set missing the rest. Opened as a stream, with no name to number it, it is refused with "Need first volume". - **With encrypted headers**, telling damage from a wrong password needs the password proven first: by RAR5's password check value, or in RAR 1.5-4 by an encrypted header whose checksum matches, which in a volume set covers the later volumes too. Until then, a cut past a header's first 16-byte block, or a header whose checksum does not match, looks the same as a wrong password, so it raises `EncryptionError` ("wrong password?") at open and lists nothing, even with the right password. That happens inside the first encrypted header of a RAR 1.5-4 archive or volume set, and inside any header of a RAR5 archive whose encryption record has no password check value. Once the password is proven, damage is reported as above, and other damage to an encrypted header is `CorruptionError`. ### Encryption - **Password lists:** RAR5 records a password check for each member, so a list is tried in order and the matching password is used. RAR 3-4 records none, so with several passwords archivey judges each by decoding the member: up to 64 KiB of `unrar` output, which a wrong password usually fails, and, when several passwords get that far, the whole member against its CRC. An uncompressed member in one part is checked by decrypting it in archivey and comparing its CRC, when `cryptography` is installed. The order of the list does not matter for the result, but every wrong password before the right one costs a decode, so putting the likely one first is faster. - **Partial reads of RAR 3-4 encrypted data** report `ENCRYPTED_MEMBER_UNVERIFIED`. With no password check, only the member's CRC at its end catches a wrong password, and a wrong password can still produce bytes before that: always for an uncompressed member, and for some wrong passwords on a compressed one. Closing the stream before the end, or after a seek, skips the CRC. RAR5 members and archives with encrypted headers have checked the password already and never report it. - RAR5 members with the HASHMAC flag check tweaked digests, as `unrar`'s `ConvertHashToMAC` does, when a password is available. The tweaked values are not in `member.hashes`. - **Encrypted old-style comments are not decoded.** A RAR 1.5 or 2.x comment block with its password or salt flag set gives `comment` as `None`. No available tool writes such a comment, so there is nothing to test a decoder against. ### Names and comments - RAR5 stores names as UTF-8, and RAR 1.5-4 usually as UTF-16 beside an 8-bit copy. A RAR 1.5-4 name with only the 8-bit bytes does not say which code page they are in. Archivey tries UTF-8 first. When the bytes are not valid UTF-8, it decodes them with `encoding=` when you pass one, and otherwise with cp437 for a member written on DOS or Windows (WinRAR writes the OEM code page) and windows-1252 for one written elsewhere. `raw_name` is always the stored bytes. `encoding=` has no effect on a RAR5 name. - A RAR 1.5-4 UTF-16 name can hold an [unpaired surrogate](#names-with-an-unpaired-surrogate). `unrar` 7.00 on Linux extracts such a name cut at that unit, so `hi\ud800.txt` becomes `hi`. - A RAR 1.5-4 comment is 8-bit text that does not say which code page it is in. Archivey reads it up to the first NUL, as UTF-8 if it is valid, otherwise with the `encoding=` you passed, and otherwise as windows-1252. A byte that code page does not define stays as a surrogate escape, as in a name. A RAR 2.9-4 comment flagged as Unicode is UTF-16LE, and an unpaired surrogate in it stays, as in a name. A RAR5 comment is UTF-8, and a byte in it that is not valid UTF-8 stays as a surrogate escape. ### Member types and fields - **File versions (`rar -ver`):** older revisions list as names like `path;1`, with `extra["rar.file_version"]` and `is_current=False`; the live path is `is_current=True`. Extraction skips the older revisions by default. - **Links and file copies.** A RAR5 hard link (`rar -oh`) is a `HARDLINK`. A RAR5 file copy (`rar -oi`, which `unrar` lists as "File reference") stores a duplicate file once and names the earlier member that holds the bytes. It is a `FILE` with `extra["is_file_copy"] = True`; `link_target` is the source's stored path and `link_target_member` the source member. Reading it gives the source's bytes, and extraction writes an independent file, as `unrar` does, by copying the source's file when it has just written it. A copy whose source is not an earlier file member raises `LinkTargetNotFoundError` when read. `stream_members(file_copy_streams=False)` yields `None` for each copy; its bytes and digests are its source's. - **RAR 2.9-4 symlinks** store their target as member data. When that data is uncompressed and encrypted, archivey decrypts it while listing if you gave the password and `cryptography` is installed, and checks it against its CRC. Without either, `link_target` is unset, and a `SYMLINK_TARGET_UNAVAILABLE` diagnostic says why. - **Compression:** method M0 is `STORED`. M1 to M5 are `CompressionAlgorithm.RAR` with `level` 1 to - Any other method byte stays `UNKNOWN`, with no `level`. The unpack version is in `extra["rar.extract_version"]` on every member whose file header records one, stored members included: RAR 3 copies the `UNP_VER` byte as stored, and RAR5 reports `50`. - **A compression version RAR does not know:** a RAR5 member whose compression version is newer than RAR 7's, or a RAR 1.5-4 member whose unpack version is outside 13-29, lists normally and raises `UnsupportedFeatureError` when read, where `unrar` says "Unknown method". An uncompressed member reads whatever version it declares. - A device, FIFO or socket from a Unix host follows [the shared rule](#device-files-fifos-and-sockets). `rar` itself skips such files when archiving. - **A directory entry that declares data** (`rar` never writes one) is reported with `MEMBER_DIRECTORY_DATA_IGNORED`, and extraction creates the directory, as `unrar` does. `read()`, and the stream `stream_members()` yields for the directory, return the bytes when they are stored and not encrypted. Compressed or encrypted directory data raises `UnsupportedFeatureError` (from the stream, on its first read), because `unrar` writes nothing for a directory entry. ## ISO 9660 - **Needs:** `[recommended]` (`pycdlib` 1.22 or later). - **Listing:** from the image's directory records. The namespace is chosen automatically, Rock Ridge, then Joliet, then plain ISO 9660, and reported in `ArchiveInfo.extra["iso.namespace"]`. - **Reading members:** direct. With `concurrent_operations=True`, reads share one file handle and take turns on it, as in TAR. - **From a pipe:** no; an ISO needs a source that can seek. - `ArchiveInfo.format_version` is `None`: ISO 9660 records no interchange level. - `ArchiveInfo.comment` is the volume identifier (`-V`), decoded as a plain ISO 9660 name is: UTF-8 first, then `encoding=`, then escaped. ### Names - Rock Ridge and plain ISO 9660 names, and Rock Ridge link targets, decode as UTF-8 first. Bytes that are not valid UTF-8 decode with `encoding=` when you pass one. Without it, a Rock Ridge name takes the Joliet name of the same file or directory when the image has a Joliet tree and the two line up, with a `member_name_encoding_inferred` diagnostic, and is escaped otherwise ([Names in other encodings](https://archivey-dev.github.io/archivey/opening/index.md#names-in-other-encodings)). - Joliet names are UTF-16 and ignore `encoding=`. They can hold an [unpaired surrogate](#names-with-an-unpaired-surrogate). - Plain ISO 9660 file names lose their `;N` version suffix (and the `.` of an empty extension), and `extra["iso.version"]` keeps the number. When a directory holds several versions of one name, the highest takes the bare name, and the others list under their stored identifier (`FOO.;1`) with `is_current=False`, the same shape as RAR file versions. Plain directory names have no version and keep any `;N`. - Two files stored with the same identifier both list, the later one current, as in ZIP and TAR. That includes a file and its associated file (such as the resource fork on a Mac hybrid image), which list as two members with one name and nothing to tell the fork apart. Entries in a directory list in the order of their records on the disc. ### Contents - **zisofs** (Rock Ridge transparent compression: `mkzftree` with `genisoimage -z`, or `xorriso -set_filter_r --zisofs`) reads. The member lists the size its data decodes to, with `compression=(CompressionMethod(algo=DEFLATE),)`, and reads decoded, seeking by block. zisofs2 (`xorriso -zisofs version_2=on -set_filter_r --zisofs /`, under the `ZF` or the `Z2` tag), and a zisofs entry too short to parse, list with `CompressionAlgorithm.UNKNOWN` and refuse to read with `UnsupportedFeatureError`. - A file of 4 GiB or more, stored in several extents, lists and reads as one member, wherever its extents sit in the image. Extents that overlap each other are refused with `CorruptionError`. - A bootable image lists its El Torito boot catalog (`boot.catalog`, `BOOT.CAT`) as an ordinary file, with the catalog's bytes as its data, as a mounted image shows it. - A Rock Ridge device node, FIFO or socket over an empty extent follows [the shared rule](#device-files-fifos-and-sockets); over a non-empty extent it is a `FILE`. - The `rr_moved` directory that holds relocated deep subtrees is not listed; those subtrees appear at their logical place. - `import archivey` wraps pycdlib's Rock Ridge parser, but the wrapper acts only while archivey itself opens an image, so other code using pycdlib sees no change. Inside archivey, a System Use entry of a type pycdlib does not know is skipped, as the SUSP specification says, instead of failing the whole image. A malformed entry ends that record's Rock Ridge data: the member lists from the entries before it, with a `MEMBER_HEADER_RECORD_SKIPPED` diagnostic, and a symlink cut this way lists with `link_target` unset and a `SYMLINK_TARGET_UNAVAILABLE` diagnostic. genisoimage writes such an entry for a long symlink target (from about 400 bytes with genisoimage 1.1.11). ### Damage and endings - An image whose directory tree loops back on itself, in any of its trees (ISO 9660, Joliet or UDF), is refused at open with `CorruptionError`, even when the tree archivey lists from is intact. pycdlib finds the loop. No tool writes such an image; a damaged or crafted one can. - A truncated image opens as long as its directories survive, and lists the sizes its records declare. A file the cut reaches reads the bytes that survive and then raises `TruncatedError`; files before the cut read normally. A file whose declared length cannot be recovered lists with `size` set to `None`. A cut that reaches the volume descriptors, path tables or directories makes `open_archive` raise `TruncatedError`. - An image with UDF does not open once it is truncated at all, even when the files before the cut survive: `open_archive` raises `TruncatedError`, because the UDF anchor that pycdlib reads is in the image's last block. - **Bytes after the image's end** are `ARCHIVE_TRAILING_DATA` with `format="iso"`. The end counts the partitions the image's MBR lists, and those of its GPT when the GPT's checksums match, so a hybrid installer image, whose EFI partition and GPT backup header sit after the ISO 9660 volume space, opens clean (7-Zip warns "There are data after the end of archive" on those images). A GPT damaged in transfer is not trusted, so the partition it lists is reported. - Raw CD sector images (the `.bin` of a `.bin`/`.cue` pair) are recognised and refused with `UnsupportedFeatureError` naming the sector layout. Convert one to a plain `.iso` first (for example with `bchunk` or `bin2iso`). ## Disk images A UDIF image (`.dmg`) is recognised by its `koly` block, in its last 512 bytes, or its first 12 when an old image puts the block at the start, and refused with `UnsupportedFeatureError`. A compressed image stores its blocks as zlib, bzip2 or xz, so without this check, opening one would extract the first block and treat the rest as trailing bytes. To get at the files, convert or mount the image first (for example with `7z x`, `dmg2img`, or `hdiutil attach` on macOS). - An uncompressed image whose disk is an ISO 9660 filesystem is read as that ISO: `CD001` at byte 32 769 is checked first. - A pipe is not rewound to read the end. A short image from a pipe is still refused, because detection has already read through to its end. A longer one whose first block is bzip2 or xz opens as that stream. One whose first block is zlib fails with `FormatDetectionError`, because no content probe runs on a source with no name. ## Directory - **Needs:** nothing. A filesystem tree is read as a pseudo-archive, with the same API, for tests and for code that handles directories and archives alike. - **Listing:** walks the tree. - **Reading members:** the same defaults as an archive: forward-only streams and one live stream, until you pass `seekable_members=True` or `concurrent_operations=True`. - Hardlinks list as a tar lists them. When several names inside the root share one file, the first name the walk reaches is a `FILE`, and each later name is a `HARDLINK` to it, with `size` `None`. The walk visits a directory's files by name, then its subdirectories, so which name keeps the data is fixed. With `streaming=True`, extracting a later name whose first name a filter left out fails, as for a streamed tar. - Names come from the filesystem, so `encoding=` has no effect, and passing it reports `ENCODING_ARGUMENT_UNUSED`. ## Single-file compressors - **Needs:** gzip, bzip2, xz, lzip, LZMA Alone, zlib and `.Z` are in core (`.Z` with archivey's own LZW decoder). Zstandard is in core on Python 3.14 and later, and needs `[recommended]` before. LZ4 and Brotli need `[recommended]`. - **Listing:** one synthetic member, named from the source path, or `data` for a source with no name. `encoding=` has no effect, and passing it reports `ENCODING_ARGUMENT_UNUSED`. - **Reading:** forward only, until you pass `seekable_members=True` ([Seeking inside a member](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member)). `archivey.open_stream(...)` follows the same rule: not seekable unless `seekable=True`. `.Z` decodes forward from a pipe, and its CLEAR codes give seek points when the source can seek. - **From a pipe:** yes, with the waits [below](#from-a-pipe-held-open). ### Size and stored values - **`size` arrives with the first full read** when listing left it `None`: always for `.gz`, `.bz2`, `.zz`, `.br`, `.zst`, `.lz4` and `.Z` (a zstd or LZ4 frame's content-size field is not used), and for `.xz`, `.lz` and `.lzma` when their size is absent or out of reach (an `.xz` from a pipe, an `.lzma` written with an unknown size). A read that reaches the end, having returned every byte from the start, which the decoder finishes without error, sets `member.size` on the member `members()` returned. Seeking back is fine, and so is seeking forward on these codecs: the seek decodes the bytes it skips anyway, and they count. A read that stops early or raises leaves it `None`, as does a copy made with `replace()`. The filled size is only as good as the codec's end check: for a `.Z` (no end marker or checksum) or an accelerated `.gz` (below), a cut the decoder cannot see fills in the size of the cut data. `member_state_final` reads `False` until the size is filled in, a `streaming=True` pass ends, or the reader closes. - **`.xz` size and `.lz` CRC-32** come from the stream's index or the lzip trailer whenever the source can seek: a file path and an in-memory stream both qualify, a pipe does not. `seekable_members=True` makes no difference, since reading the trailer is a bounded backward peek at open. A `.xz` or `.lz` opened from another archive's member stream reads neither, seekable or not: it never reports the CRC-32, and its size arrives only from a full read. - `.gz` may expose `extra["gzip.original_filename"]` when the header carries a name (`FNAME`). - `.gz` exposes `extra["gzip.isize"]`, the trailer's size field, when the file's end is cheap to reach (a path or an in-memory stream; not a pipe or another archive's member). It is the size modulo 2^32 only for a file of one gzip member with nothing after it. A file of several members stores the last member's size there (a bgzip file stores 0), so it never becomes `member.size`. - `.gz` has no `member.hashes` entry, before or after a read. The trailer's CRC-32 covers the whole member only when the file holds one gzip member, and proving that at open would mean reading the whole file. The decoder still checks every gzip member's CRC as it reads. - `.bz2`, `.xz`, zlib, Brotli and `.Z` have no whole-member digest that can be read without decoding. zlib's Adler-32 is still checked by the decompressor on read, but it is not put in `member.hashes`, because with possible concatenation or trailing bytes the end of the stream cannot be found from its fields. ### Checks on the data - **No checksum:** Brotli, `.Z` and LZMA Alone files have none, so damaged data can decode to wrong bytes with no error. A `.zst` frame has one only when its writer adds it, as a modern `.lz4` frame usually does; archivey checks it when it is there. The legacy LZ4 format (`lz4 -l`, used for Linux kernel images) reads as `.lz4` and has no checksum either. Archivey does not report a diagnostic on each such file, because there is no check to skip. - **No end mark:** a `.Z` file, and a legacy LZ4 file, can be cut between two blocks and read short with no error. `.Z` truncation is best effort: leftover bits that are not zero after the last complete code raise `TruncatedError` on the next `read()` after the bytes that could be decoded; a cut that leaves no such bits goes unnoticed. - **`open_archive` decodes the first byte** of a seekable source, so a file that is not the codec its name or detection claims (a `.gz` full of zeros, an empty `.bz2`) raises `CorruptionError` or `TruncatedError` from `open_archive` rather than from the first read. A valid empty stream still opens and reads as `b""`. The check goes one byte deep: damage further in still fails on the read. It costs one decoded block, which is noticeable only for `.bz2` (up to 900 KB of output; about 30 ms on incompressible data). A pipe is not checked at open, because the check would consume bytes of its one pass; it fails on the first read instead. - **A header the format's own tool calls unsupported** raises `UnsupportedFeatureError`, not `CorruptionError`: a gzip member with a method other than Deflate or a reserved flag bit set, an LZ4 frame in a version other than `01`, a zstd frame that needs a dictionary, and a `.Z` file with a code width over 16 bits. A damaged byte in one of those fields raises the same error, because nothing tells the two apart; the message says a damaged header reads the same way ([Damaged archives](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#damaged-archives)). - `.lz` is read in lzip format version 1, which every lzip since 1.0 writes. A member in version 0 (lzip before 1.0) or a later version raises `UnsupportedFeatureError`, wherever it is in the file: a member that starts with the `LZIP` magic is never skipped as trailing bytes. ### Bytes after the compressed stream Bytes after the stream (a signature or checksum appended to a download, a tool that pads its output) do not stop the read. For gzip, zlib, bzip2, xz, lzip, LZMA Alone, Zstandard, LZ4 and Brotli, archivey returns the whole payload, then reports one `ARCHIVE_TRAILING_DATA` whose `observed_bytes` is the offset of the first appended byte ([Bytes after the end of an archive](#bytes-after-the-end-of-an-archive)). Under `STRICT`, the read that reaches it raises `DiagnosticRaisedError`. The same check runs on a compressed tar. - A second stream of the same codec (a concatenated `.gz`, `.bz2`, `.lzma`, `.zst` or `.lz4`) is more data, not trailing bytes. - For every codec but `.xz`, zero bytes are padding only at the end of the file: a stream after them is reported as trailing bytes and not read, as GNU `gzip`, `bzip2`, `zstd` and `lz4` do. `.xz` defines padding between streams and reads past it. - For `.xz`, `.lz`, `.zst`, `.lz4` and `.bz2`, bytes after a stream that hold at least half of the codec's magic, but not all of it, are a later stream with a damaged header: the read raises `CorruptionError` rather than return the first stream alone. - `.xz` and `.lz` keep their size and seeks when the appended bytes are within 1 MiB, unless they are crafted to hold thousands of fake end markers. Further out, the index is not found and the size reads as unknown. A damaged end marker on the last of several `.xz` streams or `.lz` members is corruption, not appended bytes: the size reads as unknown, and the read or seek that reaches the damage raises `CorruptionError`. - Brotli has no end marker the library reports, so archivey finds the end by decoding the source again, which needs a source that can seek. From a pipe, bytes after a Brotli stream raise `CorruptionError`, and how much data comes before that error depends on how the bytes arrive. - `.Z` is not covered: it has no end marker, so appended bytes decode as more data. ### From a pipe held open A compressed file or a compressed tar from a pipe that the sender keeps open is decompressed from what has arrived: a compressed tar opens and yields its members, and a read returns the data whose compressed bytes have arrived. It waits for the sender to close the pipe in these places: - `open_archive` of a short file, because archivey reads a fixed amount from the start before it decides: - Under 258 bytes. A tar header starts at offset 257, and until that byte has arrived the file could still be a tar, even a `.gz` whose first two bytes say gzip. A few files up to 261 bytes wait too, when their bytes from offset 257 spell the start of a tar header. - A `.gz` under 512 bytes, whose first 512 bytes are read for the stored name. - A `.bz2` or `.xz` under 4 KiB, and a `.tar.bz2` or `.tar.xz` too. These are checked for a disk image, which needs the length of the file. - A file that decompresses to under 512 bytes, the size of a tar header. - A read past the last byte of data, and the end of a compressed tar's listing, since another stream or trailing bytes could still follow. bzip2 is the exception to decoding what has arrived: it decompresses nothing until a whole block has arrived (up to 900 KB of original data), at any file size, so its reads wait for the end of each block. `open_archive` waits for the end of the first block of a `.tar.bz2`, which it decodes for the first header, and of a `.bz2` when it detects the format, which decodes the start of the file; with `format=`, a `.bz2` opens at once. Close the sending side when the file is done. ### Limits and accelerators - A `.zst` frame declares its window, and the decoder keeps that much memory. `DecoderLimits.max_decoder_memory` (2 GiB by default) caps it, so the 2 GiB window of `zstd --long=31` reading standard input reads. A window over the cap raises `ResourceLimitError`. The cap is rounded down to a power of two for zstd. A window over 2 GiB is beyond what libzstd decodes at any setting, and raises `UnsupportedFeatureError`. Detection reads a sample of the stream with no cap, so a frame declaring 2 GiB has that much address space reserved while `open_archive` detects it, whatever the cap. - With the `[seekable]` rapidgzip accelerator on a seekable `.gz` (by default from 16 MiB, with `seekable_members=True`, or on Linux and macOS without it when the file compresses 3:1 or less), archivey adds its own checks, because rapidgzip can end a cut file with no error. If rapidgzip returns nothing, the standard library reads the file instead. Otherwise the trailer at the end of the file is checked: in a file of one gzip member, against the checksum and length of all the data read; in a file of several members, against the end of the data. For that, archivey saves the checksum every 4 MiB as it reads and reads back at most 4 MiB from rapidgzip per check, whatever the size of the last member. After a seek that skipped data, the check still runs, and also reads back the skipped part of the last member; past about 64 MiB of that, the standard library reads the whole file again instead, which gives the right result but takes longer. A wrong length in a member before the last is not caught, but every member's checksum still is, so the data is right. A cut file built on purpose can still read short with no error, if its last eight bytes are made to match the end of the data. So when you need certainty, set `ArchiveyConfig(use_rapidgzip=AcceleratorMode.OFF)`. This applies to a bare `.gz` or `open_stream` (and bare zlib or raw Deflate), not to ZIP or 7z members, whose stored CRC and size catch short or wrong data. - The `[seekable]` accelerator's bzip2 decoder reads input that is not bzip2 as an empty stream. When it yields nothing, archivey decodes the source again with the standard library, so a damaged `.bz2` raises the same error whether or not `seekable_members=True` engaged the accelerator. ## Detection - **Strongest signal first**, and the file name last; wrong extensions are expected. In order: exact magic in the first 4 KiB, then an SFX scan behind an executable stub, then exact magic further in (ISO 9660's `CD001` at 32 769, on one extended peek that a source too small for it never pays for), then the 512-byte `koly` block at the end of a seekable source ([Disk images](#disk-images)), then a TAR header with no magic ([TAR](#tar-and-compressed-tar)), then content probes for the formats with no magic (next item), then the extension. A bzip2 or xz header that is the first block of a disk image loses to its `koly` block. A step that matches nothing falls through to the next; nothing is rejected for failing an earlier one. When only the extension matched and the decode then fails, the error has `format_unconfirmed=True`, and archivey reports `EXTENSION_FORMAT_UNCONFIRMED`. - **Content probes run only for a matching name by default.** LZMA Alone, zlib and Brotli have no magic, so a trial decode of the first bytes (a content probe) is the only thing that recognises them, and ordinary binary files sometimes pass one. By default `open_archive` and `detect_format` run a probe only when the name ends in one of that format's extensions: `.lzma` or `.tlz` for LZMA Alone, `.zz` or `.zlib` for zlib, `.br` or `.brotli` for Brotli, and the `.tar.` form of each except `.tlz`. A matching probe confirms the name and still finds a TAR inside. A source with no name, or another extension, runs no probe: a raw stream of these formats with no name raises `FormatDetectionError`, and one named for another format gets that format's guess. To read such sources, give the file its format's extension, pass `format=`, use `open_stream()` (which always runs every probe), or set `ArchiveyConfig(always_probe_content=True)` (`--probe-content` on the command line). No probe runs on a file that starts with a valid executable header or an OLE (old Office) header, even when it is named for one of these formats. - **Brotli** detection adds framing checks to the probe when the source length is known (paths, `BytesIO`, and short non-seekable peeks): a first meta-block that declares more bytes than the source holds is rejected; when the source is 64 KiB or less, the whole of it is decoded, and a stream that still wants more input after a declared output drain is rejected; and a bounded walk of self-describing meta-blocks rejects a later link that overruns, or a declared end with bytes after it. On a non-seekable stream of unknown length those checks are skipped. When the decoded start is a TAR header, the format is `tar.br` with confidence `PROBABLE`. Otherwise, without a Brotli extension, the confidence is `PROBABLE` when the first meta-block is compressed and `GUESS` when it is not, and a later decode failure sets `ArchiveyError.format_unconfirmed` and reports `PROBE_FORMAT_UNCONFIRMED`: the bytes may never have been Brotli, and some made-up output may already have been returned before the error. A `tar.br` found this way is not flagged. With a Brotli extension (`.br`, `.brotli`, `.tar.br`) it is `PROBABLE`. - **zstd skippable frames** (a magic from `0x184D2A50` to `0x184D2A5F` plus a declared payload size) may come before the first real frame, so detection walks past them by their declared sizes within the bytes it peeked, and matches the regular frame behind. Skippable frames alone are not a zstd claim: there is nothing to open. - **zlib** is recognised by the RFC 1950 header rules (`CM == 8`, `CINFO <= 7`, and the mod-31 check, `FDICT` included) rather than a list of common headers, so all seven legal window sizes are recognised. A preset dictionary archivey does not hold fails the decode, and detection moves on. - **LZMA Alone** accepts any 32-bit dictionary size, zero included: the format allows every value, and decoders round anything below 4 KiB up to 4 KiB. It does not claim a header declaring an uncompressed size of exactly zero: that stream holds no payload, and 18 zero bytes are a valid empty one, so zero padding would otherwise be detected as `.lzma`. A real size and the all-ones "unknown" value are both accepted, and an empty `.lzma` still opens through its extension. - **Self-extracting (SFX) stubs** are detected when the archive sits behind an executable header (Windows `MZ`/PE, Linux ELF, or a macOS Mach-O header that parses) or a `#!` launcher line (a zipapp `.pyz`, a Spring Boot jar). The 2 MiB scan behind the stub reports where the archive starts as `payload_offset`. A launcher is text, so behind `#!` only formats with a structural check (ZIP, 7z, RAR) are searched for. Among several 7z candidates, one that ends exactly at the end of the file is preferred over an earlier one that ends short of it. - **Ties go to the earlier step, then to registration order.** Detection stops at the first step that matches, and within a step the first backend registered wins. A file that is two formats at once (a polyglot) gets one fixed answer; pass `format=` to read it as the other. - **`confidence` may change in 0.2.x**: what each step reports may be regraded later (a two-byte magic such as gzip's reported below `CERTAIN`, say). Branch on `format`, not on `confidence`. **`detected_by` is an open set**: new detection steps may add values, so handle an unknown one rather than matching every value. - `detect_format` returns the confidence and evidence on `FormatInfo`; the [API reference](https://archivey-dev.github.io/archivey/api/index.md#detection-budgets) lists what detection may read. ## What archivey does not do - **Change an archive in place.** Archivey reads archives and extracts them. It does not add, remove or replace members in an existing archive, and it doesn't write archives yet. - **Offer an async API.** The API is synchronous, and an async one is not planned before 1.0. In async code, run archivey calls in a worker thread, for example with `asyncio.to_thread`. - **Make backups.** Archivey reads the members and metadata a backup tool needs, but it is not a backup engine by itself. # Adding a format `archivey.formats` is a small set of shared structures for reading a format. A format describes itself and hands archivey its members and their bytes, and archivey does the heavy lifting: name checks, limits, links, digests, extraction, and decoding the compression around or inside the archive. The built-in ISO 9660 reader is written this way, and more built-in formats are planned on it (CPIO, ar, SquashFS, and TAR). The same structures are public, so a program or a separate package (a *plugin*) can add a format of its own. A format added this way works with `open_archive`, `detect_format`, `format=`, `format_availability`, the format lists, extraction and the `archivey` command, as the built-in formats do. The other built-in readers (ZIP, 7z, RAR, TAR, the single-file compressors and directories) are written directly against archivey's internal reader classes. That route is for formats that do not fit this model, or that need special features or more speed, and it is not public. Provisional `archivey.formats` can change in a minor release until a plugin outside archivey has used it. Pin the archivey version your plugin is tested with. A format reader reads one of two kinds of format: - **A format with an index**, where the reader can list every member first and then open any one of them, from a file or another source that can seek. ISO 9660, SquashFS, CAB and WIM are this kind. - **A format read in one pass**, a header and then the member's data, again and again to the end. It can be read from a pipe. CPIO and AR are this kind. See [Formats read in one pass](#formats-read-in-one-pass). A format read in one pass can also be read inside a compressor, such as a `.cpio.gz`, with no extra code; see [Inside a compression stream](#inside-a-compression-stream). [What is not supported yet](#what-is-not-supported-yet) lists what a format reader cannot do. ## What a format reader is A format is two classes: - A **`FormatHandler`** subclass. Its class attributes say what the format is called and how to recognise it: an `ArchiveFormat`, file extensions, and magic bytes. Its `open()` method returns a reader. - An **`IndexedReader`** subclass. It lists the members with `iter_members()`, and returns the data of one member with `open_member()`. A format read in one pass uses a `SequentialFormatHandler` and a `SequentialReader` instead, described in [Formats read in one pass](#formats-read-in-one-pass). The rest of this page applies to both kinds unless it says otherwise. The reader hands archivey members and bytes. archivey does everything after that the same way it does for a built-in format: it normalizes member names, checks them, applies the listing limits, resolves links, checks each member's size and checksum as it is read, and extracts safely. Three complete plugins outside archivey are in the repository: [`packages/archivey-squashfs`](https://github.com/archivey-dev/archivey/tree/main/packages/archivey-squashfs) for SquashFS images, a format with an index; [`packages/archivey-cpio`](https://github.com/archivey-dev/archivey/tree/main/packages/archivey-cpio) for CPIO archives and [`packages/archivey-ar`](https://github.com/archivey-dev/archivey/tree/main/packages/archivey-ar) for ar archives (static libraries), both read in one pass. None is published. ## A complete example The example format here, TOYA, is small enough to read in full. A TOYA file starts with the 8 bytes `TOYARC1\n` and a 4-byte member count. One index entry per member follows: the name, a directory flag, the size, a CRC-32 and the offset of the data. The data of all members comes after the index. All numbers are little-endian. ```python from collections.abc import Iterator from archivey import ( ArchiveFormat, ArchiveMember, HashAlgorithm, MemberType, PluginContainer, StreamFormat, ) from archivey.formats import ( FormatHandler, IndexedReader, Magic, MemberData, MemberRecord, ReaderContext, ReaderSource, register_format, ) class ToyContainer(PluginContainer): TOYA = "toya" TOYA = ArchiveFormat(ToyContainer.TOYA, StreamFormat.UNCOMPRESSED) # Where a member's data is: (offset, length). Locator = tuple[int, int] class ToyReader(IndexedReader[Locator]): def __init__(self, source: ReaderSource, context: ReaderContext) -> None: self._source = source self._context = context # read_struct() and read_exact() raise TruncatedError when the archive ends # early, where struct.unpack() would fail on the short read with an error of # its own. (self._count,) = source.read_struct(8, " Iterator[MemberRecord[Locator]]: pos = 12 for _ in range(self._count): (name_len,) = self._source.read_struct(pos, " MemberData: offset, size = locator return MemberData(self._source.view(offset, size)) class ToyHandler(FormatHandler): format = TOYA extensions = (".toya",) magic = (Magic(0, b"TOYARC1\n"),) uses_encoding = True def open(self, source: ReaderSource, context: ReaderContext) -> ToyReader: return ToyReader(source, context) register_format(ToyHandler) ``` After `register_format`, a TOYA file opens like any other archive: ```python import archivey with archivey.open_archive("photos.toya") as archive: for member in archive.members(): print(member.name, member.size) archive.extract_all("out/") ``` `detect_format("photos.toya").format` is `TOYA`, and `format="toya"` opens a TOYA file whatever its name is. ## The format's identity A format is an `ArchiveFormat`, a pair of a container and a stream. A plugin defines its container as a member of its own `PluginContainer` subclass, and uses `StreamFormat.UNCOMPRESSED` as the stream. The container's value and its member name are the format's names in `format=` and on the command line, so neither may already name a format: not a built-in one (`"zip"`, `"tar.gz"`, `"TAR_GZ"` and the others) and not another plugin's. `register_format()` refuses a plugin whose name is taken. Code that matches on `ArchiveFormat.container` needs a default case once a plugin is registered, because the set of containers is no longer only `ContainerFormat`. ## Listing members `iter_members()` yields one `MemberRecord` per member, in archive order, and must yield the same members in the same order if archivey calls it again. Each record holds an `ArchiveMember` and a locator. The member carries the public fields your format stores: at least `type`, `name` and, for a file, `size`. The name is decoded but not cleaned up. archivey normalizes it (`./a//b` becomes `a/b`) and reports `MEMBER_NAME_NORMALIZED` when that changes it, as for a built-in format. `ReaderContext.decode_name()` decodes stored bytes the way the built-in formats do: valid UTF-8 wins, then the caller's `encoding=`, then UTF-8 with no byte lost. The locator is anything your reader needs to find the member's data again: an offset, a tuple, or an object from the library you wrap. archivey keeps it and hands it back to `open_member()` without looking inside. `IndexedReader[Locator]` tells a type checker what it is, so `open_member()` gets a `tuple[int, int]` and not an `object`. Set `member_list_upfront = False` on the reader when `iter_members()` reads headers from all through the file, not from one index. archivey then reports the listing as needing a scan in `archive.cost`. ## Reading member data `open_member()` returns a `MemberData`. Its `stream` holds the member's bytes, decoded and read from the start. archivey closes the stream when the caller is done with it. The other fields say how archivey can check those bytes, as described in [What archivey checks](#what-archivey-checks). ### Where the bytes come from There are four ways to build the stream. Pick the first one that fits your format. **Stored bytes.** `ReaderSource.view(start, length)` is a stream over part of the archive with its own position, so two members open at the same time do not move each other. A format that stores members uncompressed returns a view, as TOYA does. **A compression method archivey decodes.** Hand the member's compressed bytes to `ReaderContext.decompress()` and return what it gives: ```python def open_member(self, member: ArchiveMember, locator: Locator) -> MemberData: offset, compressed_size = locator data = self._source.view(offset, compressed_size) return MemberData( self._context.decompress(data, CompressionAlgorithm.DEFLATE, size=member.size) ) ``` `decompress()` takes one of these methods: - raw Deflate and Deflate64; - bzip2; - Zstandard and LZ4 frames; - Brotli; - raw LZMA and LZMA2, with the `properties` bytes the format stores; - XZ, for `.xz` data kept inside a member, as ZIP method 95 keeps it; - `STORED`, for data kept as is, so a format with a per-member method can pass every method through the same call. It also takes a `StreamFormat` in place of a method. The data is then a whole stream with its own framing, decoded the way a file of that format is: `StreamFormat.ZLIB` for Deflate with the zlib header, and `GZIP`, `XZ`, `LZMA_ALONE`, `LZIP` and the rest. The decoding uses the caller's decoder limits, process isolation and accelerator settings. It turns the decoder's errors into archivey's own, so your `translate_exception()` never sees them. A method whose optional package is missing raises `PackageNotInstalledError`, and a method not in the list above raises `UnsupportedFeatureError`. Where the compressed data ends depends on the method: - **Zstandard and LZ4** data may be several frames, and all of them are decoded. - **XZ** data may be several xz streams, with stream padding (groups of four zero bytes) between and after them, and all of them are decoded. - **bzip2** data is one stream, and a second stream after it raises `CorruptionError`. - **LZMA and LZMA2** end at their end marker. Raw LZMA1, which may have none, then ends at `size`. - For every method, bytes left after that end raise `CorruptionError`. - A **stream format** ends where its format says. Bytes after that end are reported as `ARCHIVE_TRAILING_DATA`, as they are for a file of that format. **A file in independent blocks.** Some formats store a file as fixed-size blocks, each compressed on its own, so a reader can start at any block: SquashFS, CAB, WIM and CSO all do this. Return `block_stream(size, block_size, read_block)`, where `read_block(i)` returns block `i` decoded. archivey takes care of the rest: - it calls `read_block` when a read first needs a block, and keeps only the current one; - it handles reads that cross from one block into the next; - a seek decodes only the block it lands in, not the file up to it; - a block of the wrong length raises `CorruptionError`. `read_block` may return the stream `decompress()` gives rather than bytes. archivey then asks it for one byte more than the block's length and refuses the block if it gets it. How much the stream decodes to answer is the stream's own: a decoder that buffers decodes up to its buffer, so keep that buffer near the block size. The SquashFS package's `read_block` is short: it reads the block list as far as it needs, returns zeros for a sparse block, and decodes the rest. Its first branch returns the file's tail, which SquashFS keeps in a fragment block shared with other files: `read_block` only has to return the right bytes for each index, wherever the format stores them. **Your own decoder.** For a method archivey does not decode, wrap the view in your library's decoder, or one of your own. Such a decoder is outside archivey's decoder limits, so it must stop at the size the format allows: pass the expected output size and refuse anything longer. The SquashFS package does this for its LZ4 and LZO blocks, which can never be larger than one block. ### Seeking and short reads The stream does not need to seek. When the caller opens the archive with `seekable_members=True` and the stream cannot seek, archivey seeks for it: - a forward seek reads the bytes in between; - a backward seek calls `open_member()` again and reads forward from the start, which is slow for a large member; - when the listing gives no `size`, a seek from the end reads to the end once to find it. A view and a `block_stream()` seek by themselves, and cheaply. A `decompress()` result seeks by itself too, but a backward seek decodes again from the start, as the fallback does. A raw stream may also return fewer bytes than asked before it ends, as an `io.RawIOBase` may. One that stops each read at the end of a block is an example. archivey asks it again for the rest, and reads nothing ahead. An `io.BufferedIOBase` is taken at its word, as that class promises full reads until the end, and so are the streams archivey hands out (a view, a `decompress()` result, `block_stream()`). For a raw stream whose reads cost more than a copy, such as one call into a library per read, return `buffered_stream(stream)`, which puts an `io.BufferedReader` in front of it. Closing that also closes the stream it wraps. ### What archivey checks archivey checks every member stream against what the listing said. - **Size.** - A stream that ends before `member.size` raises `TruncatedError`, and one that runs past it raises `CorruptionError`. - A file listed with no size is read to the end of its stream, and only its checksums are checked. - To find out whether a stream runs past the size, archivey reads one more byte at the end. It skips that read for a view or a `block_stream()` no longer than the member, which cannot run past it. For another stream that cannot return more than `member.size` bytes, such as your own decoder that stops at the size, set `bounded=True` on the `MemberData`. The extra read is a measurable part of opening a small member. - **Checksums in the index.** A CRC-32 or BLAKE2sp the index stores, known when listing, goes in `member.hashes`, where callers see it. It is checked as the stream is read. A mismatch raises `CorruptionError`, and a full read with a match reaches `Verification.CONTENT`. - **Checksums you read when opening.** A checksum the member's own header stores goes in `MemberData.digests`, as a `MemberDigest` with its name and stored value. For a name archivey computes (`"crc32"`, `"blake2sp"`) that is all; for a CRC-16, a SHA-1 or any other checksum, also pass a function that makes a fresh hasher, such as `hashlib.sha1`. A hasher is any object with `update(bytes)` and `digest()`. archivey feeds it the bytes as they are read, and checks the result the same way as a CRC-32. A CRC-32 or BLAKE2sp given this way that the index does not already store is also added to `member.hashes` when the member is opened, so callers see it. For example, for a header that stores a CRC-32 and a SHA-1: ```python return MemberData( self._source.view(offset, size), digests=( MemberDigest("crc32", stored_crc), MemberDigest("sha1", stored_sha1, hashlib.sha1), ), ) ``` A name, ignoring case, appears once in `digests`, and one archivey does not compute needs a hasher; otherwise opening the member raises `ValueError`. When the index and the header both store a CRC-32, put each where it is read: archivey checks both, so a read fails when either does not match the data, rather than one replacing the other. A field the format leaves unset (a zero CRC in a header whose value comes later, say) is left out, not passed: archivey compares whatever value you give it. - **Checksums the stream checks itself.** If the stream your library returns checks its own checksum and raises when it does not match, set `checked=True` on the `MemberData`. A read to the end then counts as `Verification.CONTENT` too. Whichever way a checksum archivey has no name for gets checked, keep the stored value in `member.extra` under a key named after your format, such as `"toya.sha1"`, so callers can see it. ## Formats read in one pass A format of header, data, header, data has no index to list from: the only way to find the members is to read the archive from the start. Such a format can be read from a pipe, and a plugin for it has two other classes: - A **`SequentialFormatHandler`** subclass. It declares the format as a `FormatHandler` does, but implements `open_sequential(context)` instead of `open()`. There is no `ReaderSource`, because the archive may be a pipe. - A **`SequentialReader`** subclass. Its `walk(stream)` is a generator that reads headers from a `ForwardStream` and yields one `MemberSpan` per member. The reader parses headers and archivey owns the data. A `MemberSpan` holds the member and `data_length`, the number of bytes after the header that are the member's data, stored as they are. archivey hands those bytes to the caller as the member's stream. Before it resumes the walk, it skips whatever the caller did not read, so when `walk()` continues after its `yield`, the stream is at the end of the member's data. The walk then reads any padding and the next header: ```python import struct from collections.abc import Iterator from archivey import ArchiveMember, CorruptionError, MemberType from archivey.formats import ( ForwardStream, MemberDigest, MemberSpan, SequentialReader, ) HEADER = struct.Struct("<2scHII") # magic, type, name length, data length, CRC-32 class ToysReader(SequentialReader): link_target_in_data = True # a symlink's data is its target def __init__(self, context): self._context = context def walk(self, stream: ForwardStream) -> Iterator[MemberSpan]: while True: at = stream.offset magic, code, name_len, length, crc = stream.read_struct(HEADER) if magic != b"TS": raise CorruptionError(f"no header at offset {at}") if code == b"E": # the end-of-archive header return raw = stream.read_exact(name_len) kind = {b"F": MemberType.FILE, b"L": MemberType.SYMLINK}[code] member = ArchiveMember( type=kind, name=self._context.decode_name(raw), raw_name=raw ) digest = MemberDigest("crc32", crc.to_bytes(4, "big")) yield MemberSpan(member, length, (digest,)) stream.skip(-length % 4) # padding to four bytes ``` A real reader also refuses an unknown type code with `CorruptionError` rather than a `KeyError`. `ForwardStream` reads forward only. `read(n)` returns fewer bytes only at the end of the archive; `read_exact(n)`, `read_struct(fmt)` and `skip(n)` raise `TruncatedError` there instead. `offset` is how far the archive has been read, for messages. A walk never reads a member's data itself. archivey takes the first step of the walk when it opens the archive, so a file that is not your format fails to open, as a built-in one does. The end of the archive is `walk()`'s to check: an end marker, a trailer, or bytes after it. Raise `CorruptionError` or `TruncatedError` when they are wrong, and archivey keeps the members the walk yielded before it. What archivey does with the spans: - **Pipes and passes.** Opened with `streaming=True`, the walk is the reader's one forward pass, so `stream_members()`, iteration and `extract_all()` each read the archive once, from a pipe too. The listing limits count members as they arrive. - **Random access on a file.** Opened from a file or a seekable stream without `streaming=True`, archivey walks the archive once to list it and records where each member's data starts. `open()` and `read()` then read a member directly, in any order. - **Sizes.** A `FILE`'s `size` is its `data_length`. Leave `size` unset and archivey fills it in; a different one raises `ValueError`, since the data is stored as it is. - **Checksums.** `MemberSpan.digests` are checked as `MemberData.digests` are (see [What archivey checks](#what-archivey-checks)). A `HashAlgorithm` digest, such as the CRC-32 above, is also listed in `ArchiveMember.hashes`. - **Link targets stored as data.** With `link_target_in_data = True`, a symlink's data is its target. archivey reads it while walking, up to 4096 bytes, as UTF-8. A longer one is reported with `SYMLINK_TARGET_UNAVAILABLE` and left unset. - **Other members' data.** The data of a member that is not a file, a directory or such a symlink is skipped. A directory with data keeps it as its `size`: a read delivers the bytes, extraction creates the directory, and the listing reports `MEMBER_DIRECTORY_DATA_IGNORED`, as for a built-in format. `SequentialFormatHandler` takes the same class attributes as `FormatHandler`. The format's `archive.info.cost` says the listing needs a scan, and `format_availability()` reports that it can be read from a source that cannot seek. ### Inside a compression stream A format read in one pass is often compressed whole, as a `.tar.gz` is: `.cpio.gz`, `.a.xz`. archivey reads these with no extra code. Registering the format also registers it inside each compression stream archivey decodes, as `ArchiveFormat(, StreamFormat.GZIP)` and so on, and adds your extensions followed by the compressor's (`.cpio` gives `.cpio.gz`, `.cpio.xz`, `.cpio.zst`, …), as `format=` spellings too. archivey decodes the stream and `walk()` reads the decoded bytes, from a file or a pipe. - **Detection.** A compressed file is first recognised as one gzip or xz member. archivey then decodes its first 512 bytes, the same bytes it decodes to find a TAR inside, and looks for your `magic` in them. So a sequential format's magic must end within the first 512 bytes, and registration refuses one that does not. A format found only by its `probe` is not found inside a compressor: name it with its extension or `format=`. - **Cost.** One compression stream holds every member, so the archive is solid: `archive.info.is_solid` is `True`, the listing needs decompression, and with random access, reading a member before the last one read decodes again from the start. On a reader that has not listed the archive yet, `stream_members()` and `extract_all()` read each member's data as the walk reaches it, so they decode the archive once, as for a `.tar.gz`. Once the archive is listed (`members()` first), a pass decodes it a second time, in one forward sweep. - **Limits.** The decoder limits and the decompression-ratio guard apply as for a `.tar.gz`, and bytes after the compressed stream are reported with `ARCHIVE_TRAILING_DATA`. ### Compression that starts part-way Some formats store headers and then compress the rest: an RPM package is a few headers followed by a gzip, xz or zstd stream holding a CPIO archive. A walk reads the headers, then calls `stream.decompress(StreamFormat.XZ)` and reads on from the stream it returns: ```python def walk(self, stream: ForwardStream) -> Iterator[MemberSpan]: read_rpm_headers(stream) stream = stream.decompress(payload_format) yield from walk_cpio(stream) ``` The returned stream holds the decoded bytes, and its `offset` counts them from 0. The members the walk yields after the call are read from those bytes, from a pipe too, with random access to them on a file. archivey decodes with its own codecs, under the caller's decoder limits and the decompression-ratio guard. The stream the walk called `decompress()` on refuses further reads. archivey cannot know the archive holds a compression stream before the walk reaches it, so have `archive_info()` return `ReaderArchiveInfo(is_solid=True)`: the cost receipt then says members are read through one stream, and that listing needs decompression, from the moment the archive opens. The compressed stream is taken to run to the end of the archive: bytes after its end are reported with `ARCHIVE_TRAILING_DATA`, not refused. A pass over such an archive decodes it once, as one inside a compression stream does (see **Cost** above). ### Data a library decodes Some formats compress each member on its own, with a method only a library decodes (LHA, ARJ), or are read through a library that decodes as it walks (a libarchive wrapper). There are no stored bytes to describe with `data_length`. Hand archivey the decoded data instead, as a stream in `MemberSpan.data`, and leave `data_length` at 0: ```python def walk(self, stream: ForwardStream) -> Iterator[MemberSpan]: while (header := read_header(stream)) is not None: member = ArchiveMember(type=MemberType.FILE, name=header.name, size=header.size) data = MemberDecoder(stream, header.method, header.packed_size) yield MemberSpan(member, 0, (MemberDigest("crc32", header.crc),), data=data) stream.skip(data.packed_left) ``` - **When it is read.** The stream needs to be good only until the walk moves on. Before archivey asks `walk()` for the next member, it reads what the caller left of the data, up to the member's `size`, and closes the stream. So when `walk()` resumes, the library is past the member, and the walk reads on from there. - **Sizes.** Give `size` when the header lists it. Data longer than it raises `CorruptionError`, and shorter `TruncatedError`, as for any member. A `FILE` with no `size` is read to the end of the stream, and `size` is filled in once it has been. Draining unread data stops one byte past the listed size, so a header that lies about it cannot make archivey decode without end. - **Checks.** `MemberSpan.digests` are checked against the decoded bytes. If the library checks them itself and raises when they do not match, set `MemberSpan(checked=True)`, as for `MemberData.checked`. - **Random access.** A member's decoded data exists only while the walk is at it. In a pass (`stream_members()`, `extract_all()`), archivey reads it right then, so the archive is decoded once. `open()` or `read()` of a member afterwards walks the archive again from the start to that member, draining the members before it. Opening members in archive order carries on from the last one, so reading them all in order decodes the archive once more, not once per member. With `seekable_members=True`, a seek back walks again too. Have `archive_info()` return `ReaderArchiveInfo(is_solid=True)` so the cost receipt says so. - **Walking again.** Because of that, `walk()` is called again while an earlier walk is still open, and with `concurrent_operations=True` from several threads (archivey holds one lock across each call into the reader and each read of its streams). Keep a walk's state in the generator, not on the reader. A second walk must yield the same members in the same order: archivey finds a member again by its place, and raises `ValueError` if another member is there. - **Link targets.** With `link_target_in_data = True`, a symlink's target is read from its decoded data. ## Reading untrusted bytes archivey cannot check how your reader parses the archive, only what it hands back. These habits keep a crafted file from doing more than raising one of archivey's errors: `CorruptionError`, `TruncatedError` or `UnsupportedFeatureError`. - **Read with `ReaderSource.read_struct()` and `read_exact()`** wherever the bytes must all be there, as the example does. They raise `TruncatedError` when the archive ends early. They and `read_at()` are the cheap way to parse: each is one read of the archive, with no stream built, and safe from any thread. Keep `view()` for a member's data. - **Check every size and offset against what the format allows** before you allocate or read with it. Bound offsets by the format's own structures (a superblock's table offsets, an index's declared extent, the bytes a read returned), and also by `ReaderSource.size` when it is not `None`: it is `None` for a stream whose length archivey cannot learn cheaply. Refuse a count or length larger than the format can hold, and bound a count the format leaves wide open by `config.listing_limits`. - **Walk trees with a loop and a set of what you have seen**, not recursion. A directory that lists itself is then an error, not a hang. - **Validate names** before you yield them. A name with `/` in a format that stores one path component per entry is damage. archivey checks the final path in any case. - **Cap every decode** at the size the format allows, as [Where the bytes come from](#where-the-bytes-come-from) describes for a decoder of your own. - **Raise archivey's exceptions** for damage, and let your own bugs propagate, as [Errors](#errors) describes. The SquashFS package's tests show a way to test this. They take an image `mksquashfs` wrote, change one field (a block size, a fragment index, a directory entry's type) and check that each change raises `CorruptionError`, `TruncatedError` or `UnsupportedFeatureError`. ## Errors When the archive is damaged, raise archivey's own exceptions from the reader: `CorruptionError` for bytes that cannot be right, `TruncatedError` for an archive that ends too early, and `UnsupportedFeatureError` for a feature your reader does not handle. An exception from a library you wrap can go through `translate_exception()`. It is asked about every exception raised while archivey calls the reader, and while a member stream it returned is read. Errors from `ReaderContext.decompress()` are archivey's own already and never reach it. Return the archivey exception to raise in its place, or `None` to let it propagate as it is: ```python def translate_exception(self, exc: Exception) -> archivey.ArchiveyError | None: if isinstance(exc, somelib.BadArchive): return archivey.CorruptionError(str(exc)) return None ``` Define it on the `FormatHandler` to cover `open()` too, where there is no reader yet. The plugin's `translate_exception()` is also asked about anything the reader's returns `None` for, so a translation that needs nothing from the open archive is written once, on the plugin. ## Reporting problems that do not stop the read Some damage is worth telling the caller about without refusing the archive: a date that is not a date, a header field that had to be skipped, bytes after the end of the archive. Report it with `ReaderContext.report()`, using one of archivey's own diagnostic codes and the context type that code takes: ```python from archivey.diagnostics import DiagnosticCode, MemberTimestampContext self._context.report( DiagnosticCode.MEMBER_TIMESTAMP_INVALID, f"Invalid date for {member.name!r}: {raw!r}", MemberTimestampContext( member_name=member.name, field="modified", source="header", value_repr=repr(raw), ), member=member, ) ``` Leave `archive_name` and `member_id` unset: archivey fills them in. Pass `member` to attach the report to a member, either while listing it (before it is yielded) or while opening it. The caller's `DiagnosticPolicy` applies, so a code the caller raises on raises from `report()`. A report made while listing a member should name it as callers will see it, and archivey normalizes names only after `iter_members()` yields them. So normalize the name first with `ReaderContext.normalize_name()`, set `member.name` to the result, and yield the name you started from as `MemberRecord(member, locator, stored_name=...)`. archivey then reports the change from that name, as it would have. A symlink whose target cannot be read is reported with `ReaderContext.link_target_unavailable()`, which also leaves `link_target` unset and tells extraction whether the archive holds a target the reader could not reach. `ReaderContext.config` is the caller's `ArchiveyConfig`. A reader that parses a large index itself can bound that work by `config.listing_limits`. archivey applies those limits to the members it is handed in any case. ## Archive details `archive_info()` returns a `ReaderArchiveInfo` with what archivey cannot work out itself: whether members share one decode (`is_solid`), a format version, a comment, and extra values under keys named after your format. A key a built-in format already defines, such as `7z.volume_count`, is not yours to write. archivey fills in the rest of `archive.info`. ## Detection archivey recognises a plugin's format by its `magic`, its `extensions`, and an optional `probe`: - A **magic** is exact bytes at an offset. It counts only when no built-in format's magic matched, so registering a plugin never changes how a file archivey already reads is detected. - An **extension** that a built-in format uses is refused at registration. - A **probe** is a function that takes the first bytes of the source and returns `True` for your format. It gets only the bytes detection has already read, and runs after every built-in check. Like a built-in format's probe, it runs when the source's name has one of your extensions, when the source has no name at all (a stream passed to `open_stream()`), or when `ArchiveyConfig(always_probe_content=True)` is set. So it sees untrusted bytes that may not be your format at all. The prefix may be shorter than your format's header. A probe that raises counts as no match, and the first time it does archivey logs a warning on `archivey.detection`. `MemoryError` and `OSError` are the exception: they propagate, as everywhere else, so decide from the prefix alone and do not reach outside it. Use a probe only for a format with no fixed signature. `FormatInfo.from_plugin` is `True` when detection chose a plugin's format. ## Optional dependencies A plugin that wraps a library names its modules in `requires` and what to install in `install_hint`: ```python class CabHandler(FormatHandler): requires = ("cabarchive",) install_hint = "pip install cabarchive" ``` When a module is missing, `format_availability()` reports the format as `FormatSupport.NONE` with the hint, and opening a file of that format raises `PackageNotInstalledError`, as for a built-in format whose extra is not installed. A library that needs a file name rather than a stream can use `ReaderSource.path`, which is the path the archive was opened from, or `None` when it was opened from a stream. ## Replacing a built-in reader A plugin can also read a format archivey already reads, in place of the built-in reader, for the calls that ask for it. It is not registered. The caller names it in the config of each call: ```python from archivey import ArchiveyConfig, ContainerFormat, FormatPolicy, open_archive MY_ZIP = ArchiveyConfig(formats=FormatPolicy(replace={ContainerFormat.ZIP: MyZipPlugin})) with open_archive("a.zip", config=MY_ZIP) as reader: ... ``` The replacement is an indexed plugin whose `format` is the built-in one, here `ArchiveFormat.ZIP`, and it declares no `extensions`, `magic` or `probe`: the file is found by the built-in detection, which does not change. `reader.format` is the built-in format and `format_info.from_plugin` is false. ZIP, 7z, RAR and ISO can be replaced. `FormatPolicy` checks the plugin when it is built, so a plugin that does not fit fails there, before any open. The replacement gets what any indexed plugin gets: a source that can seek, from the start of the archive. archivey refuses, before `open()` runs, a multi-volume set and an archive behind a self-extracting stub with `UnsupportedFeatureError`, a source that cannot seek with `StreamNotSeekableError`, and a raw CD sector image detected as ISO as the built-in reader refuses it, since its bytes are not an ISO 9660 image. The format's own config options, such as `zip_unflagged_fallback_encoding` or `rar_decompressor`, reach the plugin through `ReaderContext.config`; honouring them is the plugin's job. `password=` does not reach a replacement, as it reaches no plugin, so a replaced ZIP, 7z or RAR cannot read an encrypted member; `encoding=` reaches it when the plugin sets `uses_encoding`. A missing module in `requires` raises `PackageNotInstalledError` naming the plugin, without falling back to the built-in reader. ## Registration and trust `register_format()` is the only way to add a format: nothing is loaded automatically because a package is installed. The program that calls it chooses which parsers read its input. A plugin's code runs in your process like any other import. archivey's limits cover what the plugin hands over: listing limits, member names, extraction limits, and the size and checksum checks on member data. They cannot cover the plugin's own parser, which could use unbounded memory or crash on a crafted file; [Reading untrusted bytes](#reading-untrusted-bytes) lists what a plugin's author can do about that. A program that reads untrusted archives can turn plugin formats off for a call with `ArchiveyConfig(formats=FormatPolicy(plugins=False))`; a plugin format turned off is not detected either, so its code does not run. A [replacement](#replacing-a-built-in-reader) is not a plugin format, so `plugins=False` does not turn it off: a replacement runs only in a call whose config names it in `replace=`. ## What is not supported yet - A format with an index inside an outer compressor. Only a format read in one pass can be. - Self-extracting archives, multi-volume sets and passwords. - New diagnostic codes. A plugin reports with archivey's existing codes. - Writing archives. # Migrating from other libraries Archivey reads ZIP, TAR, 7z, RAR and ISO archives, and single compressed files, through one interface, so code written for zipfile, tarfile or a `7z` subprocess becomes the same few calls. This page gives the archivey version of each, then what behaves differently. Archivey puts correctness and the same behaviour in every format ahead of matching any one library. The difference you'll meet first is in extraction: archivey refuses some members that zipfile or tarfile would write, and it stops an archive that expands past its limits. ## Equivalent calls Each item gives the archivey call first, then the calls it replaces and what's different. - **Open an archive:** `archivey.open_archive(p)`. - Standard library: `zipfile.ZipFile(p)`, `tarfile.open(p)`. - py7zr or rarfile: `py7zr.SevenZipFile(p)`, `rarfile.RarFile(p)`. - [Any format, identified from its contents, with options for how you'll read it](#opening-an-archive). - **List the names:** `[m.name for m in reader]`. - Standard library: `zf.namelist()`, `tf.getnames()`. - py7zr or rarfile: `z.getnames()`, `rf.namelist()`. - [A name can appear more than once](#members-and-their-details). - **Look up one member:** `reader.get(n)`. - Standard library: `zf.getinfo(n)`, `tf.getmember(n)`. - py7zr or rarfile: `z.getinfo(n)`, `rf.getinfo(n)`. - Returns an ArchiveMember. - **Read one member:** `reader.read(n)`. - Standard library: `zf.read(n)`, `tf.extractfile(n).read()`. - py7zr or rarfile: `z.extract(targets=[n], factory=f)`, `rf.read(n)`. - [A directory raises an error](#reading-members). - **Open one member as a file object:** `reader.open(n)`. - Standard library: `zf.open(n)`, `tf.extractfile(n)`. - rarfile: `rf.open(n)`. - [One open at a time, and not seekable, by default](#reading-members). - **Read every member, in order:** `for member, stream in reader.stream_members():`. - Standard library: `for info in tf: tf.extractfile(info)`. - py7zr: `z.extractall(factory=f)`. - [One pass over the archive, in every format](#reading-members). - **Give a password:** `archivey.open_archive(p, password="...")`. - Standard library: `zf.read(n, pwd=b"...")`. - py7zr or rarfile: `SevenZipFile(p, password="...")`, `rf.setpassword("...")`. - [Given once, when opening, and can be a list](#opening-an-archive). - **Extract everything:** `reader.extract_all(d)`. - Standard library: `zf.extractall(d)`, `tf.extractall(d)`. - py7zr or rarfile: `z.extractall(d)`, `rf.extractall(d)`. - [Unsafe members are refused, and policies and filters choose what's written](#extracting). - **Unpack any supported format:** `open_archive(p)`, then `reader.extract_all(d)`. - Standard library: `shutil.unpack_archive(p, d)`. - [More formats, and a report of what was written](#extracting). - **Decompress a single file:** `archivey.open_stream(p)`. - Standard library: `gzip.open(p)`, `bz2.open(p)`, `lzma.open(p)`. - [One call for every compression format](https://archivey-dev.github.io/archivey/opening/index.md#single-compressed-files). - **Identify the format:** `archivey.detect_format(p)`. - Standard library: `zipfile.is_zipfile(p)`, `tarfile.is_tarfile(p)`. - [Names the format, among all those archivey reads](https://archivey-dev.github.io/archivey/opening/index.md#format). ## Opening an archive `open_archive` opens every format archivey reads and identifies it from the file's contents, so a TAR file named `backup.bin` opens too, where `shutil.unpack_archive` fails because of the extension. If your code actually needs to handle only one format, you can pass `format="zip"` or `format=archivey.ArchiveFormat.TAR_GZ` to skip detection. A file in another format then fails like a damaged one ([Format](https://archivey-dev.github.io/archivey/opening/index.md#format)). A password is given once, when opening: one string, a list of candidates, or a function that supplies them ([Passwords](https://archivey-dev.github.io/archivey/opening/index.md#passwords)). zipfile takes one per read or through `setpassword`, and only for the older ZIP encryption. With the `recommended` extra, archivey also reads WinZip AES. py7zr takes the password when opening, like archivey, and rarfile takes it per read or through `setpassword`. Like rarfile, archivey reads files stored without compression in a RAR archive on its own, and needs a program for compressed ones: RARLAB's `unrar` or `rar`, or `unar` when neither is installed. rarfile can also use `7z` or `bsdtar`, and archivey can't. Three options to `open_archive` allow ways of reading that are off by default: - **Seek in a member:** `seekable_members=True`. With the [`seekable` extra](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member), seeks in bzip2 and large DEFLATE data (gzip files and most ZIP members) restart near their target. zipfile, tarfile and rarfile allow it, and a seek back decompresses again from the start. py7zr has no member streams. - **Keep several members open at once:** `concurrent_operations=True`. zipfile, tarfile and rarfile allow it. py7zr has no member streams. - **Read from a pipe:** one pass, with `streaming=True`, for TAR, ZIP and single compressed files. tarfile reads one pass with mode `"r|*"`. zipfile, py7zr and rarfile can't read from a pipe. Seeking and keeping several members open are cheap on some archives and slow on others, and nothing in the calling code shows which. Archivey turns them off by default so that code tested on a ZIP doesn't first turn slow on a `.tar.gz` in production. [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md) explains each option and what it costs. ## Members and their details Every format gives an ArchiveMember with the same fields. The ones code using `ZipInfo` or `TarInfo` reads most map like this: - `member.name`: `info.filename` in zipfile, py7zr and rarfile, `info.name` in tarfile. - `member.size`: `info.file_size` in zipfile and rarfile, `info.size` in tarfile, `info.uncompressed` in py7zr. - `member.compressed_size`: `info.compress_size` in zipfile and rarfile, `info.compressed` in py7zr. - `member.modified`: `info.date_time` in zipfile, `info.mtime` in tarfile and rarfile, `info.creationtime` in py7zr. - `member.mode`: `info.external_attr >> 16` in zipfile, `info.mode` in tarfile and rarfile. - `member.is_dir`, `member.is_file`, `member.is_link`: `info.is_dir()` in zipfile; `info.isdir()`, `info.isfile()`, `info.issym()` in tarfile; `info.is_directory`, `info.is_file`, `info.is_symlink` in py7zr; `info.is_dir()`, `info.is_file()`, `info.is_symlink()` in rarfile. - `member.created` or `member.ctime`: `info.ctime` in rarfile. - `member.link_target`: `info.linkname` in tarfile. zipfile doesn't recognize symbolic links: a ZIP entry that records one is a plain file to it, and `extractall` writes the target's path as the file's contents. Archivey reports it as a link, with `member.is_link` and `member.link_target`. A field the archive doesn't record is `None`, such as `compressed_size` for a member of a `.tar.gz`. A directory's name ends with `/` in every format, so a directory tarfile lists as `docs` is `docs/` in archivey. `reader.get` returns `None` for a name that isn't there, where `getinfo` and `getmember` raise `KeyError`. `member.modified` is a `datetime` with the time zone the archive records. Some ZIPs, such as those made by the `zip` command, carry a UTC time beside the old local one. zipfile ignores it, and archivey uses it, so `modified` is in UTC there. A ZIP without it, or a RAR 4 archive, records a local time with no zone, and `modified` has no zone either. TAR, 7z and RAR 5 archives record UTC: tarfile gives a number of seconds, and py7zr, rarfile and archivey give a UTC `datetime`. py7zr's `creationtime` holds this modification time. `member.modified_utc()` gives UTC for every format, and treats a time with no zone as your local time unless you pass another zone. RAR, 7z and ZIP archives have a slot for a creation time, and what goes in it depends on where the archive was made. Tools on Windows store when the file was created, and tools on Linux and macOS store when its metadata last changed, Unix's `ctime`. rarfile gives the slot as `ctime` whatever it holds. Archivey puts a creation time in `member.created` and a change time in `member.ctime`, so `created` never holds a change time. When an archive holds a name twice, zipfile, tarfile and archivey all list it twice, and reading by name gives the last copy. Going through the member list, `member.is_current` tells you which copy that is ([Members with the same name](https://archivey-dev.github.io/archivey/opening/index.md#members-with-the-same-name)). ## Reading members `reader.read` and `reader.open` work like their zipfile and tarfile counterparts, in every format archivey opens. For ZIP, the `recommended` extra adds compression methods zipfile can't read: Deflate64, PPMd, and zstd before Python 3.14. Calling `read` or `open` on a directory raises `ArchiveyUsageError`, where zipfile returns empty bytes and tarfile's `extractfile` returns `None`. Reading a symbolic link gives its target's contents, as tarfile does, and raises `LinkTargetNotFoundError` when the target isn't in the archive, where tarfile raises `KeyError`. zipfile gives the link's own text, the target's path. In a `.tar.gz`, in most 7z archives and in some RAR archives, members are compressed together, so reading one member on its own means decompressing everything before it. tarfile avoids repeating that work when you read each member while iterating over the archive, and py7zr when you pass all the names to one `extract` call. rarfile has no way to avoid it. With archivey, use `stream_members()` whenever you need several members and their order doesn't matter: it reads each member once, in order, in every format ([Solid archives](https://archivey-dev.github.io/archivey/reading/index.md#solid-archives)). `read` and `open` cost more. In a `.tar.gz`, the first one lists the whole archive, which takes a pass of its own, and reading members in the archive's order after that takes one more pass. Going back to an earlier member starts again from the beginning. In 7z and RAR, each read decompresses again everything before its member. If you want to make sure your code avoids these costs, open the archive with `streaming=True`: `read` and `open` then always raise, and `stream_members()` is the only way to read members. ## Extracting `extract_all` never writes outside the destination folder, in any format or under any policy. zipfile, tarfile with `filter="data"`, and archivey's `strict` (the default) and `standard` policies treat the same members differently: - **`/abs.txt`:** zipfile, tarfile and archivey's `standard` write it as `abs.txt`. Archivey's `strict` refuses it. - **`../up.txt`:** zipfile writes it as `up.txt`. tarfile raises, ending the extraction. Archivey refuses it. - **A link to `../../etc/passwd`:** zipfile writes a plain file holding the path. tarfile raises, ending the extraction. Archivey refuses it. - **A file with mode 4755 (setuid):** tarfile and archivey write it as 755. zipfile doesn't keep the mode. - **A file with mode 666:** tarfile and archivey's `strict` write it as 644, and archivey's `standard` as 666. zipfile doesn't keep the mode. - **A file with mode 755:** tarfile and archivey write it as 755. zipfile doesn't keep the mode. zipfile ignores the permissions an archive records, so each file gets the system's default for new files, usually 644. py7zr raises on `../up.txt` and on a link pointing outside, and rarfile renames `../up.txt` and `/abs.txt` as zipfile does and raises on the link. Both keep the recorded permissions. The data filter is tarfile's default from Python 3.14. Before that, tarfile with no filter writes every member as recorded, outside the destination too, and Python 3.12 and 3.13 warn about it. The Extracting page has [a longer table](https://archivey-dev.github.io/archivey/extracting/index.md#what-each-policy-does-with-unusual-members), with more kinds of member and every policy. When archivey refuses a member, it skips it and goes on with the others, where tarfile's data filter raises and ends the whole extraction. `abort_on=["blocked_member"]` makes archivey stop too. If you want zipfile's renaming, pass `filter=archivey.sanitize_names`. `filter=` also takes your own function, which can rename, skip or change each member ([Extracting all or some members](https://archivey-dev.github.io/archivey/extracting/index.md#extracting-all-or-some-members)). zipfile and tarfile write as much as the archive holds. Archivey stops at 2 GiB written, 262,144 members, or a member that expands more than 1000 times, and all three can be changed ([Archives you trust](https://archivey-dev.github.io/archivey/extracting/index.md#archives-you-trust)). `extract_all` returns an `ExtractionReport` saying what happened to each member, where `extractall` returns `None`. `shutil.unpack_archive` picks the format from the extension, handles ZIP and TAR (with gzip, bzip2, xz, and zstd from Python 3.14), and extracts them through zipfile and tarfile, so the comparison above applies to it too. ## Errors Every problem with the archive raises a subclass of `archivey.ArchiveyError`, so one `except` covers what took several exception types before: - **Not an archive:** archivey raises `FormatDetectionError`. zipfile raises `BadZipFile`, tarfile `ReadError`, py7zr `Bad7zFile` and rarfile `NotRarFile`. - **Damaged or cut short:** archivey raises `CorruptionError`, or `TruncatedError` when cut short. zipfile raises `BadZipFile`, tarfile `ReadError` or `EOFError` from the decompressor, py7zr `Bad7zFile` or `CrcError`, and rarfile `BadRarFile`. - **Wrong password:** archivey raises `EncryptionError`. zipfile raises `RuntimeError`, py7zr `LZMAError`, and rarfile `BadRarFile`, or `RarWrongPassword` when the names are encrypted too. - **No member with that name:** archivey raises `KeyError`, as zipfile, tarfile and py7zr do. rarfile raises `NoRarEntry`. A missing name raises `KeyError`, which isn't an `ArchiveyError`, because the mistake is in the calling code, not the archive. [Errors and diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md) explains each exception, and the warnings archivey reports without raising. # Reference # API reference This page lists archivey's public API, grouped by what you use it for. Each entry is headed by the full name to import, such as `archivey.open_archive` or `archivey.terminal.quoted`, and each group links to the page that explains it, where there is one. ## Opening See [Opening an archive](https://archivey-dev.github.io/archivey/opening/index.md). ### `archivey.open_archive(source, *, format=None, streaming=False, seekable_members=False, concurrent_operations=False, password=None, encoding=None, config=None)` ```python open_archive( source: OpenSourceInput, *, format: ArchiveFormat | str | None = None, streaming: Literal[False] = False, seekable_members: bool = False, concurrent_operations: bool = False, password: PasswordInput = None, encoding: str | None = None, config: ArchiveyConfig | None = None, ) -> ArchiveReader ``` ```python open_archive( source: OpenSourceInput, *, format: ArchiveFormat | str | None = None, streaming: bool, seekable_members: bool = False, concurrent_operations: bool = False, password: PasswordInput = None, encoding: str | None = None, config: ArchiveyConfig | None = None, ) -> ForwardArchiveReader ``` Open an archive for reading. `streaming=False` (the default) opens for random access and fails fast at open time on a non-seekable source. `streaming=True` promises forward-only, single-pass access (works on any source, but disables random-access methods). The return type follows the access mode: ArchiveReader for the default `streaming=False`, and ForwardArchiveReader, which lacks `members()`, `get()`, `open()` and `read()`, for `streaming=True` or a flag whose value is known only at run time. A type checker then flags a random-access call on a streaming reader. Every reader is an `ArchiveReader` at run time; the difference is in the annotations only. By default member streams are forward-only, one may be open at a time, and reader operations do not overlap. Two keyword flags opt into more, each unlocking one specific trap: - `seekable_members=True` — `seek()` on a member stream from random `open()` works. Without it, `seek()` raises `io.UnsupportedOperation`. A backward seek may re-decompress from the start when there is no index or accelerator. A `stream_members()` stream never seeks, with or without this flag. - `concurrent_operations=True` — operations on the reader may overlap, from one thread or several, on every format: random `open()` / `read()` / `get()` / `members()`, iteration, `stream_members()` passes and `extract_all()`. Each works as if it ran on a reader of its own. A single member stream object still has one owner, first-touch materialization is coordinated rather than parallel, and `close()` waits for calls in flight. Overlap is correct, not always cheaper. Without it, a second overlapping `open()`, or a random-access call or another pass beside a running pass, raises `ArchiveyUsageError`; `close()` beside a pass suspended at a yield closes the reader, and resuming the pass raises. `open_stream` uses the same vocabulary for the single-stream case (`open_stream(..., seekable=True)`); concurrency is meaningless there, so it has no counterpart. Declared concurrency does **not** gate solid open-order cost — see `AccessCost` / `stream_members()`. `streaming=True` combined with `concurrent_operations=True` is rejected (`ArchiveyUsageError`): a forward-only pass cannot fan out. `config` supplies library tuning knobs (accelerator modes, the diagnostic policy, default extraction limits, and listing resource limits via `listing_limits`). `None` selects the module default DEFAULT_ARCHIVEY_CONFIG. The format is auto-detected from the source's magic bytes (then its extension) unless `format=` is passed explicitly. A directory path opens as a directory pseudo-archive. A non-seekable stream keeps the bytes detection peeked in a replay prefix that the backend's first reads drain, so detection never consumes bytes the backend needs. A stream must be blocking: when a non-blocking one has nothing ready (its `read` returns `None`), opening or reading raises `BlockingIOError`, not an archivey error. A seekable stream source is taken to hold the archive **starting at its current position**: detection peeks from there and restores the position, and the opener then rebases a mid-positioned stream to a zero origin so every backend sees the archive begin at `tell() == 0` (an archive embedded mid-file works uniformly, without manual slicing). `encoding` names the codec for member names whose encoding the archive does not declare and whose bytes are not valid UTF-8 (ZIP, TAR, ISO and RAR 1.5-4). A name whose bytes are valid UTF-8 is decoded as UTF-8 in every format, with or without `encoding`, and `encoding` replaces only the format's fallback (cp437 or `ArchiveyConfig.zip_unflagged_fallback_encoding` for ZIP, the host's code page for RAR 1.5-4, surrogate escapes for TAR and ISO). `member.raw_name` keeps the stored bytes. A ZIP comment without the UTF-8 flag (the archive comment always) and an 8-bit RAR 1.5-4 comment decode the same way as a name without a declared encoding, and a byte the codec does not define survives as a lone surrogate. The one difference is that a RAR comment's last fallback is always windows-1252. 7z, directory and single-file sources decode names another way, ignore it, and record `ENCODING_ARGUMENT_UNUSED`. `source` may be an ordered sequence of paths or binary streams that together form a multi-volume archive (7z concatenates volumes; RAR opens volume 1 and lets `unrar` resolve siblings). A length-1 sequence is treated as a single source. `password` accepts a single value, an ordered sequence of candidate passwords, or a provider callable. List the most likely password first — especially for 7z, where each wrong candidate pays an expensive key derivation. With multiple candidates (or a provider), formats whose per-open password check is weak may need a confirmation read before a candidate is accepted. For traditional ZipCrypto this is usually cheap: compressed members are confirmed from a bounded decompressed prefix. **STORED** ZipCrypto members are the niche exception — roughly 1/256 of wrong passwords pass the one-byte open check, and with no decompressor to reject garbage the reader must scan the member once (CRC over every surviving candidate in parallel) to decide. That full pass is rare in practice (multiple passwords *and* a colliding wrong candidate *and* a STORED member) but can matter for very large stored members. With a single password there is no confirmation read, so a wrong ZipCrypto password that passes the one-byte check fails on the member's own read, as an `EncryptionError` saying the password may be wrong or the member corrupt. A password supplied for a *format* with no encryption at all (TAR, ISO, a directory, the single-file compressed streams) is **accepted, not refused** — it is a resource offered, not a claim about this archive. A concrete password (a string, bytes, or a list of them) is recorded as `PASSWORD_ARGUMENT_UNUSED`; a provider callable is not, because it is only a way to ask for a password, and such a format never asks. That is what lets a batch job pass one keyring at every archive. The check is per format, made before any header is read: an unencrypted ZIP, 7z or RAR records nothing, because those formats can use a password. Diagnostics also log at `WARNING` by default, so a job passing a password list will log once per archive of a format without encryption; silence it with `ArchiveyConfig(diagnostic_policy=DiagnosticPolicy(ignore={ DiagnosticCode.PASSWORD_ARGUMENT_UNUSED}))`, which keeps the count without the log line. Reader calls made after the open raise the same `ReadError` subclasses for damage, a wrong password or a missing link target that shows only when a member is read. Raises: - `FormatDetectionError` – No format matches the source's contents or name. - `StreamNotSeekableError` – The source cannot seek and streaming=False, or the format needs to seek in either access mode (every format except TAR, ZIP and the single-file compressed streams). - `CorruptionError` – The archive's header is damaged or cut short. A numbered split set with a missing part raises the subclass TruncatedError. - `EncryptionError` – The member list is encrypted (7z, RAR) and no password given opens it. - `UnsupportedFeatureError` – The format is recognized, but this archive uses a variant or feature archivey cannot read, or config.formats does not allow the format. - `PackageNotInstalledError` – The format needs an optional package that is not installed, such as pycdlib for ISO. - `ResourceLimitError` – The archive exceeds a limit that is checked at open, such as ListingLimits max_members while a ZIP, 7z, RAR or ISO index is parsed, or DecoderLimits for data the open decodes. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded while the archive is detected or opened. - `ArchiveyUsageError` – One argument has a type or value the call cannot use (then also a TypeError or ValueError), or streaming=True is combined with concurrent_operations=True. - `FileNotFoundError` – The path does not exist. Other OSError from reading the source is not wrapped either. ### `archivey.open_stream(source, *, format=None, seekable=False, config=None)` Open a single-file compressed stream and return a decompressing stream. This is the compressed-streams entry point for a bare `.gz` / `.bz2` / `.xz` / … payload (no archive container). `seekable` is the same capability open_archive() spells `seekable_members`; concurrency is not a concept here — the call returns exactly one stream — so there is no counterpart to `concurrent_operations`. `seekable=False` (the default) returns a forward-only stream: `seekable()` is `False`, `seek()` raises `io.UnsupportedOperation`, and no seek index or accelerator is instantiated. Pass `seekable=True` to opt into the seekable-decompressor-streams contract (native indexes, demand-driven accelerator `AUTO`, loud slow rewinds on the non-accelerated path). `format` accepts a StreamFormat, a raw-stream ArchiveFormat (e.g. `ArchiveFormat.GZ`), a compressed-tar ArchiveFormat (e.g. `ArchiveFormat.TAR_GZ`), or `None` to auto-detect. For a compressed tar (`.tar.gz`, `.tar.xz`, …), whether detected or passed as `format=ArchiveFormat.TAR_GZ` and so on, the returned stream removes the compression layer only and yields the tar bytes, as `gzip.open` does; `open_archive(open_stream(p), streaming=True)` then lists the same members as `open_archive(p)`. Any other container (ZIP, 7z, RAR, ISO, an uncompressed tar) has no compression layer to remove: detecting one raises FormatDetectionError, and passing one as `format=` raises ArchiveyUsageError. Use open_archive() for those. A stream must be blocking: when a non-blocking one has nothing ready (its `read` returns `None`), opening or reading raises `BlockingIOError`, not an archivey error. Reading the returned stream raises `CorruptionError` (or `TruncatedError`) for damaged data, on the read that finds it. Raises: - `FormatDetectionError` – No compression format matches the source, or the source is an archive container (ZIP, 7z, RAR, ISO, an uncompressed tar). - `StreamNotSeekableError` – seekable=True and the source cannot seek. - `ResourceLimitError` – The stream asks a decoder for more memory than DecoderLimits allows, raised by the read of the returned stream that starts the decoder. - `UnsupportedFeatureError` – config.formats does not allow the format. - `PackageNotInstalledError` – The format needs an optional package that is not installed. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded while the stream is detected or opened. - `ArchiveyUsageError` – One argument has a type or value the call cannot use (then also a TypeError or ValueError): a directory path, or a container format passed as format=. - `FileNotFoundError` – The path does not exist. ### `archivey.detect_format(source, *, config=None, follow_stub_volumes=True)` Identify the archive format of `source` without fully opening it. Returns a FormatInfo. Raises FormatDetectionError when no magic pattern matches and no extension guess is available. A non-blocking stream with nothing ready (its `read` returns `None`) raises `BlockingIOError` instead: bytes that have not arrived are not the same as no bytes. Detection diagnostics are on the returned FormatInfo.diagnostics, recorded into a standalone collector made from `config` (or the library default). What detection may read and decode is `config.detection_budget` (ArchiveyConfig.detection_budget), `BALANCED` by default — the same budget open_archive() detects under for that config. LZMA Alone, zlib and Brotli have no magic. Their content probes run only when the source's name ends in that format's extension, unless ArchiveyConfig.always_probe_content is set; a nameless raw stream of these formats raises FormatDetectionError. `config.formats` (FormatPolicy) does not change the answer for a built-in format. A plugin format it turns off is not detected: its magic, extensions and content probe are left out, and `unavailable_tiers` records each tier that left one out as `NOT_ENABLED_BY_POLICY`. A stub-only `.exe` / `.sfx` (no archive magic) beside a 7-Zip split first volume is detected as that volume's format when `follow_stub_volumes` is true — the default, so `detect_format("vol.exe")` agrees with `open_archive`. `open_archive` probes with this flag off, then switches the source itself. The returned `cost_receipt` and `unavailable_tiers` then cover both passes, the stub's and the volume's. Each pass runs under the full `budget`, and the receipt's `passes` is 2. A path to any part of a numbered split set (`set.7z.002`, `set.zip.003`) or to a RAR continuation is detected on the set `open_archive` reads for it: the parts joined in order, or RAR volume 1. A middle part alone has no magic at offset 0. That resolution runs before detection reads a byte, and it can raise: `TruncatedError` for a numbered set with a gap (`set.zip.002` and `set.zip.003` with no `set.zip.001`), as `open_archive` does. A lone first part (`set.zip.001` with no other part) is detected as the format its bytes show, where `open_archive` refuses it as an incomplete set. A directory path returns ArchiveFormat.DIRECTORY with `CERTAIN` confidence and `detected_by="directory"`, matching `open_archive`, which reads it as a directory archive. Nothing is read to decide that, so `cost_receipt` is the zero receipt (one pass, no bytes). Raises: - `FormatDetectionError` – No format matches the source's contents or name. - `TruncatedError` – The path names part of a numbered split set with a gap. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded during detection. - `ArchiveyUsageError` – One argument has a type or value the call cannot use (then also a TypeError or ValueError). - `FileNotFoundError` – The path does not exist. ### `archivey.FormatInfo` The result of detect_format() — the detected format plus how sure we are. Fields: - #### **`format`** (`ArchiveFormat`) – The detected format. - #### **`confidence`** (`DetectionConfidence`) – How far the evidence behind format goes. - #### **`detected_by`** (`DetectedBy`) – Which evidence decided: "magic", "extension", "content_probe", "sfx_scan" or "directory". An open set: a later release may add a detection step with a new value, so code that matches on it should handle a value it does not know. "sfx_scan" covers any archive found behind a prefix, a #! launcher included, not only a self-extractor's executable stub. - #### **`payload_offset`** (`int`, default: `0` ) – Where the archive starts in the source. Nonzero only for a self-extracting archive, so payload_offset > 0 is the test for one. - #### **`diagnostics`** (`DiagnosticSummary`, default: `DiagnosticSummary(total_count=0, counts=mappingproxy({}), retained=(), dropped_count=0)` ) – What detection reported on its way to the answer. - #### **`corroborated`** (`bool`, default: `False` ) – Provisional and informational, not part of the detect_format() contract: whether a matching extension, an inner-TAR upgrade, or a self-checking header (a TAR header checksum) corroborated a content-probe claim. Read it only together with detected_by == "content_probe". False also means "not a probe at all", so an exact magic hit reads False too. A later release may replace this field with a record of the evidence that separates the two cases. - #### **`cost_receipt`** (`DetectionCostReceipt | None`, default: `None` ) – The work detection did, as a DetectionCostReceipt: bytes read and scanned, decode input and output, and the number of passes. It covers the whole call, both passes when detect_format() followed a stub to its split volume. detect_format() always fills it; a directory gets the zero receipt. None only on a FormatInfo that detection did not produce. - #### **`unavailable_tiers`** (`tuple[TierSkip, ...]`, default: `()` ) – The detection tiers that did not run, each a TierSkip with its TierSkipReason: not enabled by policy, a capability the source lacks, or the budget ran out. Empty when every tier ran. - #### **`from_plugin`** (`bool`) – (computed property) Whether format is one a plugin added (archivey.formats), not a built-in one. A plugin's parser is not covered by archivey's limits, so a caller that reads untrusted input can refuse such a format. ### `archivey.DetectionConfidence` Bases: `Enum` How much detect_format() trusts the format it reports. Provisional in 0.2.x: which grade a detection step reports may change in a later release (a two-byte magic reported below `CERTAIN`, for instance). Branch on the format, not on the grade. Members: - **`CERTAIN`** (`'certain'`) – An exact magic-byte match at the expected offset. - **`PROBABLE`** (`'probable'`) – A structural or content probe matched: the inner-TAR probe, the SFX scan, or a content probe hit (LZMA-alone, zlib, or Brotli whose first meta-block is compressed or whose extension agrees). - **`GUESS`** (`'guess'`) – No confirmation strong enough to rely on: an extension-only guess, or a content probe hit in the weak evidence class (today: extensionless Brotli whose first meta-block is uncompressed or metadata). ### `archivey.format_availability(fmt, *, check_programs=True)` Public query: the tri-state support level of `fmt` and its missing components. `fmt` is an ArchiveFormat — the `(container, stream)` pair — or that format spelled as a string, such as `"zip"` or `"tar.gz"`. Anything else, a StreamFormat or a ContainerFormat included, raises ArchiveyUsageError rather than answering: each is half of the pair, not a format. RAR is `PARTIAL`, with `unrar` in `missing`, when no RARLAB `unrar` or `rar` 6.0 or later is on `PATH`: it still opens and lists, and stored members read. To find out, the first call runs each candidate once to read its banner, for at most 10 seconds each; the answer is cached per binary. It describes the default `rar_decompressor` (`AUTO`) and does not change with `ArchiveyConfig`. `check_programs=False` runs no program: missing programs are then not counted, and RAR reports as it would with `unrar` installed. ### `archivey.FormatAvailability` The support level of a format plus the components needed to raise it. Fields: - #### **`format`** (`ArchiveFormat`) – The format this answer is about. - #### **`support`** (`FormatSupport`) – How much of the format can be read. - #### **`missing`** (`tuple[MissingComponent, ...]`, default: `()` ) – What to install to read more of the format. Empty when support is FULL. - #### **`required_source`** (`StreamCapability`, default: `StreamCapability.SEEKABLE` ) – The weakest source shape this format can be read from. StreamCapability is ordered, so this is a minimum: a source is strong enough when availability.required_source \<= reader.cost.stream_capability. It exists so that "can I pipe this straight in, or must I buffer it to disk first?" is a query rather than a StreamNotSeekableError to catch. Independent of support — a format whose optional dependency is missing still answers the question. The SEEKABLE default is the conservative answer, and is what a format with no registered backend at all reports. ### `archivey.FormatSupport` Bases: `Enum` How much of a known format this installation can read. Members: - **`FULL`** (`'full'`) – The format can be read, with every optional codec and program it can use. - **`PARTIAL`** (`'partial'`) – Archives open and list, and members in the common methods read, but an optional codec or program is missing. - **`NONE`** (`'none'`) – The format cannot be read: its reader is missing, or for a single-file format, its only codec. ### `archivey.MissingComponent` A package, extra, or external tool required for a format (or a codec inside it). Appears on FormatAvailability when something is absent, and as the `requirement` on internal codec/backend descriptors. Defined in this leaf module (not the registry) so codec descriptors can reference it without a registry ↔ codecs import cycle. Fields: - #### **`name`** (`str`) – The package or program, such as "pycdlib", "brotli" or "unrar". - #### **`install_hint`** (`str`) – For a package, the command that installs it, such as "pip install archivey[recommended]". For a program such as unrar, the whole message a read raises without it. - #### **`unlocks`** (`tuple[str, ...]`, default: `()` ) – The codecs or features it enables, such as ("ppmd",). - #### **`reason`** (`str`, default: `''` ) – Why archivey does not use the package on this interpreter, where installing it would not help, such as "archivey does not use lz4 under PyPy, where its frame decoder crashes the interpreter". Empty when installing it is the fix. - #### **`advice`** (`str`) – (computed property) What a report of this component says to do: reason when set, otherwise install_hint. #### `message(purpose, *, note='')` Text for the `PackageNotInstalledError` raised when this component is absent. Every raise site builds its message here so the advice a caller gets when a *read* fails cannot drift from the `install_hint` that listing and FormatAvailability report. The two used to be independent strings, and the extras consolidation updated the hints while every `raise` kept advertising extras that no longer exist. `note` appends a sentence for cases where the bare hint would mislead (e.g. a backport that is pointless on a Python version shipping the module). When `reason` is set it replaces the install advice, which would not help. For package entries only: a program entry's `install_hint` is already a complete message. ### `archivey.list_supported_formats()` Public query: formats readable now (support FULL or PARTIAL). ### `archivey.list_known_formats()` Public query: every format the registry knows, including support NONE. ## Readers and streams See [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md), in particular [Reading once](https://archivey-dev.github.io/archivey/reading/index.md#reading-once). `open_archive` returns an `ArchiveReader` unless you pass `streaming=True`. Then its return type is `ForwardArchiveReader`, which leaves out `members`, `get`, `read` and `open`, so a type checker flags calls to them. The object is still an `ArchiveReader` at run time, and those calls raise `ArchiveyUsageError`. `stream_members()` is the way to read members. ### `archivey.ForwardArchiveReader` Bases: `ABC` An open archive that the code reads in one forward pass. It has every reader method except members(), get(), open() and read(), which need random access. Iterate it, call stream_members() or extract_all(), or call members_report() for the member list. `open_archive(..., streaming=True)` returns this type, so a type checker flags those four calls, which would raise `ArchiveyUsageError` at run time. The default `open_archive()` returns ArchiveReader, which adds the four, so a function annotated with `ForwardArchiveReader` accepts a reader of either mode. The check is static only: every reader is an `ArchiveReader` at run time. A second pass on a streaming reader still raises, and no type checker can see that. Use in a `with` block. **Listing APIs** (easy to mix up): - members() — complete list or raise; random-access only. - members_report() — either mode. Returns a report on archive damage (check `error is None` for completeness); raises on limits and usage errors. On a streaming reader it uses up the forward pass (starting it, or finishing an interrupted one). For complete-or-raise there, raise `report.error` yourself. - members_report_if_available() — never scans; `None` if not yet cached. Parameters: - **`format`** (`ArchiveFormat`) – (computed property) The detected (container, stream) format of the open archive. - **`format_info`** (`FormatInfo | None`) – (computed property) How archivey.open_archive() identified the format, as archivey.detect_format() would report it. The result of the detection the open ran, kept rather than repeated: its confidence, detected_by and payload_offset describe this reader's source. A directory reports DIRECTORY / CERTAIN / "directory". None when format= was passed, so no detection ran. - **`info`** (`ArchiveInfo`) – (computed property) Archive-level metadata (format, solidity, counts, encryption, cost). - **`cost`** (`CostReceipt`) – (computed property) The listing/access cost receipt for this archive (see access-mode-and-cost). - **`diagnostics`** (`DiagnosticSummary`) – (computed property) Fresh immutable cumulative snapshot of diagnostics for this reader. #### `__contains__(member)` Whether `member` (an ArchiveMember) was yielded by *this* reader. Identity-based and O(1) — no scan — so it is valid in any access mode; useful to disambiguate members when several readers are in play. Name lookup is get(), and a non-`ArchiveMember` operand raises `TypeError` (this also keeps the `in` operator from silently falling back to a full iteration, which would consume a streaming reader's single forward pass). #### `__iter__()` Iterate members in archive order (served from cache once materialized). Returns a MemberPass of members. On a streaming reader it is the reader's one pass and holds the reader while paused, as a stream_members() pass does; close it, or use it in a `with` block, to end it early. On a random-access reader it holds nothing between members. #### `close()` Release resources held by the reader. Idempotent. **Member streams do not outlive the reader.** Any still open when `close()` runs are closed with it, in the order they were opened, the same way `zipfile.ZipFile.close()` and `tarfile.TarFile.close()` behave — reading one afterwards fails as it would for any closed file, and closing it again is a no-op. The archive's own source is released after the last of them, never underneath a stream still reading through it. A MemberPass, from stream_members() or from iterating a streaming reader, that is still alive does not stop the close: the reader closes, and advancing that pass afterwards raises `ArchiveyUsageError`. Using the *reader* itself after `close()` raises `ArchiveyUsageError`. #### `extract_all(dest, *, members=None, filter=None, policy=ExtractionPolicy.STRICT, overwrite=OverwritePolicy.ERROR, on_error=OnError.STOP, abort_on=(), on_progress=None, limits=None, dry_run=False)` Extract members to `dest` (safe-by-default; see `safe-extraction`). `members` selects which members to extract: any iterable of names or `ArchiveMember`s (read once, so a generator works), a predicate, or `None` for all. `filter` runs after the `policy` transform and before the safety checks, so it sees unsafe members too; it may rename/sanitize a member (return a `.replace()`d copy) or skip it (return `None`), and the safety checks run on what it returns. The call runs under the config the reader was opened with; `limits` overrides its extraction limits for this call only. Returns an ExtractionReport whose diagnostic summary is the delta for this extraction call. `abort_on` names events that end the whole call the first time they occur — raising instead of returning a report. It is independent of `on_error`: see AbortOn. `dry_run=True` runs the same extraction into a private scratch directory and removes it afterwards. Every check runs and every member body is read and verified, but files are created empty and nothing is written under `dest`. What `dest` holds already is read, and met as a real run meets it: the `overwrite` policy applies to it, and a file or symlink in the way is refused, skipped, replaced or renamed around in the same way. The report reads as the report of a real extraction into the same `dest` would, with paths under `dest`. A member refused by a safety check is skipped and reported as `BLOCKED`; it raises only when `abort_on` names it. Raises: - `ExtractionError` – Under on_error="stop" (the default), a member could not be written, such as a file already at its destination under overwrite="error". Its subclasses FilterRejectionError, NameCollisionError and NameRewrittenError are raised for the events abort_on names, whatever on_error says. - `ReadError` – Under on_error="stop", a member's data could not be read: a CorruptionError for damage, an EncryptionError for a missing or wrong password, a LinkTargetNotFoundError. - `ResourceLimitError` – The extraction exceeds ExtractionLimits: a total stops the extraction whatever on_error says, and a single member over max_ratio fails on its own, under on_error. A member that asks a decoder for more memory than DecoderLimits allows fails on its own too. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, whatever on_error says. - `ArchiveyUsageError` – One argument has a type or value the call cannot use (then also a TypeError or ValueError), or the reader is closed. - `OSError` – dest cannot be created or written to. #### `members_report()` Materialize the member listing and return a report. `report.error is None` means `report.members` is complete. A non-`None` error is terminal archive damage found while listing (a CorruptionError, including TruncatedError): the tuple is the recovered prefix. Only that damage is returned instead of raised. Every other failure raises, as it does from members(): a ResourceLimitError when the listing exceeds ListingLimits, ArchiveyUsageError on misuse (closed reader, overlapping call), and other read failures (a password error, an escalated diagnostic, a pass that failed earlier, an I/O error). Random access: does not consume anything; the members are the ones members() returns. Streaming: runs the single forward pass to its end (from the start, or finishing one interrupted by an early `break`), so no later pass is possible; after a completed pass it returns the cached report. A stream_members() or `for member in reader` pass enforces `ListingLimits` too: it raises the same `ResourceLimitError` at the member that crosses a limit. For a complete-or-raise list on a streaming reader:: ```text report = reader.members_report() if report.error is not None: raise report.error ``` #### `members_report_if_available()` A member-list report if available without scanning, else `None`. Never scans or consumes the forward pass, so it is safe to call on any reader (including a streaming one). #### `stream_members(members=None, *, file_copy_streams=True)` Yield `(member, stream)` pairs in archive order with bounded memory. `members` is an optional selector: a predicate, any iterable of names or members (read once, so a generator works), or `None` for all. The yielded stream is valid only until the iterator advances. The stream is `None` for every member that has no data of its own: links, special entries, tombstones and most directories. A directory that declares data (`MEMBER_DIRECTORY_DATA_IGNORED`, in ZIP, TAR and RAR) gets a stream with the bytes open() delivers for it, under the same checks, so a forward-only reader reaches them too. The exception is a ZIP directory written with a data descriptor, read from a source that cannot seek: its size comes after its data, so it gets `None`. Data archivey cannot decode is refused on the first read, as `open()` refuses it. Test `is_file`, not `stream is not None`, to tell a file from such a directory. The returned MemberPass can be closed, or used in a `with` block, to end the pass early; see there for when that matters. The pass enforces `ListingLimits` as `members()` does. A pass that lists as it goes (every streaming reader, and TAR) yields the members before the limit and then raises `ResourceLimitError` at the member that crosses it; the other readers list the whole archive first, so they raise before the first yield. An archive with more members than the default `max_members` needs a higher limit, or `ListingLimits.UNLIMITED`, to be streamed; the reader then keeps every member until it closes. A member whose data is damaged raises from its own stream read and does not end the pass; on a solid archive the later members of its block fail the same way. The pass raises only when damage to the archive's structure stops it finding the next member, such as a TAR cut inside a member. `file_copy_streams=False` also yields `None` for a file that the archive stores as a copy of an earlier member (`extra["is_file_copy"]`, a RAR5 file copy). Its bytes and digests are its source's, `link_target_member`, so a caller that records hashes can reuse the source's. A solid RAR pass then keeps nothing for the copies; by default it keeps each source until the pass ends, in memory or in a temporary file within `SpoolLimits.max_bytes`. The yielded stream is forward-only on every format, with or without `seekable_members=True`: `seekable()` is `False`, `seek()` raises `io.UnsupportedOperation`, and `tell()` works. A nested archive read from it is a non-seekable source: `open_archive()` accepts it only with `streaming=True` and only for TAR, ZIP and single-file compressors. For a member you need to seek, nested archives included, use open() under `seekable_members=True`. On a default reader the pass owns the reader while it runs: open(), read(), get(), members(), members_report(), iteration and another pass raise `ArchiveyUsageError` inside the loop. On a reader opened with `concurrent_operations=True` they all work, from the loop body or from another thread, on every format, so the answer does not depend on the format. They share the pass's walk of the archive's headers, and a call that needs members the pass has not reached yet reads ahead for it. extract_all() works beside a pass too. Raises: - `ArchiveyUsageError` – The reader is closed; it is a streaming reader whose one pass has already started; or, without concurrent_operations=True, another pass or a random-access call is running. - `ResourceLimitError` – The listing exceeds ListingLimits. - `CorruptionError` – Damage to the archive's structure stops the pass from finding the next member. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded while the pass lists the archive. Errors in a member's data raise from that member's stream, not from the pass: CorruptionError for damage, EncryptionError, UnsupportedFeatureError and PackageNotInstalledError for data archivey cannot decode, and ResourceLimitError for a member that asks a decoder for more memory than DecoderLimits allows. ### `archivey.ArchiveReader` Bases: `ForwardArchiveReader` The public, read-only interface to an archive opened for random access. Returned by archivey.open_archive() with the default `streaming=False`. It has every ForwardArchiveReader method and adds four for lookup and random member access: members(), get(), open() and read(). Annotate against this type when the code needs those methods; concrete machinery lives in the internal `BaseArchiveReader` helper. Use in a `with` block. #### `get(name, default=None)` Look up a member by its normalized name, returning `default` if absent. This is the name-lookup entry point; open()/read() also accept a name directly. May trigger a scan; on a streaming reader raises `ArchiveyUsageError`, as does a `name` that is not a `str` (a `bytes` name or an `ArchiveMember`). With duplicate member names, returns the last (the one a sequential extraction would leave on disk). Raises: - `ArchiveyUsageError` – The reader is closed or streaming, or name is not a str. - `CorruptionError` – The scan finds the member list damaged, as members() raises it. - `ResourceLimitError` – The listing exceeds ListingLimits. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded while the archive is listed. #### `members()` All members as a list. May trigger a scan; raises `ArchiveyUsageError` on a streaming reader (there, use members_report(), which uses up the pass, or iterate stream_members() and ignore the streams; both apply ListingLimits). Raises terminal archive-level listing errors instead of returning an incomplete list. Raises: - `CorruptionError` – The archive is damaged partway through its member list (members_report() returns the members found before the damage instead). - `ResourceLimitError` – The listing exceeds ListingLimits. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on, recorded while the archive is listed. - `ArchiveyUsageError` – The reader is closed or streaming. #### `open(member)` Open a member as a binary stream, following symlinks/hardlinks. Accepts a member object or a name (an unknown name raises `KeyError`; a member object that was not yielded by this reader raises `ArchiveyUsageError` — same identity rule as `member in reader`). The caller is responsible for closing the returned stream. Returns an ArchiveStream (usable as `BinaryIO`). **Cost, when `reader.cost.access_cost` is `SOLID`** (solid 7z/RAR, any compressed tar): members share one compression run, so opening one decodes every member before it. Doing that for each member in turn is quadratic in the archive size. On a 7z or RAR reader, the first call that decodes again what an earlier one decoded logs one INFO record per reader on the `archivey.backends` logger saying so, and a compressed tar's backward seeks report `STREAM_REWIND_REDECOMPRESSES`. Prefer stream_members(), which decodes the run once. Raises: - `KeyError` – No member has that name. - `ArchiveyUsageError` – The member has no data of its own (a directory whose header declares none, a tombstone, a special entry); the member object came from another reader; another member stream is open and the reader was opened without concurrent_operations=True; or the reader is closed or streaming. - `LinkTargetNotFoundError` – The member is a link whose target is not a member of the archive. - `EncryptionError` – The member is encrypted and no password given decrypts it. - `CorruptionError` – The member's data is damaged, raised by open() or by the read of the returned stream that finds the damage (TruncatedError when it seems to end early). - `UnsupportedFeatureError` – The member uses a compression method or feature archivey cannot decode. - `PackageNotInstalledError` – Decoding the member needs an optional package that is not installed. - `ResourceLimitError` – The member asks a decoder for more memory than DecoderLimits allows, or a scan for the name exceeds ListingLimits. - `DiagnosticRaisedError` – A diagnostic that the DiagnosticPolicy in the config raises on. #### `read(member)` Read a member's full contents as `bytes` (unbounded — prefer open() or stream_members() for anything not known to be small). Carries open()'s solid-archive cost: on a `SOLID` archive this decodes every member preceding the requested one, so a loop over all members is quadratic. Use stream_members() for that. Raises: - `KeyError` – No member has that name. - `ArchiveyUsageError` – As open() raises it. - `ArchiveyError` – As open() and reading its stream raise it: LinkTargetNotFoundError, EncryptionError, CorruptionError, UnsupportedFeatureError, PackageNotInstalledError, ResourceLimitError or DiagnosticRaisedError. ### `archivey.ArchiveStream` Bases: `ReadOnlyIOStream` A member's bytes, as a read-only binary file. open(), stream_members() and open_stream() return one; there is no need to build one yourself. It is an `io.RawIOBase`, and `seekable()` says whether it can seek. Parameters: - **`name`** (`str`) – (computed property) The member's name inside its archive, or raise AttributeError. A member stream carries the name its archive gives it, as zipfile's ZipExtFile does, so open_archive(reader.open(member)) can match the member's extension during detection (.tar.br and other formats found only by name). It is a member name, not a filesystem path. A stream with no member (open_stream) has no name and raises, as io.BytesIO does. - **`diagnostics`** (`DiagnosticSummary`) – (computed property) Diagnostic snapshot for events this stream's own calls emitted, or empty. - **`size`** (`int | None`) – (computed property) Total decompressed byte length when cheaply known, else None. A member stream knows its size from the archive's metadata. A stream from open_stream() reports it when the format stores it where it can be read without decompressing, and None otherwise: a BGZF file reports None until a seek or a read finds its size. A stream from stream_members() that has not opened yet reports None rather than opening itself just to answer. - **`verified`** (`Verification`) – (computed property) How much of this stream's data has been checked so far, as a Verification. KEY once the stream has opened its member and nothing doubts its password, STRUCTURE once a read has reached the end with the length checks passing, and CONTENT once a checksum stored with the data has matched every byte. It never goes down: a seek back to the start runs the checks again, and what they proved stays proved. A read that raises leaves it where it was. A stream from stream_members() opens its member at its first read, and is NONE until then. Compare it with verifiable: a TAR member has no checksum, so STRUCTURE is the most it can reach. Reading this property reads nothing from the archive. - **`verifiable`** (`Verification`) – (computed property) The most verified can reach by a read from the start to the end. CONTENT when the member carries a checksum archivey can compute, STRUCTURE when it carries none, and NONE for an encrypted member with no checksum whose password passed only a weak check. It rises when another member's checksum confirms that password. A stream from stream_members() learns what its member carries when it opens, at its first read. Until then it answers from what it knows without opening (the member's stored digest), and the answer can settle lower when it opens: a password watch that only appears then takes a member with no checksum from STRUCTURE to NONE. Once open, it never falls. Reading this property reads nothing from the archive. #### `close()` Close the stream and release what it holds in the reader. Closing it again does nothing. #### `nearest_resume_offset(target)` The offset, at or before `target`, that a seek to `target` decodes from. `None` when the stream cannot say. Which offsets a stream can resume from depends on the decoder, so the answer for the same member can change between releases. #### `read(n=-1)` Read up to `n` bytes of the member; `-1` or `None` reads to EOF. A stream that checks the member's length caps each read at the bytes left while it is still checking. Otherwise `n` goes to the decoder as given, so what a very large `n` costs is the decoder's: a TAR member's reader allocates a buffer of `n` bytes, and `read(2**62)` raises `MemoryError` there, as `open(path, "rb").read(2**62)` does on a plain file. The stream is still usable afterwards. #### `read_ahead(need, n)` At least `need` bytes, or fewer only at the end, and at most `n`. For a reader's own buffer over a codec stream (the TAR walk on a source that cannot seek): the inner's `read_ahead` gives what the feed that completes `need` decoded, so the buffer does not wait for bytes nobody asked for yet. Errors are those of `read(need)`. A stream that checks its bytes (a fused verifier) or has a verdict to repeat answers with `read(n)`. #### `seek(offset, whence=io.SEEK_SET)` Move to `offset`, as on any `io` stream; raises when it can't seek. Moving backwards in compressed data can mean decompressing again from the nearest place the decoder can resume from, which nearest_resume_offset() reports. #### `seekable()` Whether seek() works on this stream. `False` for a stream from stream_members(), and for one from open() unless the archive was opened with `seekable_members=True`. #### `tell()` The current position in the member's decompressed bytes. ### `archivey.Verification` Bases: `IntEnum` How much of a member stream's data has been checked, from least to most. `ArchiveStream.verified` is the level a stream has reached, and `ArchiveStream.verifiable` the most it can reach by a read from the start to the end. The levels are ordered, so `stream.verified >= Verification.STRUCTURE` reads as "at least the structure was checked". Members: - **`NONE`** (`0`) – Nothing is checked yet. The stream has not opened its member, or the member's password passed only a check that a wrong password can pass by chance, such as the one-byte check of old ZIP encryption. - **`KEY`** (`1`) – The bytes come from the right key: the member is not encrypted, or its password passed a check a wrong password cannot pass by chance. - **`STRUCTURE`** (`2`) – The right key, and a read reached the end with the length and every end-of-data check passing. - **`CONTENT`** (`3`) – The structure, and a checksum stored with the data matched every byte: the member's CRC-32 or BLAKE2sp, the WinZip AES authentication code, or the checksum of a gzip, bzip2, lzip or zlib stream read from its start. ### `archivey.MemberPass` Bases: `Iterator[_T_co]` A pass over an archive's members: an iterator that can also be closed, directly or by a `with` block. stream_members() returns one that yields `(member, stream)` pairs, and iterating the reader returns one that yields members. Iterate it as any iterator. Closing it ends the pass early and releases the reader: the current member stream, if any, is closed. Closing does not give a streaming reader a second pass: iterating it, stream_members() and extract_all() still raise `ArchiveyUsageError`, and members_report() finishes the interrupted walk. Closing an ended pass does nothing, and advancing a closed one ends the iteration. Closing a pass while it is executing (another thread inside `next()`, or a selector closing its own pass) raises `ArchiveyUsageError` and leaves the pass running. A pass that runs to its end ends on its own, and so, on CPython, does one a `for` loop leaves with `break`. Otherwise a pass stays alive while anything refers to it: a variable, the target of `with … as`, or a traceback that holds the loop's frame while an error is handled. It also outlives the loop on an interpreter that does not free it at once, such as PyPy. Close it yourself in those cases. While a paused stream_members() pass is alive, a default reader refuses its other calls; a pass from iterating a random-access reader holds nothing between members:: ```text with reader.stream_members() as members: for member, stream in members: if member.is_file and stream is not None: process(member, stream.read()) ``` #### `close()` End the pass: close the current member stream and release the reader. Raises `ArchiveyUsageError` while the pass is executing. ### `archivey.MemberSelector = MemberSelectorArg` Which members stream_members() and extract_all() act on. One of: - an iterable of member names (`str`) and ArchiveMember objects, read once, so a generator works. A name selects every member with exactly that name (a directory is `"dir/"`); an `ArchiveMember` selects that member only, by identity. A bare `str` is refused: pass `["name"]` for one member; - a predicate `Callable[[ArchiveMember], bool]` that returns `True` for the members to select; - `None`, for all members. ### `archivey.MemberListReport` Immutable member-list outcome: recovered members plus listing honesty. `error is None` means `members` is a complete archive listing. A non-`None` error means the tuple is the recovered prefix and the error is the terminal archive-level damage that stopped listing. Like ExtractionReport, the report iterates, indexes, and sizes as its primary sequence so common `for member in report` / `len(report)` idioms work. Fields: - #### **`members`** (`tuple[ArchiveMember, ...]`) – The members listed, all of them when error is None. - #### **`error`** (`ArchiveyError | None`) – The damage that stopped the listing, or None when it completed. - #### **`diagnostics`** (`DiagnosticSummary`) – The diagnostics emitted while listing. ## Members and archive details See [Open and list](https://archivey-dev.github.io/archivey/index.md#open-and-list) and [Members with the same name](https://archivey-dev.github.io/archivey/opening/index.md#members-with-the-same-name). ### `archivey.ArchiveMember` One archive entry. Mutable on purpose: backends fill late-bound fields in place after the member is first constructed (`link_target_member`, digests, attached diagnostics). Callers must treat instances as read-only — use replace() for edits. Fields: - #### **`type`** (`MemberType`) – What kind of entry this is (file, directory, symlink, …). - #### **`name`** (`str`) – Normalized member path, /-separated, decoded for display and lookup. - #### **`raw_name`** (`bytes | None`, default: `None` ) – The member name exactly as stored in the archive, undecoded. None when the format has no name field for this member or the bytes cannot be recovered from the decoded name; b"" when the format has the field and the member left it empty (7z). name is raw_name decoded and normalized, then put through the format's own naming rules. Where the archive stores the name twice, raw_name is the spelling the name was decoded from: a ZIP name taken from its Info-ZIP Unicode Path extra field (0x7075) has that field's UTF-8 bytes here, and the header's bytes in extra["alternate_raw_name"]. The naming rules that make name differ from raw_name decoded: ISO: in a plain ISO 9660 image, name drops the ;N version suffix (and the . before an empty extension) that raw_name keeps; the version is in extra["iso.version"]. A superseded version keeps its stored spelling. A Rock Ridge name that is not UTF-8 and takes its Joliet name keeps the Rock Ridge bytes here. A gzip file's member is named after the archive source ("data" when the source has no file name); raw_name holds the original file name from the gzip header (FNAME), and extra["gzip.original_filename"] holds those bytes decoded as Latin-1. A 7z member stored with no name is named after the archive source, and raw_name is b"". A RAR5 file-version-history member gets a ;n suffix in name (as WinRAR and unrar show it); raw_name is the stored name without it. - #### **`size`** (`int | None`, default: `None` ) – Uncompressed size in bytes, or None if unknown (e.g. a streaming entry). A single-file member (.gz, .bz2, …) whose format records no size gets one from a read: a read from the start to the end that skips no byte, which the decoder finishes without error sets it to the bytes it returned. A large member of a macOS ZIP, whose size the archive stores only modulo 2\*\*32, is listed with None and filled the same way, by a read whose length and CRC-32 check out. Until then member_state_final is False. - #### **`compressed_size`** (`int | None`, default: `None` ) – Compressed size in bytes, or None if unknown. - #### **`modified`** (`datetime | None`, default: `None` ) – Last-modified time, if recorded. - #### **`accessed`** (`datetime | None`, default: `None` ) – Last-access time, if recorded. - #### **`created`** (`datetime | None`, default: `None` ) – The birth (creation) time, if recorded (rare; most formats store only mtime). Never Unix st_ctime (inode change), in any format. Where a writer stores st_ctime, or may, that time is ctime instead: a creation slot filled on Unix (a RAR or 7z made there, a ZIP from any host but FAT, OS/2, NTFS or VFAT, an unknown host too) and the Rock Ridge and PAX change times. A TAR gets created only from libarchive's PAX LIBARCHIVE.creationtime. Directory listing uses st_birthtime only. - #### **`ctime`** (`datetime | None`, default: `None` ) – The Unix inode change time (st_ctime), if recorded, or a time that may be it. From an archive: the Rock Ridge attribute-change time (ISO), the PAX ctime record (TAR), or a creation slot whose writer fills it from st_ctime or cannot be shown not to (RAR, 7z, ZIP; see created). It is what the archive stored, so a Windows writer can put a birth time here: libarchive on Windows fills the PAX ctime with it and stamps its ZIP and 7z members as Unix. A format with a single creation slot (RAR, 7z, ZIP) fills at most one of created and ctime. Rock Ridge and libarchive's PAX records store both times separately and so can fill both, as can a directory listing. member.created or member.ctime prefers the birth time when there is one. A directory listing reports st_ctime on every OS but Windows, where Python's st_ctime was the creation time before 3.12. Extraction cannot restore it: every OS sets a file's st_ctime itself. - #### **`mode`** (`int | None`, default: `None` ) – Unix permission bits, or None if the format/entry carries no mode. - #### **`uid`** (`int | None`, default: `None` ) – Owner user id, if recorded. - #### **`gid`** (`int | None`, default: `None` ) – Owner group id, if recorded. - #### **`uname`** (`str | None`, default: `None` ) – Owner user name, if recorded. - #### **`gname`** (`str | None`, default: `None` ) – Owner group name, if recorded. - #### **`link_target`** (`str | None`, default: `None` ) – For a symlink/hardlink, the raw target path string as stored. For a FILE that is a stored copy of an earlier member (extra["is_file_copy"]), the source's path as stored. - #### **`link_target_member`** (`ArchiveMember | None`, default: `None` ) – For a link, the resolved target member within this archive, if found. For a FILE that is a stored copy (extra["is_file_copy"]), the member whose bytes it repeats. - #### **`compression`** (`tuple[CompressionMethod, ...]`, default: `tuple()` ) – Codec chain in compress order — pre-filters first, packing codec last. - #### **`is_encrypted`** (`bool`, default: `False` ) – Whether this member's data is encrypted. True also when the backend could not rule encryption out. A RAR5 member whose header stopped part-way through its optional records is reported this way, because answering "not encrypted" from a header nobody finished reading would be a wrong answer rather than a missing one; the member then carries a MEMBER_HEADER_RECORD_SKIPPED diagnostic saying the header was cut short. A member with no such diagnostic is a definite answer. - #### **`is_current`** (`bool`, default: `True` ) – Last-entry-wins: True for the live final state of this path. Duplicate names keep earlier rows with is_current=False (history / superseded). get() returns the current one. Stamped when the listing is complete. A stream_members() pass sets it after its last member; one that went past ListingLimits stamps only the members it kept, so a copy it yields after the limit cannot mark an earlier one superseded. - #### **`is_sparse`** (`bool`, default: `False` ) – Whether this member is stored as a sparse file. - #### **`comment`** (`str | None`, default: `None` ) – Per-member comment, if the format records one. - #### **`create_system`** (`CreateSystem | None`, default: `None` ) – The operating system that wrote the entry. For ZIP and RAR it decides how the stored mode and attributes are read. None for formats that do not record one; see CreateSystem for what each format sets. - #### **`windows_attrs`** (`int | None`, default: `None` ) – Raw Windows file-attribute bitmask, if recorded. - #### **`hashes`** (`Mapping[HashAlgorithm, bytes]`, default: `dict()` ) – Stored content digests keyed by HashAlgorithm (values always bytes). CRC-32 is four big-endian bytes: a CRC you computed yourself compares as zlib.crc32(data).to_bytes(4, "big"). Excluded from equality. - #### **`extra`** (`MemberExtra`, default: `MemberExtra()` ) – Format-specific extra fields (e.g. extra["is_junction"]). Excluded from equality. Known keys and their value types are MemberExtra. Unknown keys (third-party or future) stay legal and read as object. The EXTRA\_\* constants on this module remain the names for the keys they cover. - #### **`diagnostics`** (`tuple[Diagnostic, ...]`) – (computed property) Read-only tuple of diagnostics attached to this member (may be empty). - #### **`member_id`** (`int`) – (computed property) - #### **`is_file`** (`bool`) – (computed property) - #### **`is_dir`** (`bool`) – (computed property) - #### **`is_link`** (`bool`) – (computed property) - #### **`is_other`** (`bool`) – (computed property) - #### **`is_anti`** (`bool`) – (computed property) - #### **`is_junction`** (`bool`) – (computed property) The archive recorded this symlink as a Windows NTFS junction. False means no junction was detected, not that the entry is not one: RAR names the link kind in a header field, so a RAR junction is reported while listing. A directory source asks the filesystem (os.DirEntry.is_junction), so a scan on Windows reports junctions under Python 3.12 and later; under 3.11 a junction is listed as a symlink and reads False. ZIP and 7z keep the junction's reparse tag in the member's data, so they report one only when the writer stored that reparse buffer. 7-Zip stores none for a directory reparse point, and a junction is always one, so a junction in an archive 7-Zip wrote reads False here. TAR and ISO have no junction concept and always read False. is_reparse_point comes from metadata the archive always carries, and is the check to use when a Windows link of either kind matters. - #### **`is_reparse_point`** (`bool`) – (computed property) The archive recorded this entry as a Windows symlink or junction. Deliberately not gated on type the way is_junction is: a member the archive flags as a reparse point whose data turns out not to be a link buffer is presented as an ordinary file or directory, and the flag still records what the archive said about it. - #### **`member_state_final`** (`bool`) – (computed property) No field of this member will change from here on. False while archivey can still update the member in place, which happens in three cases: A member of a listing whose links have not been resolved yet. That listing is complete, and the member settled, when a call that resolves links returns: members(), members_report() or get(), or the end of a stream_members() pass or of iterating the reader. Until then a later member with the same name can still make this one is_current=False, and links get link_target_member. A streaming=True pass yields every member before that point, in every format. A members_report_if_available() peek resolves no links, so its members read False until a later call or pass does. A random-access pass with no listing call first yields from a complete listing in some formats and walks as it goes in others, so read the flag rather than assume. A symlink whose target is stored as member data (ZIP, 7z, a RAR 4 symlink) that nothing has read yet, which is the case under ArchiveyConfig.read_link_targets=False. Reading it sets link_target, and a Windows reparse point whose data is not a link buffer becomes a FILE. open() and extract_all() read it when they reach the link, on the member they are given, a copy included. A member whose size is None until a full read fills it in: a single-file member (.gz, .bz2, …) whose format records no size, read from the start to the end without skipping a byte; or a large member of a macOS ZIP whose size the archive stores only modulo 2\*\*32, read to the end with a matching CRC-32. A read that skips bytes, stops early or fails leaves it False. Everywhere else it is True. Closing the reader makes it True for every member it listed, since a closed reader updates nothing. So does the end of a streaming=True pass, or its failure, for a size no read filled: nothing can read the member again. A copy made with replace() is never stamped, so only its unread link target can change, except the copy a streaming extract_all hands its filter: it reads False while the walk can still change the member it came from, and the filter may then be called again for that member. Attached diagnostics are not fields and can still grow. Which fields change in practice is listed per format in the user docs. - #### **`link_target_unrecorded`** (`bool`) – (computed property) The archive records no target for this link. Some writers store a link with nowhere to point: 7-Zip writes a directory symlink or junction this way, and a Windows reparse buffer can name nothing. Such a link has link_target None, open() raises LinkTargetNotFoundError with reason NOT_RECORDED for it, and extraction records it as ExtractionStatus.LINK_TARGET_UNAVAILABLE rather than as a failure. False does not mean the target is known. link_target can also be None because the archive stores a target this read could not produce (damaged, encrypted, or out of reach), or because the target is stored as member data that nothing has read yet (ArchiveyConfig.read_link_targets is False). This property is True only when the archive itself left the target out. #### `modified_utc(tz_for_naive=None)` The modification time as a timezone-aware UTC `datetime`, or `None`. `modified` itself is faithful to what the archive stores: **naive** when the format records local wall-clock time (ZIP's DOS field, RAR4), **aware** when it records UTC or an offset — so naive and aware values from one archive cannot be compared or sorted directly. This helper makes that usable: an aware value is converted to UTC; a naive one first gets `tz_for_naive` attached (the caller's explicit assumption about where the archive was created), defaulting to the local timezone when not given. Whether the stored value was wall-clock remains visible on the field itself: `member.modified.tzinfo is None`. #### `replace(**kwargs)` Return a copy with the given fields changed; never mutates self. `object` is not a check: neither the keyword names nor the value types are verified statically. A misspelled field is still a `TypeError` from `dataclasses.replace` at runtime; a wrongly typed value is accepted, and shows up only where it is used. Typing the names would need a per-field `TypedDict`, which is deliberately deferred. No listing holds the copy, so nothing stamps `is_current` on it or resolves its links to members; its `member_state_final` waits only on an unread link target, which reading the copy fills in place (the copy `extract_all` hands a filter is the exception: see `member_state_final`). Closing the reader cannot reach a copy, so a copy whose target is never read keeps reading `False`. A read fills a late `size` into the reader's own member, never into a copy. ### `archivey.ArchiveInfo` Archive-level metadata, available immediately after `open_archive()` without a full member scan. Fields: - #### **`format`** (`ArchiveFormat`) – The detected (container, stream) format of the archive. - #### **`format_version`** (`str | None`) – Format version string, e.g. "4.5" for ZIP or "5" for RAR5; None if unknown. - #### **`is_solid`** (`bool`) – Whether decompressing one member may require decompressing earlier ones. - #### **`member_count`** (`int | None`) – Number of members, or None when a count would require scanning the whole archive. Also None when the archive cannot supply a count: a ZIP whose central directory is damaged, or holds more entries than ListingLimits.max_members. Then members() raises too. A ZIP's count comes from walking its central directory, which reading this field runs if the listing has not. - #### **`comment`** (`str | None`) – Archive-level comment, if the format records one. - #### **`is_encrypted`** (`bool`) – Header-level encryption (7z, RAR5) — not per-member encryption (see ArchiveMember.is_encrypted). - #### **`is_multivolume`** (`bool`) – Whether the archive spans multiple volumes. - #### **`cost`** (`CostReceipt`) – Listing/access cost receipt for the archive (see the access-mode-and-cost capability). - #### **`extra`** (`ArchiveInfoExtra`, default: `ArchiveInfoExtra()` ) – Format-specific archive-level metadata, keyed by namespaced strings (mirrors ArchiveMember.extra). For example the ISO backend records the auto-selected namespace as extra["iso.namespace"]. Excluded from __eq__. Known keys and their value types are ArchiveInfoExtra; unknown keys stay legal and read as object. ### `archivey.HashAlgorithm` Bases: `StrEnum` Digest algorithms that may appear as keys in ArchiveMember.hashes. Members: - **`CRC32`** (`'crc32'`) – CRC-32, as four big-endian bytes. - **`BLAKE2SP`** (`'blake2sp'`) – BLAKE2sp, 32 bytes. Only RAR5 stores it. ### `archivey.MemberExtra` Bases: `dict[str, object]` Per-member format-specific metadata on ArchiveMember. A `dict[str, object]` whose known keys return their declared types from a subscript (`extra["zip.compress_type"]` is an `int`). Unknown keys (third-party or future) stay legal and read as `object`. The `EXTRA_*` constants on this module still name the keys they cover. Writes are not type-checked: a wrong-type assignment to a known key falls through to the `str → object` fallback, same as an unknown key. `.get()` returns `object` for every key. Assign a `MemberExtra({...})` (or mutate the existing bag); a bare dict is not assignable to the field. Known keys: - `is_junction` (`bool`) — ZIP, 7z, RAR, directory. A Windows NTFS junction; ZIP and 7z set it only when the writer stored the junction's reparse data, and 7-Zip does not. Implies `is_reparse_point`. - `is_reparse_point` (`bool`) — ZIP, 7z, RAR, directory. The weaker, metadata-only sibling of `is_junction`: a Windows symlink or junction rather than a POSIX one. - `is_file_copy` (`bool`) — RAR. A `FILE` member whose bytes the archive stores once under an earlier member (a RAR5 file reference, `rar -oi`). `link_target` names that source and `link_target_member` is it. - `special_file_type` (`"fifo" | "char_device" | "block_device" | "socket" | "unknown"`) — every format. What the entry's stored Unix mode (ZIP, 7z, RAR, Rock Ridge) or TAR typeflag named when that is a device, FIFO or socket; `"unknown"` is a mode whose file-type bits no Unix type uses. Set whatever type the member ends up with: the stream-less `OTHER` such an entry usually is, the `FILE` it becomes when it carries data (its bytes are the content, as unzip, 7-Zip, bsdtar and `zipfile` deliver them), a RAR file copy, or the `DIRECTORY` or `SYMLINK` a structural marker made of it (a ZIP name ending in `/`, an ISO directory record, a reparse-point bit) over such a mode. The key says what the mode named; the type says what the archive's structure decided. Absent on an `OTHER` member that is not a special file (a GNU dumpdir in TAR). TAR keeps `tar.type` as well. - `alternate_raw_name` (`bytes`) — ZIP. The other stored spelling of the name, when the archive stores two and `raw_name` is the one `name` was decoded from: for a ZIP name taken from its Info-ZIP Unicode Path extra field (0x7075), the header's legacy bytes. - `size_mod_2_32` (`int`) — ZIP. The member's size modulo 2³², when that is all the archive stores: a large member of a ZIP macOS Archive Utility wrote without ZIP64, whose real size only a read can tell. Set when `size` is listed as `None`; a read to the end checks the length against it and fills in `size`, and the key stays. - `rar.extract_version` (`int`) - `rar.file_version` (`int`) - `rar.tweaked_crc32` (`int`) - `rar.tweaked_blake2sp` (`bytes`) - `zip.compress_type` (`int`) - `zip.aes_vendor_version` (`int`) - `zip.aes_strength` (`int`) - `zip.aes_actual_method` (`int`) - `zip.extra_fields` (`dict[int, bytes]`) — the central directory extra fields archivey took nothing from, by tag (the first of several with one tag), such as a vendor's own field or one archivey knows but found malformed. Absent when there are none, which is the usual case. When a later version learns to read a field, it leaves this dict. A pass over a pipe sets it once it reaches the central directory, at the end. - `tar.type` (`bytes`) - `tar.pax_headers` (`Mapping[str, str]`) — the member's PAX records, the global ones in force included. Read-only: a change raises `TypeError`, and `dict(...)` gives a copy to change. It is a `dict` subclass, so `json.dumps` takes it. - `tar.devmajor` (`int`) — set only for a character or block device (typeflag `3` or `4`) in a ustar or GNU header whose field is a number; absent for a FIFO, for a v7 header, and for a device whose field is not a number (that member carries `MEMBER_HEADER_RECORD_SKIPPED`). - `tar.devminor` (`int`) — present exactly when `tar.devmajor` is. - `gzip.original_filename` (`str`) - `gzip.isize` (`int`) — the gzip trailer's ISIZE field: the last four bytes of the file, read only when its end is cheap to reach (not from a pipe or another archive's member). For a file of one gzip member with nothing after it, this is the uncompressed size modulo 2\*\*32. It is not the size for a file of several members (it covers only the last; a bgzip file ends with an empty member and reads 0) or with bytes after the last one. `size` stays `None` until a full read fills it in. - `iso.version` (`int`) — plain ISO 9660 only: the `;N` file version stripped from the name. Versions of one name share it; the highest is listed last and is the current one. #### `__getitem__(key)` ```python __getitem__(key: Literal['is_junction']) -> bool ``` ```python __getitem__(key: Literal['is_reparse_point']) -> bool ``` ```python __getitem__(key: Literal['is_file_copy']) -> bool ``` ```python __getitem__( key: Literal["special_file_type"], ) -> SpecialFileType ``` ```python __getitem__(key: Literal['alternate_raw_name']) -> bytes ``` ```python __getitem__(key: Literal['size_mod_2_32']) -> int ``` ```python __getitem__(key: Literal['rar.extract_version']) -> int ``` ```python __getitem__(key: Literal['rar.file_version']) -> int ``` ```python __getitem__(key: Literal['rar.tweaked_crc32']) -> int ``` ```python __getitem__(key: Literal['rar.tweaked_blake2sp']) -> bytes ``` ```python __getitem__(key: Literal['zip.compress_type']) -> int ``` ```python __getitem__(key: Literal['zip.aes_vendor_version']) -> int ``` ```python __getitem__(key: Literal['zip.aes_strength']) -> int ``` ```python __getitem__(key: Literal['zip.aes_actual_method']) -> int ``` ```python __getitem__( key: Literal["zip.extra_fields"], ) -> dict[int, bytes] ``` ```python __getitem__(key: Literal['tar.type']) -> bytes ``` ```python __getitem__( key: Literal["tar.pax_headers"], ) -> Mapping[str, str] ``` ```python __getitem__(key: Literal['tar.devmajor']) -> int ``` ```python __getitem__(key: Literal['tar.devminor']) -> int ``` ```python __getitem__(key: Literal['gzip.original_filename']) -> str ``` ```python __getitem__(key: Literal['gzip.isize']) -> int ``` ```python __getitem__(key: Literal['iso.version']) -> int ``` ```python __getitem__(key: str) -> object ``` ### `archivey.ArchiveInfoExtra` Bases: `dict[str, object]` Archive-level format-specific metadata on ArchiveInfo. Same shape as MemberExtra over a separate key set — do not merge the two bags. Known keys: - `iso.namespace` (`str`) - `zip.volume_count` (`int`) - `rar.volume_count` (`int`) - `7z.volume_count` (`int`) #### `__getitem__(key)` ```python __getitem__(key: Literal['iso.namespace']) -> str ``` ```python __getitem__(key: Literal['zip.volume_count']) -> int ``` ```python __getitem__(key: Literal['rar.volume_count']) -> int ``` ```python __getitem__(key: Literal['7z.volume_count']) -> int ``` ```python __getitem__(key: str) -> object ``` ## Formats and member types See [Format](https://archivey-dev.github.io/archivey/opening/index.md#format). ### `archivey.ArchiveFormat` A `(container, stream)` pair identifying how an archive is packaged. Prefer the named class attributes (`ArchiveFormat.ZIP`, `ArchiveFormat.TAR_GZ`, …) over constructing pairs by hand. Those names are assigned immediately below the class body; the `ClassVar` declarations exist so type checkers see them without per-use suppressions. `_FORMAT_NAMES` is built from the same assignments so `repr` / `display_name` stay in sync automatically. A pair built by hand from strings — `ArchiveFormat("raw_stream", "gz")`, say, from a format round-tripped through a config file — is converted to the enum members at construction, with the same spellings the other enum arguments take, and an unknown spelling raises ArchiveyUsageError there. Fields: - #### **`container`** (`ContainerFormat | PluginContainer`) – The archive format, such as ZIP or TAR, or a member of a PluginContainer subclass for a format a plugin adds. - #### **`stream`** (`StreamFormat`) – The compression around the archive, such as GZIP for a .tar.gz, or UNCOMPRESSED. - #### **`display_name`** (`str`) – (computed property) Human-readable name for this format, e.g. "ZIP", "TAR_GZ". Uses the predefined named-instance attribute name (ZIP, TAR_GZ, …). A pair with no named constant, such as tar.lz, gets CONTAINER_STREAM from the two enum member names ("TAR_LZIP"), or the container name alone when the stream is UNCOMPRESSED (a plugin's "CPIO"); its repr() stays the explicit constructor form. \_FORMAT_NAMES is populated just after the class definition — safe at runtime because this property is never called before the module is fully loaded. Members: - **`ZIP`** (`ContainerFormat.ZIP, StreamFormat.UNCOMPRESSED`) - **`TAR`** (`ContainerFormat.TAR, StreamFormat.UNCOMPRESSED`) - **`TAR_GZ`** (`ContainerFormat.TAR, StreamFormat.GZIP`) - **`TAR_BZ2`** (`ContainerFormat.TAR, StreamFormat.BZIP2`) - **`TAR_XZ`** (`ContainerFormat.TAR, StreamFormat.XZ`) - **`TAR_ZST`** (`ContainerFormat.TAR, StreamFormat.ZSTD`) - **`TAR_LZ4`** (`ContainerFormat.TAR, StreamFormat.LZ4`) - **`GZ`** (`ContainerFormat.RAW_STREAM, StreamFormat.GZIP`) - **`BZ2`** (`ContainerFormat.RAW_STREAM, StreamFormat.BZIP2`) - **`XZ`** (`ContainerFormat.RAW_STREAM, StreamFormat.XZ`) - **`ZST`** (`ContainerFormat.RAW_STREAM, StreamFormat.ZSTD`) - **`LZ4`** (`ContainerFormat.RAW_STREAM, StreamFormat.LZ4`) - **`LZIP`** (`ContainerFormat.RAW_STREAM, StreamFormat.LZIP`) - **`LZMA_ALONE`** (`ContainerFormat.RAW_STREAM, StreamFormat.LZMA_ALONE`) - **`ZLIB`** (`ContainerFormat.RAW_STREAM, StreamFormat.ZLIB`) - **`BROTLI`** (`ContainerFormat.RAW_STREAM, StreamFormat.BROTLI`) - **`Z`** (`ContainerFormat.RAW_STREAM, StreamFormat.UNIX_COMPRESS`) - **`SEVEN_Z`** (`ContainerFormat.SEVEN_Z, StreamFormat.UNCOMPRESSED`) - **`RAR`** (`ContainerFormat.RAR, StreamFormat.UNCOMPRESSED`) - **`ISO`** (`ContainerFormat.ISO, StreamFormat.UNCOMPRESSED`) - **`DMG`** (`ContainerFormat.DMG, StreamFormat.UNCOMPRESSED`) - **`DIRECTORY`** (`ContainerFormat.DIRECTORY, StreamFormat.UNCOMPRESSED`) - **`UNKNOWN`** (`ContainerFormat.UNKNOWN, StreamFormat.UNCOMPRESSED`) #### `file_extension()` The on-disk file extension for this format, without a leading dot. Used for extension-based naming and detection — e.g. choosing the output filename when converting between formats, or matching by extension in the detector. Examples: `ZIP` -> `"zip"`, `TAR_GZ` -> `"tar.gz"`, `GZ` -> `"gz"`. Formats with no on-disk file representation (`DIRECTORY`, `UNKNOWN`) return `""`. ### `archivey.ContainerFormat` Bases: `StrEnum` The archive format that holds the members, the first half of an ArchiveFormat. Members: - **`ZIP`** (`'zip'`) - **`TAR`** (`'tar'`) - **`RAR`** (`'rar'`) - **`SEVEN_Z`** (`'7z'`) - **`ISO`** (`'iso'`) – An ISO 9660 disc image. - **`DMG`** (`'dmg'`) – An Apple disk image. archivey recognises one so that opening it can refuse it by name, but cannot read it. - **`DIRECTORY`** (`'directory'`) – A folder on disk, read as if it were an archive. - **`RAW_STREAM`** (`'raw_stream'`) – No container: a single compressed file, such as a .gz. The StreamFormat says which compressor. - **`UNKNOWN`** (`'unknown'`) – What detection reports when no format matches. Passing a format with this container to open_archive() or open_stream() raises ArchiveyUsageError. ### `archivey.PluginContainer` Bases: `StrEnum` Base class for a container enum that a format plugin defines. A plugin subclasses it with one member per container it adds, and builds its ArchiveFormat from that member:: ```text class CpioContainer(PluginContainer): CPIO = "cpio" CPIO = ArchiveFormat(CpioContainer.CPIO, StreamFormat.UNCOMPRESSED) ``` Provisional, with the rest of `archivey.formats`. Code that matches on `ArchiveFormat.container` needs a default case once a plugin is registered. ### `archivey.StreamFormat` Bases: `StrEnum` The compression around the whole container, the second half of an ArchiveFormat. Each value is the format's file extension. Members: - **`UNCOMPRESSED`** (`'uncompressed'`) – No compression around the container. ZIP, 7z, RAR, ISO and a plain TAR use this; their members can still be compressed inside. - **`GZIP`** (`'gz'`) - **`BZIP2`** (`'bz2'`) - **`XZ`** (`'xz'`) - **`ZSTD`** (`'zst'`) - **`LZ4`** (`'lz4'`) - **`LZIP`** (`'lz'`) - **`LZMA_ALONE`** (`'lzma'`) – The legacy .lzma file format, also called LZMA Alone. - **`ZLIB`** (`'zz'`) - **`BROTLI`** (`'br'`) – Brotli has no signature, so archivey detects it by the file name. - **`UNIX_COMPRESS`** (`'Z'`) – The Unix compress format. ### `archivey.MemberType` Bases: `Enum` Kind of archive entry. Members: - **`FILE`** (`'file'`) – A regular file. A device, FIFO or socket entry that carries data is a FILE too, with extra["special_file_type"] saying what the archive called it. - **`DIRECTORY`** (`'directory'`) - **`SYMLINK`** (`'symlink'`) – A symbolic link. link_target holds the target as stored. - **`HARDLINK`** (`'hardlink'`) – A hard link to another member, whose data it shares. link_target names that member, and link_target_member is the member when the archive has it. - **`OTHER`** (`'other'`) – An entry with no data that is not a file, directory or link, such as a device, FIFO or socket. Extraction refuses it under every policy. - **`ANTI`** (`'anti'`) – A deletion marker from a 7z incremental update, saying that a file was removed. It is not a file (is_file is false), and extraction skips it. ### `archivey.CompressionMethod` One codec in a member's filter chain. Members store `tuple[CompressionMethod, ...]`. Order matches the compress / pack direction: pre-filters first, packing codec last (closest to the stored bytes). Example: 7z `(BCJ2, LZMA2)` — decompress by applying LZMA2, then BCJ2. Fields: - #### **`algo`** (`CompressionAlgorithm`) – The method or filter. - #### **`level`** (`int | None`, default: `None` ) – RAR only: the compression level, from 1 (fastest) to 5 (best), as rar -m1 to -m5 set it. A stored RAR member is STORED with no level, and other formats leave it None. - #### **`properties`** (`bytes | None`, default: `None` ) – 7z only: the coder's properties as stored, such as LZMA's dictionary size. Other formats leave it None. ### `archivey.CompressionAlgorithm` Bases: `Enum` A compression method or filter in a member's CompressionMethod chain. Later versions may add members, so treat the set as open-ended. Members: - **`STORED`** (`'stored'`) – No compression. - **`DEFLATE`** (`'deflate'`) - **`DEFLATE64`** (`'deflate64'`) - **`BZIP2`** (`'bzip2'`) - **`LZMA`** (`'lzma'`) - **`LZMA2`** (`'lzma2'`) - **`ZSTD`** (`'zstd'`) - **`LZ4`** (`'lz4'`) - **`BROTLI`** (`'brotli'`) - **`PPMD`** (`'ppmd'`) - **`XZ`** (`'xz'`) – A complete .xz stream, with its own header and checks, as a ZIP member with method 95 stores it. - **`BCJ`** (`'bcj'`) – A filter that makes x86 executable code compress better. It does not compress by itself. - **`BCJ2`** (`'bcj2'`) – 7z's variant of BCJ, which splits the code into four streams. - **`DELTA`** (`'delta'`) – A filter that stores the difference between bytes a fixed distance apart. It does not compress by itself. - **`RAR`** (`'rar'`) – RAR's own compression, any version. Not the same as ContainerFormat.RAR, which is the archive format. - **`UNKNOWN`** (`'unknown'`) – A method archivey does not recognise. Listing still works. ### `archivey.CreateSystem` Bases: `Enum` The operating system that wrote a member, in ArchiveMember.create_system. The values are ZIP's "version made by" host numbers. Other formats map their own host field onto them: - ZIP: any of these values, and `UNKNOWN` for a number not listed here. A pass over a pipe leaves it `None` until it reaches the central directory. - RAR 1.5 to 4: `FAT` (MS-DOS), `OS2_HPFS`, `WINDOWS_NTFS`, `UNIX`, `MACINTOSH` or `BEOS`; `UNKNOWN` for any other host byte. - RAR5: `WINDOWS_NTFS` or `UNIX`. - 7z records no host. archivey reports `UNIX` when the member stores a Unix mode, and `WINDOWS_NTFS` otherwise. - TAR, ISO, a directory and a single-file format such as `.gz` leave `create_system` as `None`. Members: - **`FAT`** (`0`) - **`AMIGA`** (`1`) - **`OPENVMS`** (`2`) - **`UNIX`** (`3`) - **`VM_CMS`** (`4`) - **`ATARI_ST`** (`5`) - **`OS2_HPFS`** (`6`) - **`MACINTOSH`** (`7`) - **`Z_SYSTEM`** (`8`) - **`CPM`** (`9`) - **`WINDOWS_NTFS`** (`10`) - **`MVS`** (`11`) - **`VSE`** (`12`) - **`ACORN_RISC`** (`13`) - **`VFAT`** (`14`) - **`ALTERNATE_MVS`** (`15`) - **`BEOS`** (`16`) - **`TANDEM`** (`17`) - **`OS_400`** (`18`) - **`OS_X_DARWIN`** (`19`) - **`UNKNOWN`** (`255`) ## Extraction See [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md). ### `archivey.ExtractionReport` Immutable extraction outcome: fixed result tuple plus diagnostic summary. `results` is a frozen outcome structure. Each ExtractionResult is frozen, but `ExtractionResult.member` refers to the live mutable ArchiveMember (caller-read-only), whose late-bound metadata and member diagnostics may still be filled in place. The report iterates, indexes, and sizes as its `results` sequence, so the common `for result in reader.extract_all(...)` / `len(...)` / `report[0]` idioms keep working while `report.diagnostics` exposes the call's diagnostic summary. It covers that ForwardArchiveReader.extract_all() call only; detection and open diagnostics stay on `reader.diagnostics`. Fields: - #### **`results`** (`tuple[ExtractionResult, ...]`) – One result per member processed, in the order they were processed. - #### **`diagnostics`** (`DiagnosticSummary`) – The diagnostics emitted during this extract_all() call. ### `archivey.ExtractionResult` One entry per member processed, returned from `extract_all()`. Fields: - #### **`member`** (`ArchiveMember`) – The member this result is about. It is the reader's own member, so fields that are filled in late may still change; treat it as read-only. - #### **`path`** (`Path | None`) – Where the member was written, or None if it was not. - #### **`status`** (`ExtractionStatus`) – What happened to the member. - #### **`error`** (`ArchiveyError | OSError | None`, default: `None` ) – Why the member failed or was blocked, when extraction went on past it. An OSError for a file system error on this member, otherwise an ArchiveyError. - #### **`requested_path`** (`Path | None`, default: `None` ) – Where the member was to be written before overwrite settled a collision. Usually the same as path. Under OverwritePolicy.RENAME the two differ when the member, or a directory it is in, was renamed (dd/f written at dd (1)/f). After a collision settled by SKIP or ERROR, this is set and path is None. On an OVERWRITTEN result it keeps the path this member wrote to, which a later member then took over. That later member's collided_with equals this path, whether this member wrote a file or a directory (outside ExtractionPolicy.TRUSTED), so join the two through collided_with: the later member's own path can differ from this one in case on a case-insensitive file system, or in spelling through a symlink of the archive's. None for a member refused or failed before its destination was settled. - #### **`presented_name`** (`str | None`, default: `None` ) – The member's name before a safety rewrite changed it on disk, or None when none did; rewrites says which ones ran. After a re-root this is the stored name. When a filter renamed the member, it is the filter's name, which neither member.name (the archive's spelling) nor path (the final one) shows. - #### **`rewrites`** (`frozenset[NameRewrite]`, default: `frozenset()` ) – The safety rewrites that changed the name. Empty exactly when presented_name is None. - #### **`failure_group_id`** (`str | None`, default: `None` ) – Set when one failed hardlink source makes several link results fail: those results share this id. Compare ids for equality only; their format and order mean nothing, and they differ between runs. - #### **`failure_group_size`** (`int | None`, default: `None` ) – How many results share failure_group_id. - #### **`collided_with`** (`Path | None`, default: `None` ) – The destination an earlier member of this run wrote, which this member collided with, whatever overwrite then did. None when no member of this run held the name: an entry already on disk before the run started is not counted, and stays in requested_path. The member it collided with is the result whose path equals this, or whose requested_path does once that result is OVERWRITTEN. A directory this run created on disk, from a DIRECTORY member or as the parent of an earlier member, holds its name too: a file landing on it records the directory's path. A directory that was only created as a parent has no result of its own. A directory that was already in dest is not counted, even when a DIRECTORY member of this run merged into it and reports it. - #### **`kept_mode`** (`int | None`, default: `None` ) – On a DIRECTORY result whose directory existed before the run, such as the destination itself for a ./ member: the mode that directory kept, when the member asked for another one. On Windows only the read-only attribute is compared. None otherwise. Such a directory keeps its times too, so None does not mean the member's times were applied. ### `archivey.ExtractionStatus` Bases: `StrEnum` The outcome recorded for a single member in its ExtractionResult. Members: - **`EXTRACTED`** (`'extracted'`) – Written to path. - **`NOT_OVERWRITTEN`** (`'not_overwritten'`) – Not written, because an entry was already at the destination and overwrite is OverwritePolicy.SKIP. - **`SUPERSEDED`** (`'superseded'`) – Not written, because a later member with the same name replaces it. - **`OVERWRITTEN`** (`'overwritten'`) – Written, then replaced by a later member under OverwritePolicy.REPLACE. The result changes to this when the later member arrives, so two members never both report EXTRACTED at one path, as when two names differ only in case. - **`BLOCKED`** (`'blocked'`) – Refused by a safety check or by the policy. - **`FAILED`** (`'failed'`) – Extraction failed: damaged data, a limit such as the expansion ratio, or a write error. - **`LINK_TARGET_UNAVAILABLE`** (`'link_target_unavailable'`) – A symlink whose target the archive never recorded, so it cannot be written. Not a failure: extraction goes on whatever on_error says. The archive is reported with the SYMLINK_TARGET_UNAVAILABLE diagnostic, which DiagnosticPolicy.STRICT raises. ### `archivey.NameRewrite` Bases: `StrEnum` A safety rewrite that changed a member's name before it reached disk. `ExtractionResult.rewrites` holds one value per rewrite that ran on the member, and `ExtractionResult.presented_name` holds the name before them. Both can run on one member: under `STANDARD`, `/a/what?.txt` is re-rooted to `a/what?.txt` and then written `a/what%3F.txt`. Members: - **`REROOTED`** (`'rerooted'`) – An absolute name was moved inside the destination by dropping its root (/etc/x extracts as etc/x). STRICT refuses such a name instead. - **`PORTABLE_NAME`** (`'portable_name'`) – The name was respelled so that every supported OS stores it the same way: a \\ written as /, a byte that is not UTF-8 or a character Windows refuses (\<>"|?\* and 0x01-0x1F) written as %XX, and, under STRICT only, a trailing dot or space removed from each segment. TRUSTED never rewrites a name. ### `archivey.ExtractionProgress` Progress snapshot for the `on_progress` callback. For FILE members the callback MAY fire multiple times as bytes are written (about once per copy chunk); each member still gets a terminal report with `member_bytes_written` equal to the member's size (or the final observed byte count when size is unknown). Directories, symlinks, and hardlinks produce a single report with `member_bytes_written == 0`. Fields: - #### **`member`** (`ArchiveMember`) – The member being extracted. - #### **`bytes_written`** (`int`) – Bytes written so far in the whole call. - #### **`total_bytes_estimated`** (`int | None`) – An estimate of the bytes the whole call writes, or None if the archive records no sizes. - #### **`members_done`** (`int`) – Members processed so far, whatever their outcome. - #### **`members_total`** (`int | None`) – Members the call processes, or None when counting them would need a scan of the whole archive. - #### **`member_bytes_written`** (`int`) – Bytes written so far for the current member. - #### **`members_extracted`** (`int`) – Members written so far. Like members_blocked, it leaves out a member still in progress. - #### **`members_blocked`** (`int`) – Members refused so far by a safety check or by the policy. ### `archivey.ExtractionPolicy` Bases: `Enum` How much of an archive member to trust when writing it to the destination. The policy decides how names and permissions are written. Some checks hold under every policy, because the write itself would be unsafe or would differ by OS: a name or link that leads outside the destination, a name with a NUL, a device node, and a symlink target with a Windows drive letter or UNC root (`C:/Windows`, `//server/share`), which Windows refuses and so is refused on every OS. A hardlink is refused when the member it takes its data from is. Members: - **`STRICT`** (`'strict'`) – The default, for archives from anywhere. A name is rewritten to a spelling every supported OS stores the same way, or refused when that is not possible: absolute names, Windows reserved names such as CON, names with :, and names with characters that reorder how they display are refused. Files keep their permissions without the setuid, setgid and sticky bits and without write access for group and others, and a file its owner cannot execute is executable by nobody. Directories are rwxr-xr-x, and owners are not applied. - **`STANDARD`** (`'standard'`) – For archives you mostly trust, such as your own. As STRICT, but trailing dots and spaces are kept (a folder named only dots or spaces, such as ..., is still refused), an absolute name is extracted inside the destination (/etc/x as etc/x, as tar and unzip do), and permissions are kept as stored without the setuid, setgid and sticky bits. - **`TRUSTED`** (`'trusted'`) – For archives you trust, to reproduce them faithfully. Names are written as stored, except that an absolute name is extracted inside the destination. Permissions are kept as stored, and owners too when running as root. A name built to display as something else is extracted, where the other policies refuse it: it lands inside the destination under its stored bytes, so the risk is to a person reading the directory, not to the file system. ### `archivey.OverwritePolicy` Bases: `Enum` What to do when a destination entry already exists where a member would be written. Members: - **`ERROR`** (`'error'`) – The default. The member fails with ExtractionError, and OnError decides whether extraction goes on. - **`SKIP`** (`'skip'`) – The existing entry stays, and the member's result is NOT_OVERWRITTEN. This is not a failure, so OnError does not apply. - **`REPLACE`** (`'replace'`) – The existing entry is deleted, then the member is written, so it is never written through a link that was there. A directory member merges into an existing directory. Any other member replaces an existing directory only if it is empty, and otherwise fails. - **`RENAME`** (`'rename'`) – The member is written next to it, with a number before the extension: a.txt becomes a (1).txt. ### `archivey.OnError` Bases: `Enum` What to do when an individual member cannot be extracted. It covers failures only: damaged data, a write error, an existing entry under `OverwritePolicy.ERROR`. A member refused by a safety check is recorded as `BLOCKED` and extraction goes on under either value; `AbortOn.BLOCKED_MEMBER` stops on one instead. Members: - **`STOP`** (`'stop'`) – The default. The first failure raises, and extraction stops there. - **`CONTINUE`** (`'continue'`) – The failure is recorded in the report as FAILED, its partial output is removed, and extraction goes on with the next member. ### `archivey.AbortOn` Bases: `StrEnum` Events that abort the whole extraction the first time they occur. Passed as `abort_on=` to `extract_all()` (a collection; empty by default). Independent of OnError and of `DiagnosticPolicy`: an event named here aborts whatever those are set to, and one not named here never aborts. The abort is immediate. The triggering member's partial output is removed, no later member is processed, and no `ExtractionReport` is returned. Output already written for earlier members stays on disk, as with `OnError.STOP`: an abort stops the run, it does not roll it back. The exception is a ZIP read from a pipe, whose pass stops before the central directory says which members are links: it removes the files it wrote that could still be links (see "A ZIP from a pipe" in the extraction guide). There is no member for extraction failures, because `OnError.STOP` already raises on the first one. Members: - **`BLOCKED_MEMBER`** (`'blocked_member'`) – A member is refused by a safety check or by the policy. Raises the FilterRejectionError that refused it. - **`NAME_COLLISION`** (`'name_collision'`) – Two members of the archive would be written to the same path. Raises NameCollisionError before overwrite is applied, so even with OverwritePolicy.RENAME the second member is not written. Not checked under ExtractionPolicy.TRUSTED. - **`NAME_SANITIZED`** (`'name_sanitized'`) – A name is rewritten to its portable spelling, or an absolute name is extracted inside the destination. Raises NameRewrittenError. Under ExtractionPolicy.STRICT an absolute name is refused instead, so BLOCKED_MEMBER is the one that catches it. No policy implies this: it is for callers who refuse any name on disk that differs from the archive's, such as mirroring tools. To list rewrites without stopping, read ExtractionResult.rewrites. ### `archivey.MemberFilter = Callable[[ArchiveMember], 'ArchiveMember | None']` ### `archivey.sanitize_names(member)` Rewrite a member's name so that extraction writes it instead of refusing it. Pass it as `filter=` to `extract_all()` (or call it from your own filter) to extract every member that has a safe place to go under a rewritten name, instead of refusing the members with an unsafe one. It changes: - a rooted name (`/etc/x`, `C:\x`, `\\host\share\x`): the root is dropped, so the member lands at `etc/x` inside the destination. `STANDARD` and `TRUSTED` already do this; under `STRICT` it happens only with this filter. - a `..` component: resolved against the segment before it (`a/../b` → `b`), and dropped when there is none (`../x` → `x`). - a bidirectional override or isolate character: removed. - a Windows-reserved device name (`CON`, `NUL.txt`) or a `:` in a segment: `_` is added after the reserved stem (`CON_`, `NUL_.txt`) and `:` becomes `_`. - a NUL character: becomes `_`. A hardlink's target is left as stored: it names a member, not a path. A symlink's target gets the character and segment rewrites only (`file:stream` → `file_stream`, `sub/NUL` → `sub/NUL_`), so a link to a rewritten member points at the name that was written. Its root and its `..` components are kept: it is a path on the filesystem, and one that points outside the destination is still refused. A symlink target with a Windows drive or UNC root (`C:/x`, `//host/share`) gets the character rewrites only, and is still refused. A symlink target the archive stores as member data and that is read only after this filter has run (`ArchiveyConfig.read_link_targets=False`, or a streaming pass) gets the same rewrite: `extract_all` calls the filter again once the target is read. Two members can end up with the same name (`a/../x` and `x`). The second is then handled by the `overwrite` option like any other clash. The result's `member.name` keeps the stored name and `path` shows where it was written. Returns `member` unchanged when nothing needed rewriting. ## Configuration and passwords See [Configuration](https://archivey-dev.github.io/archivey/opening/index.md#configuration), [Passwords](https://archivey-dev.github.io/archivey/opening/index.md#passwords) and [Hardening](https://archivey-dev.github.io/archivey/security/index.md#hardening). ### `archivey.ArchiveyConfig` Library tuning knobs passed as `config=` to open_archive(), open_stream() and detect_format(). Per-call operationals (`format`, `streaming`, `password`, extraction's `members`/`filter`/`policy`/…) stay keyword arguments — not fields here. Fields: - #### **`use_rapidgzip`** (`AcceleratorMode`, default: `AcceleratorMode.AUTO` ) – Whether to use the rapidgzip accelerator for gzip, zlib and raw deflate. It gives seekable member streams over those codecs, and decodes in parallel. Under AUTO it is used only when the compressed input is known to be at least RAPIDGZIP_AUTO_MIN_COMPRESSED_SIZE bytes (or its size is unknown) and the decompressed size can be verified, so a truncated stream cannot be swallowed silently. With seeking asked for, that is enough. Without it, AUTO uses it only on Linux and macOS, when the size is known, the source can seek and the data compresses 3:1 or less: more compressible data, and Windows, where the child starts more slowly, read faster with the standard library. rapidgzip runs in a child Python process, because it aborts the process on a stream that ends early; starting one costs about 45 ms, which is why the AUTO threshold is 16 MiB. Each open stream has a child of its own. On Linux, the members of one archive file on disk reuse children: a member whose stream closed without an error hands its child to the next one, which skips the start. A reused child counts what earlier members left in its memory against max_decoder_memory, so one that kept more than a quarter of the cap is not reused. Those children end when the archive closes. Where no child can be started (a frozen application, archivey imported from a zip, or a spawn or temporary file the operating system refuses), AUTO uses the standard library, logging one warning per process on the archivey.streams logger, and ON raises ResourceUnavailableError. OFF never starts a child. - #### **`use_indexed_bzip2`** (`AcceleratorMode`, default: `AcceleratorMode.AUTO` ) – Whether to use rapidgzip's bundled bzip2 backend for random access into bzip2. Under AUTO it is used only when the compressed input is known to be at least INDEXED_BZIP2_AUTO_MIN_COMPRESSED_SIZE bytes (or its size is unknown). Like the gzip decoder, it runs in a child Python process, one per open stream, reused the same way (see use_rapidgzip). No crash has been seen in it; it is isolated as a precaution, because it comes from the same library as the gzip decoder, which does crash on some input. Where no child can be started, AUTO uses the standard library, logging one warning per process on the archivey.streams logger, and ON raises ResourceUnavailableError. - #### **`process_isolation`** (`ProcessIsolation`, default: `ProcessIsolation.KNOWN_CRASHES` ) – Which native decoders run in a child process. See ProcessIsolation. Accepts the member or its name ("known_crashes", "native_decoders", "none"). - #### **`zip_unflagged_fallback_encoding`** (`str`, default: `'cp437'` ) – Encoding for a ZIP member name that is neither flagged nor valid UTF-8. A name stored without the UTF-8 flag is tried as UTF-8 first; this encoding is used when those bytes are not valid UTF-8. The default is cp437, as the ZIP specification (APPNOTE) says. Set a local code page (for example "cp1252" or "shift_jis") for archives known to come from one. An explicit encoding= on open_archive() replaces this fallback for that archive; a name whose bytes are valid UTF-8 is still decoded as UTF-8. - #### **`rar_allow_glob_member_concatenation`** (`bool`, default: `False` ) – Read a RAR member whose stored name contains * or ? even when that name also matches other members. unrar selects members by an include mask, so such a name can match other members as well. Names like this are almost always constructed, so the read is refused by default. On a nonsolid archive the extra decode is also unbounded and not reported (ExtractionLimits does not cover open() or read()); on a solid archive those bytes are already inside AccessCost.SOLID. Set True to read it anyway. A glob name that matches no other member is unaffected either way. Under rar_decompressor=AUTO, an installed unar reads such a member by index instead of the refusal (see RarDecompressor), so this flag matters only when unrar must read it. - #### **`rar_decompressor`** (`RarDecompressor`, default: `RarDecompressor.AUTO` ) – Which external program decompresses RAR member data. See RarDecompressor. Accepts the member or its name ("unrar", "unar", "auto", "none"). The listing does not depend on it, except that the selected program decodes compressed RAR 1.5-4 comments. - #### **`read_link_targets`** (`bool`, default: `True` ) – Whether the reader reads a symlink's target when the format stores it as member data (ZIP, 7z, RAR3/4) rather than in the header. True: listing and a finished stream_members() pass read every such target, selected or not, so the report matches random access. On ZIP and 7z that can decompress data nobody selected and consult the password provider. False: the reader reads none of them on its own; extract_all reads the targets of links its selector and filter accept, and open() reads the target of a link it follows. Fixed for the reader's lifetime. - #### **`extraction_limits`** (`ExtractionLimits`, default: `ExtractionLimits(max_extracted_bytes=2147483648, max_ratio=1000.0, ratio_activation_threshold=5242880, max_entries=262144, live_ratio_activation_threshold=268435456)` ) – Decompression-bomb guards for extraction. See ExtractionLimits. - #### **`listing_limits`** (`ListingLimits`, default: `ListingLimits(max_members=262144, max_metadata_bytes=67108864)` ) – Caps on the size of a member listing. See ListingLimits. - #### **`decoder_limits`** (`DecoderLimits`, default: `DecoderLimits(max_decoder_memory=2147483648, max_key_derivation_rounds=33554432, max_ppmd_held_input=16777216)` ) – Caps on what the archive may make a decoder allocate or compute. See DecoderLimits. - #### **`spool_limits`** (`SpoolLimits`, default: `SpoolLimits(max_bytes=1073741824)` ) – Caps on what a reader writes to temporary storage. See SpoolLimits. - #### **`detection_budget`** (`DetectionBudget`, default: `DetectionBudget(max_prefix_bytes=4096, max_far_bytes=32774, max_scan_bytes=2097152, max_decode_input=1048576, max_decode_output=1048576, completion_window_bytes=65536)` ) – Upper bounds on what format detection may read and decode. Used by open_archive() and detect_format() alike. The default, BALANCED, covers every format archivey reads at the offset it specifies, a self-extracting archive within 2 MiB of the start, and the content probes. FAST_BUDGET and THOROUGH_BUDGET in archivey.detection_cost are the other presets; to change a single limit, dataclasses.replace one. - #### **`always_probe_content`** (`bool`, default: `False` ) – Whether detection tries every content probe, whatever the source is named. LZMA Alone, zlib and Brotli have no magic that detection can trust, so only a content probe (a trial decode of the first bytes) recognises them. Arbitrary binary files sometimes pass a probe, so by default open_archive() and detect_format() run a probe only when the source's name ends in one of that format's extensions: .lzma or .tlz for LZMA Alone, .zz or .zlib for zlib, .br or .brotli for Brotli, and the .tar. form of each except .tlz. An extensionless raw stream of these formats is then refused with FormatDetectionError. If you read such sources, set this to True, pass format=, or use open_stream(), which always tries every probe because its caller already says the source is a compressed stream. - #### **`diagnostic_policy`** (`DiagnosticPolicy`, default: `DiagnosticPolicy()` ) – Whether each diagnostic code is ignored, collected or raised. See DiagnosticPolicy. - #### **`max_retained_diagnostic_references`** (`int`, default: `256` ) – How many references to diagnostics the library keeps per collector. Each diagnostic retained in the summary takes a slot, and each one attached to a member takes another. Counts stay exact past the cap. A random-access member walk also keeps, until it ends, a record that a walk started over after a failure replays: the codes of the diagnostics emitted for the members it has built, and the full diagnostics only for the member being typed. That record takes no slot. - #### **`on_diagnostic`** (`OnDiagnostic | None`, default: `None` ) – A callback called with each diagnostic the policy collects or raises, as it is emitted, or None. - #### **`formats`** (`FormatPolicy`, default: `FormatPolicy()` ) – Which formats this call may open. See FormatPolicy. The default allows every format archivey reads and every registered plugin format. #### `__post_init__()` Check the fields at construction, and convert the two that hold enums. A config field is read wherever it is needed, which is never where it was written: `ArchiveyConfig(extraction_limits="none")` builds fine and then fails part-way through an extraction as `AttributeError: 'str' object has no attribute 'max_extracted_bytes'` — a private attribute name, and no mention of the argument the caller actually got wrong. Checking `config=` at the entry points does not reach this: the object passed there *is* an `ArchiveyConfig`, and the wrong type is one field in. The two accelerator fields are **converted** rather than only checked, because their consumers test them with `is` (AcceleratorMode.enabled_for()): a string that survived construction would not be refused on use, it would read as "neither ON nor OFF" and silently take the AUTO path. Converting here means the field always holds a member, and a bad spelling names itself at the call site that wrote it rather than during some later stream open. They stay annotated `AcceleratorMode` rather than `AcceleratorMode | str` because that is what they hold once constructed, and it keeps every consumer honest. A string is still accepted at construction — a type checker flags it, which is the right answer for a typed caller who has the enum imported anyway, and an untyped script gets the conversion. That conversion goes through `object.__setattr__` because the dataclass is frozen and it *rewrites* the field rather than only inspecting it. The checks above reject without writing, so they need no such thing. ### `archivey.DEFAULT_ARCHIVEY_CONFIG = ArchiveyConfig()` The configuration used when a call is given no `config=`: `ArchiveyConfig()`. ### `archivey.FormatPolicy` Which formats one call may open. Set it on ArchiveyConfig.formats:: ```text # ZIP, and TAR with or without gzip or xz. Nothing else opens. UPLOADS = ArchiveyConfig( formats=FormatPolicy(only={C.ZIP, C.TAR, S.GZIP, S.XZ}) ) # Everything except RAR, and no plugin formats. NO_RAR = ArchiveyConfig(formats=FormatPolicy(exclude={C.RAR}, plugins=False)) ``` An ArchiveFormat is a container and a compression, and both halves must pass. `only={C.TAR}` refuses `.tar.gz` until `S.GZIP` is listed too; `StreamFormat.UNCOMPRESSED` always passes. The default policy allows every format. open_archive() and open_stream() refuse a format the policy does not allow with UnsupportedFeatureError, whose message names every entry to change; `format=` does not bypass the policy. detect_format() reports the true format of a built-in file whatever the policy says, so a caller can check allows() before opening. A plugin format the policy turns off is not detected at all: its magic, extensions and content probe are left out of that call, and the detection receipt records each tier that left one out as `NOT_ENABLED_BY_POLICY`. The policy decides which reader runs, not what detection reads: magic, the self-extracting archive scan and the built-in content probes still run on every input, within `ArchiveyConfig.detection_budget`. `replace=` reads a built-in container with a plugin instead of the built-in reader (see replace):: ```text MY_ZIP = ArchiveyConfig(formats=FormatPolicy(replace={C.ZIP: MyZipPlugin})) ``` Fields: - #### **`only`** (`frozenset[ContainerFormat | PluginContainer | StreamFormat] | None`) – The containers and compressions that may open, or None for all of them. - #### **`exclude`** (`frozenset[ContainerFormat | PluginContainer | StreamFormat]`) – The containers and compressions that may not open. - #### **`plugins`** (`bool`) – Whether registered plugin formats may open. Use False with exclude; under only a plugin opens only when its container is listed. - #### **`replace`** (`Mapping[ContainerFormat, type[FormatHandler]]`) – Built-in containers read by a plugin instead of the built-in reader. A key is ContainerFormat.ZIP, SEVEN_Z, RAR or ISO. Its plugin is a archivey.formats.FormatHandler that is not registered: its format is the built-in one (ArchiveFormat.ZIP), and it declares no extensions, magic or probe, because the file is found by the built-in detection. The reader's format is the built-in format and format_info.from_plugin is false. The plugin gets what any indexed plugin gets, a seekable source from the start of the archive, so a multi-volume set and an archive behind a self-extracting stub are refused with UnsupportedFeatureError, a source that cannot seek with StreamNotSeekableError, and a raw CD sector image detected as ISO as the built-in reader refuses it. The format's own ArchiveyConfig options reach the plugin through ReaderContext.config. - #### **`_turns_off_plugins`** (`bool`) – (computed property) Whether some plugin format could be off: the cheap test detection makes before it filters each plugin signal through allows(). #### `allows(fmt)` Whether `fmt` may open under this policy. ### `archivey.ExtractionLimits` Decompression-bomb limits for extraction. Applied by extract_all(). `None` on a guard field disables that guard. UNLIMITED sets the three guard fields to `None`; ratio_activation_threshold and live_ratio_activation_threshold are parameters of the ratio guard rather than guards of their own, and are moot once `max_ratio` is `None`. Fields: - #### **`max_extracted_bytes`** (`int | None`, default: `2147483648` ) – Most bytes one extraction may write in total, across every member. 2 GiB. Bytes copied rather than decoded count too: a hard link written as a copy because it crosses a device boundary or goes past the filesystem's link-count limit. Only the link-count copies also count toward the archive-wide max_ratio, so this is the one guard that bounds a cross-device copy. Crossing it stops the whole extraction, even under on_error="continue". - #### **`max_ratio`** (`float | None`, default: `1000.0` ) – Largest decompressed-to-compressed ratio allowed. 1000.0. Checked per member (a member over it fails on its own, and on_error="continue" moves on) and across the archive (which stops the extraction). The per-member check needs the member's compressed size; where the format or access mode does not give one, only the archive-wide check applies. Neither check starts before ratio_activation_threshold bytes of output, and the archive-wide check on a source of unknown size, such as a pipe, not before live_ratio_activation_threshold. The archive-wide check also counts the copies written when a file runs out of hard-link slots (1024 names on NTFS), since the archive's declared link count drives them. A member's output never passes the size the archive records for it. Where both sizes are recorded, a member over the per-member ratio is refused before any of it is written. - #### **`ratio_activation_threshold`** (`int`, default: `5242880` ) – Output bytes before max_ratio is checked. 5 MiB. Small members compress extremely well without being bombs, so the ratio is only judged once a member (or the archive, for the archive-wide check) has produced this much. The archive-wide check on a source of unknown size, such as a pipe, waits for live_ratio_activation_threshold as well. It cannot be None; set max_ratio to None to turn the ratio guard off. - #### **`max_entries`** (`int | None`, default: `262144` ) – Most entries one extraction may create: files, directories and links. Crossing it stops the whole extraction, even under on_error="continue". It also bounds, separately, how many times one extraction rechecks a symlink because a later member changed a path the link resolves through. Only a hostile archive comes near it, and crossing it stops the extraction the same way. None removes both bounds, so a hostile archive can then make the rechecks take time quadratic in its size. - #### **`live_ratio_activation_threshold`** (`int`, default: `268435456` ) – Output bytes before the archive-wide ratio is checked on a source of unknown size. 256 MiB. When the archive's size is not known up front, as for a .tar.gz read from a pipe, the archive-wide check divides the output by the compressed bytes read so far. An honest archive whose data starts with a long blank run, such as a disk image, looks like a bomb until the data after that run arrives: blank data compresses past 1000:1. The same archive read from a file passes, because there the check divides by the whole file. So from such a source the archive-wide check waits for this much output, and a blank start shorter than this passes as it does from a file. Past it, the check still divides by the bytes read so far, so a longer blank start can still be refused; once a pipe has been read to its end, the two checks agree. A bomb read this way can write this much before the check stops it; max_extracted_bytes still bounds the total. Lower it to stop such a bomb sooner; raise it for piped archives with longer blank starts. A value below ratio_activation_threshold has no effect. It cannot be None; set max_ratio to None to turn the ratio guard off. ### `archivey.ListingLimits` Caps on the member listing a reader builds and keeps. Applied from the reader's open ArchiveyConfig.listing_limits for its lifetime. `None` on a field disables that guard. UNLIMITED disables both. Every call that lists members enforces them, in both access modes: `members()`, `members_report()`, `stream_members()`, `for member in reader` and `extract_all`. Each raises ResourceLimitError at the member that crosses a cap. ZIP, 7z, RAR and ISO apply `max_members` at parse (ZIP at open, or as it reads the directory when the end record understates the count), 7z and RAR weigh the names and comments their headers hold against `max_metadata_bytes` (RAR's compressed RAR 1.5-4 comments by declared or packed size, whichever is larger, before decoding them), and ISO the directory records, path tables and UDF descriptors `pycdlib` parses, so `open_archive` raises. A ZIP read from a stream that cannot seek has no directory to read at open: its pass charges `max_members` on each local header as it yields the member, and again on the directory's entries after the last member, and weighs the directory's text against `max_metadata_bytes` as it arrives. A ZIP file whose end record sits behind more than 64 KiB of trailing bytes is walked the same way at open, which charges `max_members` on each local header. ISO also counts each path-table entry against `max_members` (a table of more than `max_members + 1` entries is refused): every entry is a directory, which is a member anyway, so real images never notice. ISO counts the UDF tree `pycdlib` parses against `max_members` too, one per UDF name, although `members()` does not list UDF: an image whose UDF tree has more names than its ISO 9660 tree can be refused for the UDF tree. ISO counts members per tree but weighs bytes as one sum for the whole image, every tree together. TAR refuses, in every mode, an extended header (PAX or GNU long name) that declares more than is left of `max_metadata_bytes`, before reading it. Fields: - #### **`max_members`** (`int | None`, default: `262144` ) – Most members a listing may hold. Each listed member costs about 1.5 KiB of memory (measured by scripts/measure_limit_costs.py), so the default allows about 384 MiB at open on 7z, RAR and ISO, where a tiny compressed header can declare that many members (ZIP checks the count too, but its directory costs about 46 bytes per entry). A current Linux kernel source tree has about 90 000 files, a third of the default; raise this for archives with more members, and lower it when opening untrusted archives on a server with less memory to spare. - #### **`max_metadata_bytes`** (`int | None`, default: `67108864` ) – Most bytes of text a listing may retain across its members. 64 MiB. Counts member names (and raw names), comments, link targets, owner and group names and the string or bytes values in extra, plus the keys of a dict nested in it (such as TAR's PAX keywords), plus the archive comment. The top-level extra keys are fixed per format and do not count. Non-ASCII text counts four bytes per character, so it is an upper bound rather than an exact size. A TAR sparse member's map, which the listing keeps to read the member, counts 24 bytes per entry, and each ZIP extra field kept in extra["zip.extra_fields"] counts 72 bytes plus its data. A ZIP read from a stream that cannot seek also keeps the data of each member short enough to be a symlink until its directory says which members are links. That data does not count here; the pass holds at most this many bytes of it, dropping the longest first. A link whose data was dropped is listed without its target and reports SYMLINK_TARGET_UNAVAILABLE. ### `archivey.DecoderLimits` Caps on what a decoder may allocate or compute because the *archive* said to. Several codecs size their working memory from a number in the archive's own header rather than from anything the caller chose: 7z PPMd var.H carries a 32-bit window size, ZIP method 98 an 8-bit megabyte count, LZMA a 32-bit dictionary size. Those numbers are attacker-chosen, they are read before a single byte of member data is, and the allocation that follows is not proportional to the archive's size — a 153-byte 7z can ask for 4 GiB. **What is capped today:** both PPMd paths, and the LZMA dictionary size wherever an archive declares one — 7z LZMA and LZMA2, ZIP method 14, each xz block, `.lzma` and each lzip member — and the window each zstd frame declares, on `.zst`, ZIP method 93 and 7z. RAR's dictionary is checked too, before `unrar` or `unar` is started, at what that program allocates for it. `unar` writes to the whole declared dictionary, so the declared size counts. `unrar` writes to it only as output is produced, so the declared size counts only up to the unpacked bytes the read decodes: the member's own size, or in a solid archive the members up to and including it. The two decoders that allocate what the header says, pyppmd and liblzma, differ. A refused allocation inside pyppmd takes its process down (a child process unless `process_isolation` is `NONE`). liblzma does raise `MemoryError` when it cannot reserve the dictionary, but a reservation that succeeds is its real cost: the dictionary fills as output is written, so a 151 KB stream declaring 4 GiB held 1.1 GiB resident after producing 1 GiB, where the same stream declaring 1 MiB held 59 MB. The dictionary bounds how much of the output the decoder keeps, and the archive picks it. `max_decoder_memory` also caps the rapidgzip accelerator for gzip, zlib and raw DEFLATE, though no archive declares a size there. rapidgzip keeps whole decoded chunks in memory, so a 1 MB gzip file of zeros made it hold close to 1 GiB. Its child process is stopped when its memory grows past the cap, and the standard library reads the rest of the stream, so the result does not change; only the speed does. The child checks every millisecond and again before it answers each read, so the peak can pass the cap by what the decoder allocates between two checks: measured, about 17 MB on four cores. On a busy machine the timed check can wait for a processor, and then the check before the next read stops it. Where the platform does not report a process's peak memory, the child is not capped. The same shape holds for key derivation, which costs time rather than memory: RAR5 and 7z headers say how many hashing rounds turn a password into a key, and max_key_derivation_rounds caps their total over one open archive. This is not an ExtractionLimits field, and the difference is not cosmetic. The bomb guards there measure *output*: they count bytes as an extraction produces them and stop when the total or the ratio says the archive is lying about its size. A decoder's working memory is neither output nor proportional to it, it is claimed up front, and it is claimed on `open()` and `read()` as much as on `extract_all()` — paths `ExtractionLimits` does not cover at all. Applied from the reader's open ArchiveyConfig.decoder_limits for its lifetime, as ListingLimits is: the codec-layer view is built once when the reader is. `None` on a field disables that guard. UNLIMITED disables every one. Where an archive declares the size, exceeding a guard raises ResourceLimitError *before* the allocation, which is the only place it can be raised in this process: the process has no recourse once the request is in the allocator's hands. The rapidgzip child is the exception: nothing declares its size, so its memory is measured after the allocation, in its own process, and going over the cap raises nothing to the caller. The child is stopped and the standard library continues the stream. **Detection is not capped.** Formats without magic (`.lzma`, and the compressed tar inside `.tar.xz` and its siblings) are recognised by decoding a small sample, and that sample is decoded with no decoder limit, because a capped probe would report a different format for a caller who passed UNLIMITED. It does not need one: a probe builds its LZMA decoder with only the dictionary its sample needs (4 to 64 KiB, whatever the header declares), which decodes the sample to the same bytes, and lowers zstd's window limit to libzstd's default 128 MiB, treating a frame over it as unrecognised. The open that follows detection is capped as described here. Under a memory cap (a container limit, `RLIMIT_AS`, a small machine) a refused native allocation does not always surface as `MemoryError` — pyppmd 1.3.1 dies on `double free or corruption` and takes the interpreter with it, so no `try`/`except` around the decode can contain it. Fields: - #### **`max_decoder_memory`** (`int | None`, default: `2147483648` ) – Largest archive-declared working set a single decoder may allocate. 2 GiB. A 7z folder runs all its decoders at once (the stages of a coder chain, and a BCJ2 folder's main, call and jump branches), so when it has more than one, their LZMA dictionaries, PPMd memory sizes and zstd windows count together against it. A zstd window counts as the first frame declares it, when the zstd coder reads a packed stream directly. That number is a policy choice, not a limit of the format, so here is what it was chosen against. Measured on 7-Zip 23.01, a writer declares whatever -m0=PPMd:mem=… asked for, reduced for a small member to 16× its size rounded up to a power of two (floor 64 KiB). Its presets never ask for much: plain -m0=PPMd declares 16 MiB and -mx9 256 MiB, whatever the input. Reaching 2 GiB therefore takes an explicit mem=2g, which cannot go above it at any member size; only mem=3g and mem=4g do. The field itself is 32 bits, so a header may declare just under 4 GiB — and a 153-byte archive may declare it, which is the case the cap is really for, since asking costs an attacker nothing and it is read before any member data is. 2 GiB is the last round value below that 32-bit ceiling, matching ExtractionLimits.max_extracted_bytes. It admits every archive 7-Zip's own presets write, by a factor of eight, and admits a deliberate mem=2g as well; it refuses the top of the field. Reading archives written with mem=3g or above means raising it or passing UNLIMITED. The LZMA dictionary sits further under it. xz Utils refuses to write a dictionary over 1.5 GiB, and its -9 preset writes 64 MiB whatever the input. 7-Zip 23.01 shrinks the dictionary it declares to about the member's size (4 KiB for a 25-byte member, 7 MiB for 6.7 MB, 96 MiB for 100 MB, all asked for with -md=1536m), so declaring more than 2 GiB takes a member over 2 GiB and an explicit -md above 2g. lzip's field cannot say more than 512 MiB. The same number serves both codecs because it answers the same question, and xz and .lzma are checked through liblzma's own memory limit, which also counts about 64 KiB of the decoder's overhead; a 128 KiB allowance keeps a dictionary exactly at the cap readable there as on every other path. What the default is not is a promise about the machine. The cap bounds what an archive may ask for; whether an allocation succeeds is a property of the host, and the two are independent. A process with less headroom than the cap — a container under a memory limit, a small VM — gets nothing from the default: a declaration below 2 GiB passes the guard, and the allocation that follows is the one that fails, which is the unsurvivable case described above. Such a process should set the cap under its own headroom, anchored on the limit it runs with rather than on anything 7-Zip writes. Code that opens files it did not choose — an upload endpoint, a mail scanner — wants the same move for a different reason: 256 MiB still takes everything the PPMd and LZMA presets produce. - #### **`max_key_derivation_rounds`** (`int | None`, default: `33554432` ) – Total rounds of password-to-key derivation one open archive may run. 2\*\*25. RAR5 and 7z let the archive choose how expensive a key is to derive: RAR5's kdf_count asks for 2\*\*kdf_count PBKDF2-HMAC-SHA256 rounds, 7z's NumCyclesPower for 2\*\*cycles SHA-256 rounds, each up to 2\*\*24 (a few seconds). Each derivation is capped already; this caps the sum. RAR5 salts each encryption record and 7z each folder, so an archive can make every member cost a fresh derivation, and a candidate password list multiplies that again. The work runs inside hashlib with the GIL released and cannot be interrupted, so what the caller sees is a process that stops responding. Rounds are counted as the archive declares them, per derivation that actually runs. A key the reader already derived for the same password and salt comes from its cache and costs nothing, and real writers reuse one salt across an archive (rar 7.00 writes one per archiving run, 7-Zip 23.01 writes none), so an ordinary archive spends one or two derivations whatever its size. Every candidate password that is tried counts, right or wrong. The RAR3 scheme, whose cost is fixed at 2\*\*18 SHA-1 rounds, counts at that number; ZIP AES (a fixed 1000 rounds) is not counted. RAR5 charges one derivation per output it needs from a password: the PswCheck (2\*\*kdf_count + 32 rounds), the checksum HashKey (+ 16) and, when archivey decrypts a stored member or the headers itself, the AES key (+ 0). A RAR5 read through unrar costs two derivations per password, a stored member three. 2\*\*25 is about ten seconds of hashing: two 7z derivations at the 2\*\*24 maximum, but not one maximum-cost RAR5 member, which needs two or three; 64 passwords at 7-Zip's 2\*\*19; about 500 at rar's 2\*\*15. Exceeding it raises ResourceLimitError before the derivation that would cross it starts. Code that opens archives it did not choose may want 2\*\*24, which still admits every archive the usual tools write. - #### **`max_ppmd_held_input`** (`int | None`, default: `16777216` ) – Largest PPMd member, in compressed bytes, held and handed to the decoder whole. 16 MiB. The decoder is pyppmd, in a child process unless process_isolation is ProcessIsolation.NONE. Fed whole, it is asked little after a corrupt member ends early, the state in which it can crash. Fed in pieces, it cannot tell a corrupt stream that ended from a valid one waiting for input. Holding costs memory in this process: the held input, and about as much again while it is copied. A larger member, and a ZIP member read from a stream that cannot seek with its sizes stored after its data, is fed in pieces. None holds every member whose size is known, at a memory cost that grows with the member. ### `archivey.SpoolLimits` Caps on what a reader writes to temporary storage, mostly copies of its source. Some reads need the archive as a file on disk even when the caller passed a stream. Today that is RAR: `unrar` and `unar`, the programs that decode member data, take a filesystem path, so a RAR opened from a `BytesIO` or another file object is copied to a temporary file (a volume set, to a temporary directory) the first time a member has to go through one of them. The copy is of the whole archive, and it is removed when the reader closes. A source opened from a path is read in place and not copied, with two exceptions that are bounded here too. A RAR with a prefix before it (an SFX stub, say) read with `rar_decompressor='unar'` is copied from where the RAR starts. And files that must be linked into a temporary directory, which `unar` always reads from and `unrar` needs for an explicit list of volumes it would not find by name, are copied there when the system can make neither a symlink nor a hard link. Applied from the reader's open ArchiveyConfig.spool_limits for its lifetime. `None` on a field disables that guard. UNLIMITED disables it. Fields: - #### **`max_bytes`** (`int | None`, default: `1073741824` ) – Most bytes one reader may write to temporary storage. 1 GiB. That is the copy of its source, and the sources a solid RAR pass keeps for file copies. A volume set counts as one copy: the limit applies to the total across its volumes. When the size is known before the copy starts, an archive over the limit raises ResourceLimitError before anything is written. Otherwise the copy stops before it passes the limit. Either way the partial copy is removed, and the error names this field. The limit holds for the reader, not per attempt: once a copy is refused, later reads that need it are refused without copying again. 0 refuses every copy: a stream source then reads only the members archivey can read without unrar, such as stored, unencrypted RAR members. The copy goes to the platform temporary directory (tempfile.gettempdir()); where that is memory-backed, such as tmpfs, this limit is a memory limit. The same allowance holds the sources of RAR5 file copies (rar -oi) in a solid pass (stream_members, extraction): the pass keeps each source as it decodes it, up to 8 MiB in memory and the rest in a temporary file within this limit, so a copy does not decode the solid stream again. That file counts only from the source's first decoded byte, after any copy of the archive the pass needs, until the pass ends and deletes it. Unlike a copy's bytes, which keep counting after the copy is removed, the kept file's bytes then stop counting. A source the limit has no room for is not kept, and its copies decode it again from the archive; that is never refused. ### `archivey.AcceleratorMode` Bases: `Enum` Whether to use an optional accelerator that gives seekable streams over a codec. Members: - **`AUTO`** (`'auto'`) – Use it when seekable streams were asked for (seekable_members=True on open_archive, seekable=True on open_stream), or for gzip, zlib and raw deflate on Linux and macOS a large sequential read (below), its package is installed, and the compressed input is known to be large enough to pay for it: RAPIDGZIP_AUTO_MIN_COMPRESSED_SIZE for gzip, zlib and raw deflate, and INDEXED_BZIP2_AUTO_MIN_COMPRESSED_SIZE for bzip2. An input of unknown size counts as large enough. For gzip, zlib and raw deflate the decompressed size must also be known, so that a truncated stream cannot be read short without an error. Also for gzip, zlib and raw deflate, on Linux and macOS, a read with no seeking asked for uses it when the source can seek, the compressed input is known to be at least RAPIDGZIP_AUTO_MIN_COMPRESSED_SIZE, and the data compresses 3:1 or less, since rapidgzip decodes in parallel. Otherwise the standard library decodes the stream. - **`ON`** (`'on'`) – Always use it. Raises PackageNotInstalledError when its package is not installed. - **`OFF`** (`'off'`) – Never use it. The standard library decodes the stream. Set both use_rapidgzip and use_indexed_bzip2 to this when results must not depend on the machine. #### `enabled_for(*, seekable, available, input_size=None, min_size=None)` Resolve the tri-state to "use the accelerator?". `ON` always returns `True` (the caller checks availability and raises `PackageNotInstalledError` if the package is missing — the user asked for it explicitly; `min_size` is ignored). `AUTO` enables it only when `seekable` is true and the package is available, so a missing package falls back silently. `seekable` is the demand for the accelerator: declared seeking, or for the DEFLATE family also a large sequential read judged worth it (`_sequential_read_pays` in `codecs/rapidgzip_select.py`). When `min_size` is set and `input_size` is known and strictly below that threshold, `AUTO` also falls back (tiny members do not repay per-stream accelerator setup). Unknown `input_size` keeps the pre-threshold AUTO behaviour. ### `archivey.ProcessIsolation` Bases: `Enum` Which native decoders run in a child process, where a crash on hostile input costs the member being read instead of the caller's process. A decoder in a child process whose process crashes raises CorruptionError; one killed from outside raises ResourceUnavailableError (SIGKILL, most often the out-of-memory killer) or ReadError. Checksums and AES decryption always run in the caller's process, and no setting bounds how long a decoder takes. Members: - **`KNOWN_CRASHES`** (`'known_crashes'`) – The default: only the decoders seen to crash on some input run in a child process: rapidgzip (use_rapidgzip, use_indexed_bzip2) and pyppmd (every PPMd member). Every other decoder runs in the caller's process. Processes are reused for later decoders. Measured on one Linux machine (scripts/bench_process_isolation.py), each PPMd decoder costs about 1.4 to 2 ms more than in-process: a non-solid 7z of 1024 PPMd members of 4 KiB took about twice as long. A ZIP member or a non-solid 7z member has a decoder of its own. A solid PPMd folder is one decoder for all its members: it pays that once, and about 0.1 to 0.35 ms for each 4 KiB member read from it; the decoded bytes cross between processes on top of that, at the copying cost NATIVE_DECODERS gives for single streams. Where no child process can be started, a PPMd member raises ResourceUnavailableError and rapidgzip under AUTO falls back to the standard library. - **`NATIVE_DECODERS`** (`'native_decoders'`) – Every native decoder runs in a child process: zlib, bz2, lzma, zstd, lz4, brotli, inflate64, pyppmd and rapidgzip, including the trial decodes format detection runs. Each open decoder has a process of its own, and processes are reused for later decoders. Measured on one Linux machine (scripts/bench_process_isolation.py), a single stream (1 to 64 MiB for gzip and zstd, 1 to 16 MiB for bzip2 and xz) took about 1.05 to 1.5 times as long for bzip2, 1.1 to 1.6 times for xz, 1.4 to 2 times for gzip and deflate, and 2 to 6 times for zstd, mostly copying data between processes. Each decoder also costs about 0.6 to 1 ms (ZIP deflate and non-solid 7z LZMA2 members), and a PPMd decoder about 1.4 to 2 ms, as under KNOWN_CRASHES; an archive of many small members pays mostly that. A solid 7z folder is one decoder, plus about 0.1 to 0.3 ms for each 4 KiB member read from it, with the copying of larger members on top. An idle process holds about 9 MB. Where no child process can be started (a frozen application, archivey imported from a zip, a refused spawn), opening a decoder raises ResourceUnavailableError; nothing is decoded in the caller's process. - **`NONE`** (`'none'`) – No child process at all: every decoder runs in the caller's process. Not recommended for archives you do not trust: rapidgzip and pyppmd are isolated by default because they crash the process on some input, and here a crash takes the caller with it. It suits trusted files where speed matters, tests, and environments that cannot start processes. PPMd members decode in-process at any size, and rapidgzip is not used: use_rapidgzip and use_indexed_bzip2 under AUTO use the standard library, and ON is refused with ArchiveyUsageError when the config is built. ### `archivey.RarDecompressor` Bases: `Enum` Which external program decompresses RAR member data. The native parser lists a RAR archive without either program, except for a compressed RAR 1.5-4 comment, which the selected program decodes (without it the comment is `None`). Otherwise one of them is needed only to read a compressed member, or an encrypted one archivey does not decrypt itself. Stored members are read directly; a stored, encrypted member in one part is decrypted by archivey (AES, RAR 2.9 and later) when `cryptography` is installed, and by the program otherwise. With `UNRAR` or `UNAR`, archivey never changes from one program to the other. Members: - **`UNRAR`** (`'unrar'`) – RARLAB unrar 6.0 or later (or the RARLAB rar writer). It reads every RAR archive, including encrypted ones. - **`UNAR`** (`'unar'`) – unar 1.10 or later, from The Unarchiver's XADMaster library. It is free software, and brew install unar installs it on macOS. It is not a full substitute: archivey refuses, before unar runs, every read that unar is known to get wrong. A password is passed on unar's command line, where other local users can see it while it runs. Selecting UNAR when unar is not installed raises PackageNotInstalledError, even when unrar is available. - **`AUTO`** (`'auto'`) – The default: unrar when a usable one is on PATH, otherwise unar. The choice is made once per opened archive, when it is opened, and holds for every read of it: a read that unar refuses is not retried with unrar. One exception goes the other way: a member whose name unrar cannot be given, or whose glob name also matches earlier members (rar_allow_glob_member_concatenation), is read with unar when an accepted one is installed, because unar selects members by index. That applies only to a read that needs no password, from a file with nothing before the RAR (so unar gets no password and no copy), and to a member unar reads correctly; otherwise the unrar refusal stands. With neither installed, a read raises PackageNotInstalledError naming unrar. On a machine with unar and no RARLAB program, a password is passed on unar's command line; select UNRAR to rule that out. - **`NONE`** (`'none'`) – No program at all, for a caller who does not want unrar or unar run on its archives. Opening and listing work as with the others, and so does reading a stored (uncompressed) member that is not encrypted, also when it is split across volumes or sits in a solid archive, and one that is encrypted, in one part, with cryptography installed. A compressed member, an encrypted one split across volumes, or a split one with a volume missing, raises UnsupportedFeatureError before anything runs. A stored, encrypted member in one part raises PackageNotInstalledError naming cryptography when that package is missing. A compressed RAR 1.5-4 comment is None. The format_availability() query takes no config and still runs the unrar identification probe; pass it check_programs=False to avoid that. ### `archivey.PasswordInput = str | bytes | Sequence[str | bytes] | PasswordProvider | None` Accepted `password=` shapes: one value, an ordered candidate list, a provider, or None. ### `archivey.PasswordRequest` Context passed to a PasswordProvider when a password is needed. Fields: - #### **`member`** (`ArchiveMember | None`) – The member being decrypted, or None for archive-level (header) decryption. - #### **`attempt`** (`int`) – 1 on the first ask for this unit; increments on every later ask for it. Each ask follows a failure: either the previous answer failed to decrypt the unit, or it was a password that had already failed for this unit (a known-good password from an earlier unit, a listed candidate) and was skipped without a second try. A repeated answer is never decrypted twice. Asking stops when the provider returns None, or after three answers in a row that it had already given for this unit; one or two repeats in a row do not stop it, so a person at a prompt who types the same password again is asked again. A provider walking a stored list should skip duplicates in it: three equal entries in a row end the asking. ### `archivey.PasswordProvider = Callable[[PasswordRequest], str | bytes | None]` Callable consulted when static password candidates fail for an encrypted unit. ## Diagnostics See [Diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md#diagnostics). ### `archivey.Diagnostic` One immutable advisory occurrence: something archivey noticed and reported instead of raising. `message` and `context` carry the same facts for different readers. `message` is display text, and its control characters are escaped: it includes names from the archive, and a name carrying `\x1b[2K\r` could otherwise erase the terminal line reporting it and write something else in its place. `context` keeps the raw values, for matching on a name or for a structured log (see to_dict()). ArchiveyError escapes its `message` and keeps `member_name` raw for the same reason. Fields: - #### **`occurrence_id`** (`str`) – A random id, unique to this occurrence. - #### **`code`** (`DiagnosticCode`) – What kind of event this is. - #### **`message`** (`str`) – A sentence describing the event, with control characters escaped. - #### **`context`** (`DiagnosticContext`) – The event's details as typed fields with raw values, such as the archive and member names. Its kind field says which context class it is. ### `archivey.DiagnosticContext = NameNormalizationContext | NameEncodingContext | MemberNameControlsContext | UnusedArgumentContext | EmptyArchiveContext | UnconfirmedFormatContext | FormatConflictContext | ScanRaceContext | ArchiveEofContext | UnlistedDataContext | MemberTimestampContext | MemberHeaderRecordContext | SymlinkTargetContext | DigestContext | DirectoryDataContext | LinkDataContext | EncryptedVerificationContext | SeekIndexContext | StreamRewindContext | SelectorUnmatchedContext | SpecialFileDataContext | LocalHeaderContext | OffsetWrapContext | SizeWrapContext | UnknownMemberTypeContext` ### `archivey.DiagnosticCode` Bases: `StrEnum` Stable machine codes for advisory events. The codes in ARCHIVE_INTEGRITY_CODES report the archive's own bytes or metadata as anomalous, and `DiagnosticPolicy.STRICT` raises them. A diagnostic's `context` carries the details, such as the member and the field involved. Members: - **`MEMBER_NAME_NORMALIZED`** (`'member_name_normalized'`) – A member's stored name was changed to give name, for example its separators or a . segment. - **`MEMBER_NAME_ENCODING_INFERRED`** (`'member_name_encoding_inferred'`) – A member's name was decoded with an encoding archivey inferred, not one the archive or the caller declared. - **`MEMBER_NAME_BIDI_CONTROL`** (`'member_name_bidi_control'`) – A member's name or link target contains bidi controls, which change the order the text displays in and can make an .exe look like a .png. - **`MEMBER_NAME_ZERO_WIDTH_CHARACTER`** (`'member_name_zero_width_character'`) – A member's name or link target contains a zero-width space, word joiner or byte-order mark, so two different names can look the same. - **`MEMBER_NAME_INVISIBLE_CHARACTER`** (`'member_name_invisible_character'`) – A member's name or link target contains another invisible format character, such as a zero-width joiner or a soft hyphen. These occur in ordinary names, such as in emoji. - **`FORMAT_EXTENSION_CONFLICT`** (`'format_extension_conflict'`) – The file name's extension suggests one format and the content another. archivey goes by the content. - **`EXPLICIT_FORMAT_LISTED_EMPTY`** (`'explicit_format_listed_empty'`) – An archive opened with format= listed no members. The context's detected_format says what detection finds in the content. - **`EXTENSION_FORMAT_UNCONFIRMED`** (`'extension_format_unconfirmed'`) – A format chosen from the file name alone, with nothing in the content to confirm it, listed no members or failed to read. - **`PROBE_FORMAT_UNCONFIRMED`** (`'probe_format_unconfirmed'`) – A format found only by decoding a sample of the content failed to read, so the file may not be in that format. - **`EMPTY_ARCHIVE`** (`'empty_archive'`) – The listing finished without error and found no members. - **`ENCODING_ARGUMENT_UNUSED`** (`'encoding_argument_unused'`) – encoding= was passed for a format that decodes member names without it (7z, a directory, a single-file format), or for a format whose replacement reader (FormatPolicy(replace=)) does not apply one. - **`PASSWORD_ARGUMENT_UNUSED`** (`'password_argument_unused'`) – A password was passed for a format with no encryption (TAR, ISO, a directory, a single-file format), or for a format whose replacement reader (FormatPolicy(replace=)) takes no password. A password provider callable is not reported, since such a reader never calls it. - **`SCAN_DIRECTORY_VANISHED`** (`'scan_directory_vanished'`) – A directory opened as an archive changed while it was read: a subdirectory listed earlier was gone. - **`SCAN_ENTRY_VANISHED`** (`'scan_entry_vanished'`) – A directory opened as an archive changed while it was read: an entry listed earlier was gone. - **`ARCHIVE_EOF_MARKER_MISSING`** (`'archive_eof_marker_missing'`) – The record that marks the end of the archive is missing or short, such as a TAR's two zero blocks or a ZIP's end of central directory record. The archive may be truncated. - **`ARCHIVE_TRAILING_DATA`** (`'archive_trailing_data'`) – Bytes other than zeros follow the end of the archive or of the compressed stream. - **`ARCHIVE_UNLISTED_DATA`** (`'archive_unlisted_data'`) – A ZIP read from a pipe holds a member its central directory does not list. It is left out of the listing, as a read from a file never sees it. - **`MEMBER_TIMESTAMP_INVALID`** (`'member_timestamp_invalid'`) – A stored timestamp could not be used, so that time is left unset. - **`MEMBER_HEADER_RECORD_SKIPPED`** (`'member_header_record_skipped'`) – An optional record in a member's header could not be read and was dropped, so the field it would fill is left unset rather than wrong. RAR also reports two faults here that drop no record, because they too are header damage a strict policy should refuse: bytes in a RAR5 header that belong to no record, and a RAR5 quick-open record holding FILE header copies that match no header the walk reached. The context's record tells them apart. - **`SYMLINK_TARGET_UNAVAILABLE`** (`'symlink_target_unavailable'`) – A link's target could not be read: it is empty, too long, damaged, or needs a codec or package that is missing. The link stays listed with link_target unset, and the context's reason says why. - **`DIGEST_UNVERIFIABLE`** (`'digest_unverifiable'`) – A member's data cannot be checked against a stored checksum, so damage may go unnoticed. - **`MEMBER_DIRECTORY_DATA_IGNORED`** (`'member_directory_data_ignored'`) – A directory entry declares data, which extraction does not write. The directory is still created. - **`MEMBER_LINK_DATA_IGNORED`** (`'member_link_data_ignored'`) – A TAR symlink or hard link header declares data, which nothing reads. The link is listed and extracted as a link. - **`ENCRYPTED_MEMBER_UNVERIFIED`** (`'encrypted_member_unverified'`) – An encrypted member's stream was closed or seeked before its checksum confirmed the password, and nothing stronger had confirmed it, so the bytes read may come from a wrong password. - **`SEEK_INDEX_DEGRADED`** (`'seek_index_degraded'`) – Building a stream's seek index failed, was skipped or was thinned, so a seek may decompress more than usual. - **`STREAM_REWIND_REDECOMPRESSES`** (`'stream_rewind_redecompresses'`) – A backward seek decompressed at least 1 MiB again from an earlier point. - **`MEMBER_SELECTOR_UNMATCHED`** (`'member_selector_unmatched'`) – An entry of members= matched no member of the archive. - **`MEMBER_SPECIAL_FILE_HAS_DATA`** (`'member_special_file_has_data'`) – An entry stored as a device, FIFO or socket carries data, so it is listed as a FILE with that data, as unzip, 7-Zip and bsdtar extract it. - **`MEMBER_LOCAL_NAME_DIFFERS`** (`'member_local_name_differs'`) – A ZIP member's local header stores another name than its central directory entry. The member keeps the central directory's name, as unzip and 7-Zip read it; a tool that trusts the local header shows the other one. - **`MEMBER_LOCAL_METHOD_DIFFERS`** (`'member_local_method_differs'`) – A ZIP member's local header names another compression method than its central directory entry. The member is decoded by the local header's method, the one a read from a pipe uses, and the data is still checked against its CRC-32. - **`MEMBER_LOCAL_CRC_USED`** (`'member_local_crc_used'`) – A ZIP member's central directory entry stores CRC-32 0 and its local header another value, which the data is checked against. Adobe AIR writes such files. - **`ARCHIVE_OFFSET_WRAPPED`** (`'archive_offset_wrapped'`) – A ZIP stores an offset past 4 GiB in a 32-bit field without ZIP64, as macOS Archive Utility does, and archivey corrected it. - **`MEMBER_SIZE_WRAPPED`** (`'member_size_wrapped'`) – A ZIP member's stored sizes are its real sizes modulo 2³², as macOS Archive Utility writes a member over 4 GiB without ZIP64. archivey corrects them, or lists size as None until a read can tell. - **`MEMBER_TYPE_UNKNOWN`** (`'member_type_unknown'`) – A member's stored type is one archivey does not know: a TAR typeflag, or the file-type bits of another format's mode (a format plugin may report it). One that declares data is listed as a FILE, as GNU tar extracts it; one that declares none stays OTHER. ### `archivey.DiagnosticPolicy` Which diagnostic codes to ignore and which to raise; every other code is collected. Matching is by code only. Both arguments take any iterable of DiagnosticCode (or their values as strings), and the fields always hold a `frozenset`; a code in both is refused. A policy is built with set operations on the code sets, for example `DiagnosticPolicy(raise_on=ARCHIVE_INTEGRITY_CODES - {DiagnosticCode.X})`. Two ready-made instances: - `DiagnosticPolicy.STRICT` is `DiagnosticPolicy(raise_on=ARCHIVE_INTEGRITY_CODES)`: RAISE on the codes about the archive's own bytes, COLLECT on everything else. It is the recommended strict mode because it is version-stable in the way that matters: a patch release may add new codes but adds none to `ARCHIVE_INTEGRITY_CODES`, unless the new code reports damage archivey should already have caught. A minor release may add to the set, and each addition is a deliberate decision. - `DiagnosticPolicy.PEDANTIC` is `DiagnosticPolicy(raise_on=frozenset(DiagnosticCode))`: RAISE on every code, including the argument-hygiene and access-pattern ones. `frozenset(DiagnosticCode)` is every code of the installed version, so it raises on codes added after the caller wrote it. Fields: - #### **`ignore`** (`frozenset[DiagnosticCode]`) – Codes that are counted but not kept, logged or passed to on_diagnostic. - #### **`raise_on`** (`frozenset[DiagnosticCode]`) – Codes that raise DiagnosticRaisedError once recorded. ### `archivey.ARCHIVE_INTEGRITY_CODES = frozenset({DiagnosticCode.MEMBER_NAME_NORMALIZED, DiagnosticCode.MEMBER_NAME_ENCODING_INFERRED, DiagnosticCode.MEMBER_NAME_BIDI_CONTROL, DiagnosticCode.MEMBER_NAME_ZERO_WIDTH_CHARACTER, DiagnosticCode.EXTENSION_FORMAT_UNCONFIRMED, DiagnosticCode.SCAN_DIRECTORY_VANISHED, DiagnosticCode.SCAN_ENTRY_VANISHED, DiagnosticCode.ARCHIVE_EOF_MARKER_MISSING, DiagnosticCode.ARCHIVE_TRAILING_DATA, DiagnosticCode.ARCHIVE_UNLISTED_DATA, DiagnosticCode.MEMBER_TIMESTAMP_INVALID, DiagnosticCode.MEMBER_HEADER_RECORD_SKIPPED, DiagnosticCode.SYMLINK_TARGET_UNAVAILABLE, DiagnosticCode.DIGEST_UNVERIFIABLE, DiagnosticCode.MEMBER_DIRECTORY_DATA_IGNORED, DiagnosticCode.MEMBER_LINK_DATA_IGNORED, DiagnosticCode.SEEK_INDEX_DEGRADED, DiagnosticCode.MEMBER_LOCAL_NAME_DIFFERS, DiagnosticCode.MEMBER_LOCAL_METHOD_DIFFERS})` Codes reporting the archive's own bytes or metadata as anomalous. The `raise_on` set of `DiagnosticPolicy.STRICT`. Fifteen codes are deliberately **out**, and the reasons are part of the contract rather than an oversight: - `EMPTY_ARCHIVE` — an empty archive is legitimate, and `diagnostics` forbids treating zero members as an error. - `ENCODING_ARGUMENT_UNUSED` / `PASSWORD_ARGUMENT_UNUSED` — argument hygiene. A pipeline that speculatively passes a password to every call would otherwise raise on every unencrypted archive. - `EXPLICIT_FORMAT_LISTED_EMPTY` — `format=` is an override, and an override that halts the caller is not an override. - `STREAM_REWIND_REDECOMPRESSES` — reports the caller's access pattern rather than the archive, and is most useful as a deliberately targeted tripwire. - `ENCRYPTED_MEMBER_UNVERIFIED` — fires only when the caller abandons a member stream before EOF (extraction reads every member to EOF and never fires it). In `STRICT` it would turn a peek at a ZipCrypto or RAR3/4 encrypted member into `DiagnosticRaisedError`. Revisit when member streams report their verification state (`dev-docs/IDEAS.md`, "Verification state as data") and retire this code. - `PROBE_FORMAT_UNCONFIRMED` — a probe-only identification is an advisory about what the file *is* (its bytes passed that format's content check), not a finding about the archive's own bytes, and it only accompanies a read that already failed with a typed error that carries `format_unconfirmed=True`. A policy that resolves it to RAISE still gets that typed error, through `escalate_as`. Its sibling `EXTENSION_FORMAT_UNCONFIRMED` is **in** the set: it also fires on an extension-only empty listing, a successful open that no byte confirmed. - `MEMBER_SELECTOR_UNMATCHED` — reports the caller's `members=` argument, not the archive. A job that passes one fixed list of names to many archives would otherwise raise on every archive that lacks one of them. - `MEMBER_SPECIAL_FILE_HAS_DATA` — a documented writer option (Info-ZIP `zip -FI`) produces the shape, and the member's bytes are delivered and checked like any file's, so nothing is hidden; `STRICT` would refuse an ordinary archive (DR-25). - `MEMBER_NAME_INVISIBLE_CHARACTER` — zero-width joiners are part of ordinary emoji sequences, non-joiners of Persian and Indic spelling, and tag characters of subdivision flags. `STRICT` would refuse an archive of legitimately named files. Its sibling `MEMBER_NAME_ZERO_WIDTH_CHARACTER` is **in** the set: U+200B, U+2060 and U+FEFF have no use inside a filename. - `FORMAT_EXTENSION_CONFLICT` — reports the file's name, not its bytes: the bytes win and the archive reads in full. A tarball saved as `b.gz` is an ordinary mistake, and `STRICT` would refuse a well-formed archive over it. The code is still recorded. - `MEMBER_LOCAL_CRC_USED` — a common packager (Adobe AIR) writes the shape, and the data is still checked, against the local header's CRC-32, so nothing is left unverified. Its siblings `MEMBER_LOCAL_NAME_DIFFERS` and `MEMBER_LOCAL_METHOD_DIFFERS` are **in** the set: there a tool that trusts the other header shows the member under another name, or decodes other bytes. - `ARCHIVE_OFFSET_WRAPPED` — macOS Archive Utility (Finder's Compress) writes a ZIP past 4 GiB this way by default and reads it back, and a correction is kept only when a local header with the member's name is there, with the CRC-32 still checked. `STRICT` would refuse an ordinary Mac archive. - `MEMBER_SIZE_WRAPPED` — the same Mac archives: a member past 4 GiB, or a deflated one over about 4 MB compressed in an archive of any length, whose size only a read can tell. A size is corrected only where the layout or deflate's ratio forces it, and an unknown one is `None`, with the length and CRC-32 checked on a read. `STRICT` would refuse ordinary Mac archives. - `MEMBER_TYPE_UNKNOWN` — a TAR typeflag archivey does not know. GNU tar extracts such a member as a regular file with a warning, and archivey lists its data as a `FILE`, so nothing is hidden. `STRICT` would refuse an archive GNU tar extracts. ### `archivey.DiagnosticSummary` Immutable point-in-time snapshot of diagnostic counts and retained detail. Fields: - #### **`total_count`** (`int`) – Every diagnostic emitted in the summary's scope, ignored ones included. - #### **`counts`** (`Mapping[DiagnosticCode, int]`, default: `mappingproxy({})` ) – The exact count for each code, ignored ones included. - #### **`retained`** (`tuple[Diagnostic, ...]`, default: `()` ) – The diagnostics kept in full, in the order they were emitted. Ignored codes are not kept, and nor is anything past the ArchiveyConfig.max_retained_diagnostic_references budget. - #### **`dropped_count`** (`int`, default: `0` ) – How many of total_count are not in retained. ### `archivey.OnDiagnostic = Callable[[Diagnostic], None]` Optional synchronous callback invoked for COLLECT/RAISE diagnostics. ## Errors archivey's exceptions have two roots. `ArchiveyError` covers problems with the archive or its environment. `ArchiveyUsageError` covers mistakes in the calling code and is deliberately outside that tree, so `except ArchiveyError` does not hide them. The entries below follow the class tree: each group starts with its base class, except the group from `ResourceLimitError` to `DiagnosticRaisedError`. Those four are direct subclasses of `ArchiveyError` and unrelated to each other, and only `ResourceLimitError` has a subclass, `ResourceUnavailableError`, listed after it. [Errors and diagnostics](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md) explains which one to catch. ### `archivey.ArchiveyError` Bases: `Exception` Root of all Archivey exceptions. **The message is escaped.** Call sites build messages by interpolating archive-derived text — a member name, or a destination path built from one — and those are attacker-controlled. An exception message reaches a terminal by more routes than any one consumer controls: `print(e)`, `logging.exception`, a third-party error reporter, and above all an uncaught exception whose traceback the interpreter prints itself, whose final line is `str(e)`. Escaping in the handler that displays it protects only the routes someone remembered to configure; escaping here protects all of them, with no configuration. So `message` is stored escaped, and that escaped form is what str(), `args[0]` and `repr()` all render. `raw_message` keeps the text as the call site wrote it, for the one job the escaped form cannot do: being embedded in *another* message that will escape it in turn. `archive_name`, `member_name`, `link_target` and `source_format` stay **raw** too, for callers that need the real value to act on rather than to print. str() renders the names through `!r`, which escapes them for display in turn — so a name available as an attribute should **not** also be interpolated into the message, or it prints twice. Prefer prose plus attributes: `FilterRejectionError("Symlink target escapes destination", member_name=name, link_target=target)`. `format_unconfirmed` is a boolean (default `False`): `True` when the format claim rested only on a content probe with nothing corroborating it (no matching extension, no inner-TAR upgrade), or only on the file extension because every content signal declined, so a decode failure should not be read as "this known format truncated." Confidence is irrelevant to the flag. **Escape exactly once, at the outermost message.** Everything a message interpolates should therefore be raw when it goes in — which is what the two helpers are for, and why neither of them is a matter of taste: - a member name, link target or path → quoted(), not `!r`. `!r` escapes first, and this escapes the backslashes it introduced. - a caught exception that might be one of ours → exception_detail(), not `{exc}`. It gives the unescaped text of one of ours (as raw_message_of() does), and the type name of an exception that has no text. A handler that catches only a third-party decoder's type can use `{exc}`. - never `{exc!r}` for a caught exception: it shows Python's spelling (`error('unpack_from requires …')`), not what is wrong with the archive, and escapes one of ours a second time. When the exception's own text does not describe the archive (`struct.error`, `IndexError` from a parser), write that description instead and keep the original as `__cause__`. `tests/test_escaping.py` checks this rule. Note this is the opposite of the rule for `logger.*` calls, whose records the CLI does **not** escape: there, `%r` is what makes an interpolated name inert and must stay. ### `archivey.OpenError` Bases: `ArchiveyError` Reading could not start: the source is not a recognized archive, is not seekable where the format needs it, or a volume file cannot be opened. A recognized archive whose header is damaged, cut short or encrypted raises a ReadError subclass instead, from `open_archive()` as from any later call. ### `archivey.FormatDetectionError` Bases: `OpenError` Could not detect an archive format, or detected one this call cannot use. The second case is `open_stream()` on a container such as a ZIP: detection succeeded, but the source is not a single-file compressed stream. ### `archivey.StreamNotSeekableError` Bases: `OpenError` Source is non-seekable but this format/backend needs seek. ### `archivey.ReadError` Bases: `ArchiveyError` The archive's data is bad or cannot be read, at open or later. Raised directly when an external decoder or a child process fails without saying why; the subclasses name the common causes. ### `archivey.CorruptionError` Bases: `ReadError` The archive's bytes are damaged: a checksum mismatch, a bad header or data block. Data that ends early is damage too, so TruncatedError is a subclass and `except CorruptionError` catches it. ### `archivey.TruncatedError` Bases: `CorruptionError` Damage that looks like the data ending early. A best-effort label, not a diagnosis: damage that decodes short can raise it, and some cut-short headers raise a plain CorruptionError. Do not branch on it. ### `archivey.EncryptionError` Bases: `ReadError` Password required or wrong password. ### `archivey.LinkTargetNotFoundError` Bases: `ReadError` A symlink/hardlink target, or a file copy's source, is absent from the archive. reason says why, so a caller can act on the cause without reading the message. #### `reason = reason` Why the link has no member to follow to. ### `archivey.LinkTargetNotFoundReason` Bases: `StrEnum` Why a link has no member to follow to: LinkTargetNotFoundError.reason. Members: - **`NOT_RECORDED`** (`'not_recorded'`) – The archive records no target for this link. Not damage: a writer such as 7-Zip stores some links this way, and ArchiveMember.link_target_unrecorded is True for them. - **`UNREADABLE`** (`'unreadable'`) – The archive records a target, and this read could not produce it: the data holding it is damaged, encrypted, or out of reach. A diagnostic on the member says which. - **`UNRESOLVED`** (`'unresolved'`) – The target is known, but following it reaches no member that holds data: it names no member of the archive, or one the link may not use (a symlink target that leaves the archive root, a hardlink to a later member), or the chain of links loops. ### `archivey.ExtractionError` Bases: `ArchiveyError` Error extracting a member to disk. ### `archivey.FilterRejectionError` Bases: `ExtractionError` A safety check refused to write the member. One type for every check: a path that escapes the destination (`..` or an absolute path), a symlink that resolves outside it, a device node, FIFO or socket, a name the destination OS cannot store safely under the active policy (see `safe-extraction`), and a name built to display as something it is not (a Unicode bidi override or isolate). The member is `BLOCKED` whichever check fired, and the message says which. ### `archivey.NameCollisionError` Bases: `ExtractionError` A member resolved to a destination another member of this run already claimed. Raised only when the caller opted in with `AbortOn.NAME_COLLISION`; without that opt-in a collision is not an error at all — `OverwritePolicy` resolves it and both members' `ExtractionResult` records the outcome. It is raised for *every* non-`TRUSTED` collision regardless of how the policy would have resolved it, since the trigger is the collision itself, not its resolution. ### `archivey.NameRewrittenError` Bases: `ExtractionError` A member name was rewritten: to its portable spelling, or re-rooted inside the destination. Raised only when the caller opted in with `AbortOn.NAME_SANITIZED` — a narrow escape hatch for callers who require the on-disk name to match the archive's byte for byte. Without the opt-in the rewrite is a success, recorded as `ExtractionResult.presented_name`. ### `archivey.ResourceLimitError` Bases: `ArchiveyError` A configured resource limit was exceeded, or the system did not give a resource. Covers ListingLimits listing caps, parse-time `max_members` on ZIP, 7z, RAR and ISO, ExtractionLimits bomb guards, DecoderLimits caps on archive-declared decoder memory and key-derivation work, and SpoolLimits, the cap on copying a stream source to temporary storage; opening the archive from a path avoids that copy. A shortage of the system rather than a configured limit raises the subclass ResourceUnavailableError. Sibling of ExtractionError (not a subclass): these are not filter/path failures. ### `archivey.ResourceUnavailableError` Bases: `ResourceLimitError` The system did not give a resource the read needed; no configured limit was exceeded. Raised when a decoder or helper program is ended by SIGKILL (most often the out-of-memory killer), when a decoder that ProcessIsolation runs only in a child process cannot have one (a frozen application, a refused spawn), and when a PPMd decoder process cannot allocate the member's model (a container memory limit). The data may be valid: read it again with more memory, or where child processes can start. Raising a limit in the config does not help. A subclass of ResourceLimitError, so code that catches that type catches this one too. ### `archivey.UnsupportedFeatureError` Bases: `ArchiveyError` The archive is recognized, but uses something archivey cannot handle. A variant, codec or layout of a known format (a raw CD sector image, a UDIF disk image, a 7z coder graph that is not a tree of chains, a multi-volume set where the format has none), or a request this archive or backend cannot serve (a password `unrar` cannot be given). The problem is the archive, not the calling code: that raises ArchiveyUsageError. It is also raised for a format the caller's FormatPolicy (`ArchiveyConfig.formats`) does not allow. `source_format` is that format, and `config.formats.allows(exc.source_format)` tells this case apart. An unknown compression method, codec or version number read from a header with no checksum (a ZIP method, for example) may also mean that header is damaged. Archivey reports it as unsupported because it cannot tell the two apart. A refused `seek()` or `tell()` on a member stream is *not* this class: a member stream has to keep behaving like a file object, so it raises io.UnsupportedOperation (a subclass of OSError and ValueError). ### `archivey.PackageNotInstalledError` Bases: `ArchiveyError` A required optional package or external tool is absent. Raised both when a whole format needs it (opening an ISO without `pycdlib`) and when one member does (a PPMd member without `pyppmd`). The message names what to install. ### `archivey.DiagnosticRaisedError` Bases: `ArchiveyError` A diagnostic was escalated to an error via DiagnosticPolicy. Always-stop: extraction MUST NOT catch this as a per-member failure under `OnError.CONTINUE`. Carries the escalated Diagnostic. ### `archivey.ArchiveyUsageError` Bases: `Exception` Caller misuse of the Archivey API — deliberately not an ArchiveyError. `except ArchiveyError` wraps archive/environment problems; usage errors indicate a bug in calling code and must not be swallowed by those handlers. It covers what the argument types alone cannot rule out: calling a method the reader's mode forbids (`members()` on a `streaming=True` reader), using a closed reader, opening a second overlapping member stream without `concurrent_operations=True` (the message names the `open_archive()` call site), and driving the reader from inside a diagnostic callback. Those raise this class itself. A wrong argument raises it too, as a private subclass that is also a `TypeError` (an argument of a type the call cannot use: `config="strict"`, `open_archive(0)`) or a `ValueError` (a usable type with a value the call refuses: an unknown format spelling, a negative limit). So `except TypeError`, `except ValueError` and `except ArchiveyUsageError` all catch a bad argument. The subclasses are not public names; catch the builtin or this class. The message is escaped on the same terms as ArchiveyError's. A usage error's text is mostly archivey's own, so the escaping is usually a no-op — but "mostly" is not a property worth carving an exception into, and a usage error is free to name the member that provoked it. `refused_member_type` marks one case by type rather than by message. It is the type of the member `open()` / `read()` refused because it is not a file (`DIRECTORY`, `ANTI` or `OTHER`), and `None` on every other usage error. When the caller opened a link, it is the type of the member the link resolved to: a symlink to a directory gives `MemberType.DIRECTORY`. ## Access cost See [Solid archives](https://archivey-dev.github.io/archivey/reading/index.md#solid-archives) for what these mean for your reads. ### `archivey.CostReceipt` Machine-readable description of an opened archive's access costs. The three axes are **orthogonal** and must not be conflated: `listing_cost` is about enumeration, `access_cost` about the format layout, and `stream_capability` about the source bytes. See the `access-mode-and-cost` capability spec for the full model. Fields: - #### **`listing_cost`** (`ListingCost`) – Cost of enumerating all members. - #### **`access_cost`** (`AccessCost`) – Cost of reading one member's data, given the format layout. - #### **`stream_capability`** (`StreamCapability`) – Seekability of the underlying source bytes. - #### **`solid_block_count`** (`int | None`, default: `None` ) – Number of distinct solid blocks (each one decompress pass), or None when not applicable / unknown. is_solid lives on ArchiveInfo, not here, to avoid duplicating the flag. - #### **`notes`** (`tuple[str, ...]`, default: `tuple()` ) – Human-readable caveats about the cost figures. ### `archivey.ListingCost` Bases: `Enum` How expensive it is to *enumerate* all members (list names + metadata). The value names how the member list is read through the source the archive was opened on: from an index, by walking member headers, or by decompressing. It does not count the work of finding the index: a ZIP whose end record sits behind a long tail is walked to find its central directory and is still `INDEXED`. Nor does it say when the work is paid: a backend may do it inside `open_archive` and serve `members()` from memory (RAR). One format can need different work on different sources: a ZIP read from a pipe is `REQUIRES_SCANNING`. The value is set per backend and source, so a damaged archive listed by a fallback walk (a ZIP whose directory is gone) keeps its backend's value. Members: - **`INDEXED`** (`'indexed'`) – Members can be listed without scanning header-to-header or decompressing payload. Examples: ZIP central directory, 7z header, ISO directory tree at open, and a single-file compressed stream (gzip, xz, …), whose one member is known from the header without decompressing. A filesystem directory is not indexed (its walk is REQUIRES_SCANNING). - **`REQUIRES_SCANNING`** (`'requires_scanning'`) – No index, but members can be enumerated by seeking/scanning header-to-header without decompressing payload (e.g. an uncompressed tar, a RAR archive, or a filesystem directory walk). RAR walks its headers inside open_archive, so members() then returns from memory. - **`REQUIRES_DECOMPRESSION`** (`'requires_decompression'`) – The stream must be decompressed to reach the member headers (e.g. a compressed tar). ### `archivey.AccessCost` Bases: `Enum` How expensive it is to *read* one member's data, given the format layout. Members: - **`DIRECT`** (`'direct'`) – Any member can be read without touching other members. - **`SOLID`** (`'solid'`) – Reading member N may require decompressing earlier members in its solid block. ### `archivey.StreamCapability` Bases: `Enum` A property of the underlying *source* bytes, independent of the format layout. **Ordered by strength**, weakest first: `FORWARD_ONLY < SEEKABLE`. There is only one possible direction — a seekable source can serve every read a forward-only one can — so the comparison means "is at least as strong as", not a preference. That is what lets a requirement stated as a capability (`FormatAvailability.required_source`) be tested against a source's actual capability (`CostReceipt.stream_capability`) with `<=` instead of a lookup table. `ListingCost` and `AccessCost` are deliberately *not* ordered: their members name kinds of work, not strengths of one resource. Members: - **`SEEKABLE`** (`'seekable'`) – The source supports arbitrary seek(); positions can be revisited. - **`FORWARD_ONLY`** (`'forward_only'`) – Non-seekable source (pipe/socket): it cannot be rewound at all. Re-reading any earlier position requires a brand-new stream. ## Detection budgets These live in the `archivey.detection_cost` module and are not re-exported from `archivey`. A budget bounds what detect_format and `open_archive` may read and decode before they answer. You set it with `ArchiveyConfig(detection_budget=...)`, and `BALANCED_BUDGET` is the default. The receipt on FormatInfo.cost_receipt says what detection actually did. ### `archivey.detection_cost.DetectionBudget` Upper bounds on what detection may spend. Content-probe header reads at an offset are not a budget field: the Brotli chain walk caps them itself, at `CHAIN_MAX_LINKS` (8) header reads of 24 bytes. The Brotli chain decode that follows reads `[0, end)`: it is charged to `max_decode_input` and runs only when `end` is within the prefix/far/scan ceiling. Fields: - #### **`max_prefix_bytes`** (`int`) – Bytes detection may read from the start of the source. - #### **`max_far_bytes`** (`int`) – Bytes the far-magic tier may read, for a signature at a fixed offset past the prefix. Separate from max_prefix_bytes because ISO's CD001 at 32 769 needs a window of about 32 KiB that a 4 KiB prefix budget would forbid. - #### **`max_scan_bytes`** (`int`) – How far into the source the scan for an archive behind a prefix (an SFX stub) looks for a signature. A candidate found there is judged on its whole header, which may extend past the window by up to 132 KiB or max_scan_bytes, whichever is smaller; a header that does not fit is rejected, and sfx_scan recorded as cut short. - #### **`max_decode_input`** (`int`) – Compressed bytes decoding may consume, one allowance for the whole call, shared by the content probes, the Brotli chain decode, their completion check and the inner-TAR probe. - #### **`max_decode_output`** (`int`) – Decoded bytes the inner-TAR probe may produce. A content probe's output is bounded by the codec's own drain instead: 4 KiB, 64 KiB with the whole source in hand, and the sample length (up to 1 MiB) for the Brotli chain decode. That output is not charged, as it is mostly a copy of input already charged to max_decode_input. - #### **`completion_window_bytes`** (`int`) – The largest source a content-probe hit is re-checked against in full; 0 turns the check off. ### `archivey.detection_cost.DetectionBudgetPreset` Bases: `Enum` Named detection budgets. `BALANCED` is the `detect_format` default. Members: - **`BALANCED`** (`'balanced'`) - **`FAST`** (`'fast'`) - **`THOROUGH`** (`'thorough'`) ### `archivey.detection_cost.BALANCED_BUDGET = DetectionBudget(max_prefix_bytes=4096, max_far_bytes=_ISO_FAR_BYTES, max_scan_bytes=_SFX_SCAN_BYTES, max_decode_input=_INNER_TAR_DECODE, max_decode_output=_INNER_TAR_DECODE, completion_window_bytes=_COMPLETION_WINDOW)` ### `archivey.detection_cost.FAST_BUDGET = DetectionBudget(max_prefix_bytes=4096, max_far_bytes=_ISO_FAR_BYTES, max_scan_bytes=256 * 1024, max_decode_input=64 * 1024, max_decode_output=64 * 1024, completion_window_bytes=0)` ### `archivey.detection_cost.THOROUGH_BUDGET = DetectionBudget(max_prefix_bytes=4096, max_far_bytes=_ISO_FAR_BYTES, max_scan_bytes=_SFX_SCAN_BYTES, max_decode_input=_INNER_TAR_DECODE, max_decode_output=_INNER_TAR_DECODE, completion_window_bytes=_INNER_TAR_DECODE)` ### `archivey.detection_cost.DetectionCostReceipt` Measured detection work — charged as reads happen, not reconstructed afterwards. Fields: - #### **`prefix_bytes`** (`int`, default: `0` ) – Sum of range lengths requested via peek_range (overlapping peeks accumulate). Not comparable 1:1 with max_prefix_bytes: seek-based read_at does not charge here, and growing peeks bill each request in full. Prefer unique_bytes_read for "how much did we fetch from the source". - #### **`unique_bytes_read`** (`int`, default: `0` ) – Bytes fetched from the source. A forward read of the prefix counts each of those bytes once. The trailer read counts its block as well, including when a later tier then reads the same bytes as part of the prefix. - #### **`far_bytes`** (`int`, default: `0` ) – Bytes the far-magic tier peeked. Bounded by max_far_bytes. - #### **`scanned_bytes`** (`int`, default: `0` ) – Bytes the scan for an archive behind a prefix looked through. Bounded by max_scan_bytes. - #### **`decode_input`** (`int`, default: `0` ) – Compressed bytes decoded. Bounded by max_decode_input. - #### **`decode_output`** (`int`, default: `0` ) – Decoded bytes the inner-TAR probe produced. Bounded by max_decode_output. - #### **`passes`** (`int`, default: `1` ) – Detection passes this receipt sums, each run under the full budget. 2 when detect_format followed a stub-only executable to its sibling split volume: the stub's pass and the volume's pass. ### `archivey.detection_cost.TierSkip` A tier that detection did not run, with the reason. Fields: - #### **`tier`** (`str`) – The tier's name. - #### **`reason`** (`TierSkipReason`) – Why it did not run. ### `archivey.detection_cost.TierSkipReason` Bases: `Enum` Why a detection tier did not run. Distinct reasons matter: `NOT_ENABLED_BY_POLICY` does not make the search incomplete, while `CAPABILITY_UNAVAILABLE` and `BUDGET_EXHAUSTED` do. Members: - **`NOT_ENABLED_BY_POLICY`** (`'not_enabled_by_policy'`) - **`CAPABILITY_UNAVAILABLE`** (`'capability_unavailable'`) - **`BUDGET_EXHAUSTED`** (`'budget_exhausted'`) ## Display and naming helpers These live in the `archivey.terminal` and `archivey.paths` modules and are not re-exported from `archivey`. archivey's own command-line tool is built on them. The `archivey.terminal` helpers are for showing archive-derived text, such as member names, to a person without letting it control the terminal ([Names in other encodings](https://archivey-dev.github.io/archivey/opening/index.md#names-in-other-encodings)). ### `archivey.terminal.escape_control_chars(text)` Backslash-escape non-printable characters in attacker-controlled text. Rendering of each escape is delegated to repr(), whose escape set is exactly `not str.isprintable()` (verified across the whole code space) and whose spellings — `\n`, `\x1b`, `\u202e`, `\U0001d173` — are the ones a Python developer already reads. Only two characters are handled here: backslash, which `repr` would escape but which we must escape ourselves since we are not emitting quotes, and a surrogateescaped byte, which is the one place we deliberately differ. A surrogateescaped byte renders as the **underlying octet** (`\x80`) rather than `repr`'s `\udc80`: the name never contained U+DC80: that code point is an artifact of decoding a byte that was not valid in the declared encoding, and the operator cares about the byte. Printable characters pass through, so ordinary `café` and `日本語.txt` member names stay readable — which rules out ascii() and the `unicode_escape` codec, and `str.encode(errors="backslashreplace")` is unusable here for the opposite reason: error handlers fire only on *unencodable* characters, so `\x1b` — the byte this exists to defend against — would pass through raw. Backslash is escaped, without which the form would be ambiguous: a member literally named `a\nb` would render identically to one containing a newline. The cost is that a native Windows path passed through this doubles its separators; callers holding a path should render it `/`-separated first. **Not fully lossless.** A real C1 character and a surrogateescaped byte of the same value collide: both `U+009B` and `U+DC9B` render as `\x9b`, as do `U+00A0` and byte `0xA0`. Distinguishing them needs a separate notation for surrogateescaped bytes; the guarantee here is that the output is **inert** and that no character is silently dropped, not that the input is uniquely recoverable. ### `archivey.terminal.display_path(path)` Render a filesystem path for a message, `/`-separated. Escaping doubles backslashes, so a native Windows path interpolated raw comes out as `C:\\Users\\out\\a.txt` — lossless, but harder to read and to paste back. Rendering `/`-separated first leaves the escape nothing to double, and any backslash that does survive into the output is then a character in a *name*, which is exactly what should be escaped. This is the message-side counterpart of the rule the CLI's print sites already follow (`cli` spec: member-derived paths are rendered relative and `/`-separated before escaping). ### `archivey.terminal.quoted(text)` Delimit archive-derived text in a message **without** escaping it. `f"...{name!r}..."` in a message would escape the name, and then the message's own escaping (ArchiveyError, Diagnostic) would escape the backslashes `repr` introduced, rendering a hostile name doubly-escaped: `EV\\x1b[2KIL` where `EV\x1b[2KIL` was meant. Quoting here and letting the message escape once produces the delimiters without the doubling. Use this — not `!r` — for member names, link targets and paths interpolated into a message that escapes itself. `!r` remains correct for values that are not archive-derived (a format, a code, an exception type) and for `logger.*` call sites, whose records are **not** escaped by the CLI and where `%r` is what makes an interpolated name inert. The delimiter is **chosen**, the way `repr` chooses it, rather than escaped: a name may legitimately contain a quote, and `'it's'` leaves a reader unable to see where the name ends. Choosing costs nothing, whereas escaping the quote would write a backslash for the message escape to double — the failure this helper exists to avoid. A name containing *both* quote characters falls back to `'` and stays ambiguous; that tail is unreachable without escaping. `archivey.paths` holds the naming rules extraction applies to destination paths, for a front end that moves or renames what extraction wrote and wants the names a direct extraction would have chosen. ### `archivey.paths.numbered_name(name, n, *, is_dir)` Return `name` with the counter `n` added, as `OverwritePolicy.RENAME` spells it. When a member's destination is taken, `OverwritePolicy.RENAME` writes it under the first free name of this form, counting from `n=1`. A file keeps its extension: the counter goes before the final suffix. A directory has no suffix, so the counter goes after the whole name. `name` is a single path component, not a path: only its final suffix moves. > > > numbered_name("photo.jpg", 1, is_dir=False) 'photo (1).jpg' numbered_name("notes.tar.gz", 2, is_dir=False) 'notes.tar (2).gz' numbered_name("photos.2024", 1, is_dir=True) 'photos.2024 (1)' Parameters: - **`name`** (`str`) – The name that is taken. - **`n`** (`int`) – The counter, 1 or more. - **`is_dir`** (`bool`) – Whether name is a directory. Returns: - `str` – The candidate name. This function does not look at the filesystem: the caller - `str` – checks whether the candidate is free and tries n + 1 when it is not. It also holds two functions that turn patterns into a `members=` argument, for selecting by pattern where `members=` itself matches names exactly. They read the same pattern differently: `match_glob` keeps `*` inside one folder, as archivey's CLI does, and `match_tar_glob` follows `tar`, where `*` also matches `/` ([Extracting all or some members](https://archivey-dev.github.io/archivey/extracting/index.md#extracting-all-or-some-members)). ### `archivey.paths.match_glob(*patterns, with_contents=False)` Return a `members=` predicate that selects the members matching any of `patterns`. > > > with archivey.open_archive("site.zip") as archive: # doctest: +SKIP ... archive.extract_all("out/", members=match_glob("docs/\*\*", "\*.md")) The patterns are matched against the whole member name, `/`-separated, and case-sensitively: - `*` matches any run of characters inside one folder: `*.md` selects `README.md` but not `docs/guide.md`. - `?` matches one character other than `/`. - `[abc]`, `[a-z]` and `[!abc]` match one character from, or not from, the set, never `/`. `[*]` matches a literal `*`. - `**` as a whole path component matches any number of folders, including none: `**/*.md` selects `README.md` and `docs/api/guide.md`, and `docs/**` selects the `docs/` folder and everything under it. Elsewhere, `**` is `*`. `*` and `?` match a leading `.` too, so hidden files are not skipped. A directory member is matched by its name without the trailing `/`, so `docs/*` selects the folder `docs/api/` as well as the file `docs/index.md`. A pattern that ends in `/` selects only directories: `*/` selects every top-level folder. With `with_contents=True`, a member is also selected when a folder it is in matches, so a folder comes with everything under it: `match_glob("docs", with_contents=True)` selects `docs/` and `docs/api/guide.md`, and `match_glob("*/", with_contents=True)` selects everything that is in a folder. archivey's command-line tool matches its patterns this way. A `\` in a pattern is an ordinary character, on Windows too, because member names use `/` and a TAR member name can contain a `\`. A pattern that matches nothing is not reported: `MEMBER_SELECTOR_UNMATCHED` covers only the names in the collection form of `members=`. For `tar`'s rules, where `*` also matches `/` and `docs` selects the folder and everything in it, use match_tar_glob(): `match_tar_glob("*.md")` selects `.md` files at every depth, while `match_glob("*.md")` selects top-level ones only. Parameters: - **`*patterns`** (`str`, default: `()` ) – One or more glob patterns. A member is selected when any of them matches. - **`with_contents`** (`bool`, default: `False` ) – Also select every member inside a matching folder. Returns: - `Callable[[ArchiveMember | str], bool]` – A callable that takes an ArchiveMember and returns whether it - `Callable[[ArchiveMember | str], bool]` – is selected. It also takes a member name as a str, where a trailing / - `Callable[[ArchiveMember | str], bool]` – marks a directory. Raises: - `ArchiveyUsageError` – No pattern was given, a pattern has an empty component ("", "/", "a//b"; a leading / is allowed), or a [...] set is malformed, such as the reversed range [z-a]. It is also a ValueError. ### `archivey.paths.match_tar_glob(*patterns)` Return a `members=` predicate that matches `patterns` the way `tar` does. > > > with archivey.open_archive("site.tar") as archive: # doctest: +SKIP ... archive.extract_all("out/", members=match_tar_glob("docs", "\*.md")) Each pattern is matched against the whole member name with `fnmatch.fnmatchcase`, so it is case-sensitive and: - `*` matches any run of characters, `/` included: `*.md` selects `README.md` and `docs/api/guide.md`. `**` is the same as `*`. - `?` matches any one character, `[abc]` / `[!abc]` one character from, or not from, the set. - A pattern that names a folder, with or without its trailing `/`, selects the folder and everything under it: `docs` and `docs/` both select `docs/` and `docs/api/guide.md`, but not a file named `docs.txt`. For patterns where `*` stays inside one folder, use match_glob(). A `\` in a pattern is an ordinary character, on Windows too. A pattern that matches nothing is not reported. Parameters: - **`*patterns`** (`str`, default: `()` ) – One or more patterns. A member is selected when any of them matches. Returns: - `Callable[[ArchiveMember | str], bool]` – A callable that takes an ArchiveMember, or a member name as a - `Callable[[ArchiveMember | str], bool]` – str, and returns whether it is selected. Raises: - `ArchiveyUsageError` – No pattern was given, or a pattern has an empty component ("", "/", "a//b"; a leading / is allowed). It is also a ValueError. # Background # Philosophy Python can already open archives, but each format has its own library, its own API and its own gotchas. Archivey aims to put them all behind one interface that behaves the same way for every format, does what you'd expect, and helps you avoid footguns. Simple tasks stay simple, but the data and options are there if you need more control. ## What archivey is for Archivey reads archives. It opens ZIP, TAR (plain or compressed), 7z, RAR and ISO files, as well as single compressed files such as `.gz` or `.xz`, all through one interface. Code that lists or reads a ZIP works unchanged on a 7z. It's built for working with an archive's contents from your code: you can list the members, read one into memory, or go through all of them in one pass, without extracting anything to disk first. It can also extract to disk, with checks that stop a hostile archive from writing outside the folder you chose. It doesn't write archives yet. Each member carries every field its format records: times, permissions, owner, link targets, comments, stored checksums, and the format's own extras. When formats record the same thing, archivey puts it in the same field. When they record it differently, the difference stays visible. A TAR modification time is in UTC, so `modified` carries a time zone. A ZIP that has only the old DOS time stores local time with no zone, so its `modified` has none. Permissions are set only when the format stores them, and otherwise `mode` is `None`, rather than a guess. ## No surprises Archivey tries to make the easy way to use it also the right one, so that problems show up while you write your code, not later on a file you hadn't tried. Where an operation can quietly cost far more than it looks, such as seeking inside a compressed member, the default refuses it until you ask for it. Where refusing would get in the way of ordinary use, such as reading a solid archive out of order, it's allowed, and [Choosing how to read](https://archivey-dev.github.io/archivey/reading/index.md#why-these-arent-on-by-default) explains the cost. If the data is damaged or cut short, reading it through raises an error instead of handing back short data as if it were complete, and you still get what archivey could read before the damage. A strict default can be relaxed later without breaking anyone's code, but a permissive one can't be tightened. ## Safe with untrusted archives Archivey is meant to be safe on archives from anyone, such as files uploaded to a server. Extraction never writes outside the folder you chose, and listing, decompression and extraction have limits on memory and size, so a hostile archive raises an error instead of exhausting the machine. If you trust your archives, you can relax the limits and the checks on names and permissions, but not the check that keeps extraction inside the folder. [Extracting](https://archivey-dev.github.io/archivey/extracting/index.md#archives-you-trust) and [Security](https://archivey-dev.github.io/archivey/security/index.md) have the details. [VISION.md](https://github.com/archivey-dev/archivey/blob/main/VISION.md) explains these principles in more depth, and the [design rules](https://github.com/archivey-dev/archivey/blob/main/dev-docs/design-rules.md) show how they settle specific cases. # How it works You don't need this page to use archivey. It's for when you want to know what's underneath. Maybe you're deciding whether to trust archivey with archives from strangers and want to know which code reads them. Maybe an error names a package or program you didn't expect, or a file was recognized as a different format than you thought. The page follows what happens when you open an archive and read from it: how archivey recognizes the format, reads the archive's structure, and decodes the data. For why the defaults are the way they are, see [Philosophy](https://archivey-dev.github.io/archivey/philosophy/index.md). ## Recognizing the format Archivey decides the format from the file's contents first. Most formats start with a fixed signature. Some keep it further in, as ISO images do at 32 KiB, or behind a program stub, as self-extracting archives do. A single compressed file such as a `.gz` is also checked for a TAR header inside, so a tarball named `.gz` opens as a `.tar.gz`. The name is used only when the contents don't settle it. A few formats, such as raw zlib, LZMA alone and Brotli, have no signature, or one too weak to identify them. Archivey recognizes them by trying to decode the start of the data, which is called a probe. A probe can mistake other data for one of these formats, because the start of an unrelated file sometimes decodes without an error. So a probe runs only when the file's extension names that format, when you set `always_probe_content=True`, or when you call `open_stream()`. `detect_format()` runs the same steps without opening the archive. ## Reading the archive's structure Once the format is known, a parser reads the archive's structure: the list of members, their names, sizes and times, and where each one's data starts. Archivey has its own parsers for ZIP, TAR, 7z and RAR, and uses pycdlib for ISO images. All of them are written in Python, so parsing a malicious archive can result in wrong data but can't corrupt memory. Its own parsers mean every format follows the same rules for metadata, and any archive, solid ones included, can be read in a single forward pass with `stream_members()`. In archivey's tests, the ZIP, TAR, 7z and RAR parsers are checked against zipfile, tarfile, py7zr, rarfile and `unrar`, comparing both metadata and bytes. Archivey never imports zipfile, tarfile, py7zr or rarfile at run time. For each format's details, see the [format handbooks](https://github.com/archivey-dev/archivey/tree/main/dev-docs/formats) and the [library analysis](https://github.com/archivey-dev/archivey/blob/main/dev-docs/library-analysis.md), both written for people working on archivey itself. ## Decoding member data Many formats share the same compression methods. LZMA appears in 7z, `.xz` and `.tar.xz`, and DEFLATE in ZIP, 7z and gzip. Archivey implements each method once, as a stream that decodes another stream, and each format reader stacks the streams a member needs. A method therefore behaves the same, with the same limits and checks, whichever format it comes in. As the data passes, the stack also computes the checksum the format recorded, where it records one, and compares it when the member's last byte comes out. The decoders come from three places. Python's standard library covers zlib, bzip2 and LZMA, and Zstandard on Python 3.14 and later. The `[recommended]` extra includes third-party packages that add support for PPMd, Deflate64, Brotli, LZ4, Zstandard on older Pythons, and AES encryption. Archivey's own Python code handles `.Z` files, the 7z BCJ2, ARM64 and RISC-V filters, the legacy LZ4 frame, and the wrapping around each compressed stream in formats such as gzip and xz. Some decoders written in C can crash on bad data, so archivey runs those in a child process, and can run every decoder in one. [Security](https://archivey-dev.github.io/archivey/security/index.md#decoders-in-a-child-process) says which. RAR is the exception. No good library decodes it, so archivey decompresses RAR data by running an external program, `unrar`, `rar` or `unar`, and reading its output through a pipe, as the rarfile library does. Archivey still reads the RAR structure with its own parser, and reads members stored without compression or encryption itself. The [decision record](https://github.com/archivey-dev/archivey/blob/main/dev-docs/decisions/0002-native-rar-metadata-unrar-data.md) explains the choice. Compressed data can't be read starting from the middle, so a seek inside a member takes one of three routes. Without help, archivey decompresses the member again from its start, or from where it already is if the target is ahead. Some formats record where they can be restarted: xz keeps an index of its blocks, so a seek starts decoding at the nearest block before the target. For bzip2 data, and large DEFLATE data such as gzip files, the `seekable` extra adds an accelerator that marks restart points as it reads, and later seeks resume from the nearest one. [Seeking inside a member](https://archivey-dev.github.io/archivey/reading/index.md#seeking-inside-a-member) says what decoding again from the start and the accelerator mean for the checksum. # How it is built Almost all of archivey's code, tests and documentation are written by AI coding agents, mostly Claude Code and Cursor. The maintainer designs the library, makes the decisions, directs the work and reviews it. If you're deciding whether to trust a library built this way, this page explains who decides what, how a change gets made and tested, and what that does and doesn't promise. Archivey is a work in progress, and so is the process described here. ## Who decides what The maintainer designed the building blocks and the public API: one opener, one reader and one member model for every format; reading as a stream, with a member's bytes decoded as you read them; and the modes you choose at open, such as `streaming=True` for a pipe, `seekable_members=True` and `concurrent_operations=True`. [Philosophy](https://archivey-dev.github.io/archivey/philosophy/index.md) explains the thinking behind them. The maintainer also sets the scope, the architecture and the [design rules](https://github.com/archivey-dev/archivey/blob/main/dev-docs/design-rules.md) the agents follow, and rules on the trade-offs they can't settle from the code. Two examples: - **When formats disagree.** Archivey weighs following each format's official tool, such as `unrar` for RAR and GNU tar for TAR, against being the same across formats. Neither wins automatically. Device files that store no data are one case where being the same won: every format lists them as "other". - **What counts as damage.** Some problems refuse the archive, and others, which are only unusual, are reported as diagnostics. Decisions are recorded in the repository as [architecture decision records](https://github.com/archivey-dev/archivey/tree/main/dev-docs/decisions), [behaviour specs](https://github.com/archivey-dev/archivey/tree/main/openspec/specs) and [archived change proposals](https://github.com/archivey-dev/archivey/tree/main/openspec/changes/archive). The agents do the rest: they write the code and the tests, build the test archives, run investigations and review each other's work. They also draft the docs you're reading, which the maintainer reviews paragraph by paragraph. ## How a change gets made AI makes mistakes, so every change is reviewed before it's merged. 1. An agent writes the change, with tests. 1. A separate AI session reviews it, starting from the diff and the repository alone, not from the session that wrote the code. Each finding gets a fix or a stated reason, and the fixes are reviewed too. The [review workflow](https://github.com/archivey-dev/archivey/blob/main/dev-docs/review-loop.md) and the instructions the agents follow to [review](https://github.com/archivey-dev/archivey/blob/main/.claude/skills/code-review-skill/SKILL.md) and to [address findings](https://github.com/archivey-dev/archivey/blob/main/.claude/skills/address-review-findings/SKILL.md) are in the repository. 1. Anything the agents can't settle from the code or the design rules goes to the maintainer as a decision. 1. Sometimes the maintainer reviews the change directly. 1. The change is merged only when review and CI pass. The maintainer merges anything tricky, and an agent may merge a straightforward change once its review approves it and CI is green. Even then, problems and cases nobody thought of creep in. So agents also review the whole codebase rather than one change, each time with one goal, such as finding bugs, finding dead, redundant or unclear code, or evaluating the public API. One full review of the codebase ([PR 315](https://github.com/archivey-dev/archivey/pull/315)) produced nearly 700 comments, and a bug hunt ([PR 512](https://github.com/archivey-dev/archivey/pull/512)) found about 40 bugs, each reproduced with a test and later fixed. ## How it is tested Review catches mistakes in the reasoning. These checks catch the rest, and each one finds things the others miss. - **Thousands of tests on every pull request.** They run on Linux with Python 3.11 to 3.14, on macOS and Windows with 3.11 and 3.14, on a free-threaded build, with no optional packages installed, with the oldest dependency versions archivey allows, and on PyPy. - **A test for every fix.** Every bug that has been fixed has a test, and so do many corner cases that were only worried about, so a fixed problem can't quietly come back. - **One set of test archives, built in every format.** Each test archive is described once and then written in every format that can hold it, never by archivey itself: by other libraries (`zipfile`, `tarfile`, py7zr and pycdlib) and, for RAR and encrypted ZIP, by the `rar` and `7z` programs. Every copy must open, list, read back and extract safely. Hand-built hostile archives sit beside them: path traversal, links that point outside the destination, decompression bombs, and forged sizes and counts. - **Other libraries as a reference.** Tests check what archivey reads against zipfile, tarfile, py7zr, rarfile and `unrar`. Archivey never imports zipfile, tarfile, py7zr or rarfile when it runs; `unrar` is also the program it uses to decompress RAR data. - **Fuzzing.** Every test archive is cut short, bit-flipped, zeroed and padded, and reading it must either succeed or raise an `ArchiveyError`. A coverage-guided fuzzer (Atheris) runs over every format's parser and every decoder, briefly on each pull request and for longer every night. It fixes up checksums so its changes reach the code behind them, and it checks that the optional fast decoders give the same result as the standard ones. Property-based tests cover name handling, extraction safety and link resolution. - **Benchmarks.** Each pull request checks numbers that shouldn't change by accident, such as how many seeks a read takes and that a solid archive is decoded only once. A daily run measures time and fails if things get noticeably slower. - **A threat model.** Known attacks and gaps are written down with their status, including the limits archivey accepts rather than fixes. [Security](https://archivey-dev.github.io/archivey/security/index.md) covers what you need to know as a user, and the [full threat model](https://github.com/archivey-dev/archivey/blob/main/dev-docs/threat-model.md) has the rest. - **Bugs found in other libraries.** Comparing decoders turned up a known issue in pyppmd, three decoder bugs in pybcj and a hang in py7zr. Archivey avoids or works around each one. ## What this makes practical AI lets the maintainer build a better library, faster. It changes what is practical to do. - **Breadth.** Archivey reads ZIP, TAR, 7z, RAR and ISO archives and ten single-file compression formats. That includes encrypted archives, multi-volume sets, self-extracting executables and archives inside archives. Each of these has corner cases that libraries usually leave unsupported. Here most of them have a test. - **Hostile input.** Agents are good at the tedious question "what if this field is wrong?", asked of every field in every format. Many entries in the threat model are attacks the maintainer wouldn't have thought of alone. Two examples are a 7z archive that sizes a memory allocation from a count nobody checked, and a password check where the archive chooses how much work each attempt costs. - **Review as routine.** Reviewing every change from zero, and reading the whole codebase again and again, would take a team of people. With agents, both are part of the normal workflow. - **Measurement over memory.** Claims about speed, memory use and seeks are measured, and the benchmarks above catch a change that makes them worse. ## Versioning and stability Archivey follows [semantic versioning](https://semver.org/). Before 1.0, this is what each kind of release may change: - **A patch release (0.2.1)** fixes bugs and makes improvements that need no change in your code. It may read archives it used to refuse, report a new [diagnostic code](https://archivey-dev.github.io/archivey/errors-and-diagnostics/index.md), or add a value to an enum that says it is open-ended, such as `CompressionAlgorithm`. It doesn't add, remove or change a function, class, parameter or attribute. A new code that `DiagnosticPolicy.STRICT` raises waits for a minor release, unless it reports damage archivey should already have caught. - **A minor release (0.3.0)** may change it. A removed or renamed name keeps working for one more minor release first and emits a `DeprecationWarning` that names its replacement. A change that can't be phased in that way, such as a different result for the same call, has a line starting with **Breaking:** in that release's entry in the [changelog](https://github.com/archivey-dev/archivey/blob/main/CHANGELOG.md). Python shows a `DeprecationWarning` by default only when the deprecated call is in your top-level script, so the changelog's "Deprecated" section is the place to look before upgrading. To find deprecated calls anywhere in your code, run your tests with `pytest -W error::DeprecationWarning`. The public API is what `archivey` and its public submodules export, as listed on the [API page](https://archivey-dev.github.io/archivey/api/index.md). Anything under `archivey.internal` can change in any release. A few things you can see aren't part of the API either, and can change in any release: - The text of messages: an exception's, a diagnostic's `message` and a log record's. Match on archivey's exception types and diagnostic codes, not on message text. A diagnostic's `occurrence_id` means something only within the process that made it. - Which library or program decodes the data, such as rapidgzip, pyppmd or `unrar`, as long as the result stays the same. What follows from it can change with it: - the type and message of the library's exception that archivey chains to its own as `__cause__`, so catch archivey's exception types, not the cause; - `accelerator` in a stream rewind's diagnostic context, and `scan` and `error_type` in a seek index's. The diagnostic codes, and the other fields of those contexts, are part of the API. - Where a compressed stream can resume after a backward seek, which `ArchiveStream.nearest_resume_offset()` reports, and so which seeks report `STREAM_REWIND_REDECOMPRESSES`. It depends on the decoder. The same promise covers the `archivey` command's verbs, options and exit codes, as listed on the [command-line page](https://archivey-dev.github.io/archivey/cli/index.md): a removed or renamed one keeps working for one more minor release and prints a deprecation warning. The text it prints for people to read, such as messages and summaries, may change in any minor release, so a script should rely on the exit code and the options rather than on that text. A patch release may add a line of that text, but it doesn't change or remove one. A minor release may also add a verb, and from then on that word runs the verb rather than naming a file. A script that passes a path should write `archivey list ` or `./`, which always reach the file. 1.0 comes once early users have run archivey for a while and the API has held up. After that, breaking changes wait for 2.0. The larger open questions, such as a plugin interface for new formats, writing archives and an async API, would each add to the API rather than change the reading API that exists today. ## What this does not promise - The maintainer doesn't read every line. Agents do the line-by-line reading, and the maintainer reads designs, decisions, the tricky parts and review findings. - AI reviews find a lot, but their blind spots overlap: two sessions of similar models can miss the same thing. - The fuzzing doesn't run at OSS-Fuzz scale. That waits until archivey has more users. - Nobody can promise that archivey has no security bugs. If you process untrusted archives, take the precautions you would with any other library, such as running it with limited privileges or [in a process you can stop](https://archivey-dev.github.io/archivey/security/index.md#where-the-guarantees-stop). ## Checking it yourself You don't have to take this page's word for it. The history is public: every [pull request](https://github.com/archivey-dev/archivey/pulls?q=is%3Apr), its review threads and the decisions behind it are on GitHub, next to the decision records, specs and threat model linked above. Reviews are welcome, whether you read the code yourself or point an agent at it. So are bug reports, questions and pull requests; see [CONTRIBUTING.md](https://github.com/archivey-dev/archivey/blob/main/CONTRIBUTING.md). If you find a security problem, please report it privately as [Security](https://archivey-dev.github.io/archivey/security/index.md#reporting-a-vulnerability) describes, not in a public issue. # Acknowledgements Archivey builds on the work of the people who designed these formats and wrote the libraries and tools around them. This page credits the code archivey adapted, the projects it learned from and checks itself against, and the packages it uses. If we missed a project that should be here, please open an issue or a pull request. ## Code adapted from other projects Two parts of archivey are adapted from other projects' code, and their licence notices are kept in the files that hold it: | Project | What archivey adapted | Licence | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | [uncompresspy](https://github.com/kYwzor/uncompresspy) (Tiago Gomes) | The LZW decoder for `.Z` files ([notice](https://github.com/archivey-dev/archivey/blob/main/src/archivey/internal/streams/codecs/unix_compress_decoder.py)) | BSD-3-Clause | | [rarfile](https://github.com/markokr/rarfile) (Marko Kreen) | RAR3 password key derivation and the RAR3 Unicode filename decoder ([notice](https://github.com/archivey-dev/archivey/blob/main/src/archivey/internal/backends/rar_parser.py)) | ISC | The ARM64 and RISC-V branch filters used in 7z archives follow the algorithms in liblzma, from the [xz](https://github.com/tukaani-project/xz) project, and are checked against 7-Zip's. ## Projects archivey learned from [py7zr](https://github.com/miurahr/py7zr) (Hiroshi Miura and contributors) and rarfile were the main references for how 7z and RAR archives behave in practice. Archivey's 7z and RAR readers were checked against them at every step and take ideas from them, though not code (apart from the rarfile pieces above). Python's [`zipfile`](https://docs.python.org/3/library/zipfile.html) and [`tarfile`](https://docs.python.org/3/library/tarfile.html) played the same part for ZIP and TAR: archivey read both formats through them before it had its own readers, which still follow them in places. [7-Zip](https://www.7-zip.org/)'s and [RARLAB](https://www.rarlab.com/)'s UnRAR sources were the references for the formats themselves, and WinZip's AES specification for encrypted ZIP files. The indexed and seekable decompressors ([python-xz](https://github.com/Rogdham/python-xz), [indexed_bzip2](https://github.com/mxmlnkn/indexed_bzip2), [indexed_gzip](https://github.com/pauldmccarthy/indexed_gzip), [ratarmount](https://github.com/mxmlnkn/ratarmount) and [indexed_zstd](https://github.com/martinellimarco/indexed_zstd)) shaped how archivey seeks inside compressed streams, even though archivey doesn't depend on them. ## Packages archivey uses Archivey itself needs no third-party package. Optional packages add formats and features, and [Install](https://archivey-dev.github.io/archivey/install/index.md) says which extra brings in which: - **[pyppmd](https://github.com/miurahr/pyppmd):** PPMd compression in 7z and ZIP. - **[inflate64](https://github.com/miurahr/inflate64):** Deflate64 in ZIP and 7z. - **[brotli](https://github.com/google/brotli), [lz4](https://github.com/python-lz4/python-lz4) and [backports.zstd](https://github.com/Rogdham/backports.zstd):** Brotli, LZ4 and Zstandard, as single files and inside 7z archives (Zstandard also inside ZIP). Python 3.14 and later reads Zstandard without backports.zstd. - **[cryptography](https://github.com/pyca/cryptography):** AES decryption for 7z, RAR and WinZip AES ZIP files. - **[pycdlib](https://github.com/clalancette/pycdlib):** ISO images. - **[tqdm](https://github.com/tqdm/tqdm):** progress bars in the command-line tool. - **[rapidgzip](https://github.com/mxmlnkn/rapidgzip):** faster seeking in large gzip, bzip2 and DEFLATE data, such as ZIP members, and faster reading of large DEFLATE data where it pays off. Compressed or encrypted RAR data is read with RARLAB's `unrar` (or `rar`), or with [The Unarchiver](https://theunarchiver.com/)'s `unar`. [Install](https://archivey-dev.github.io/archivey/install/index.md) says which one runs when. ## Projects archivey checks itself against Tests compare what archivey reads with py7zr, rarfile, `zipfile` and `tarfile`, field by field, and can also read [libarchive](https://github.com/libarchive/libarchive)'s own test archives through [libarchive-c](https://github.com/Changaco/python-libarchive-c). 7-Zip, RARLAB's `rar`, [ncompress](https://github.com/valgur/ncompress) and several of these libraries write archivey's test archives. Three of those archives are copied from other projects: two old RAR archives from rarfile and an LZ4 7z archive from py7zr, each credited in its folder's README. You don't need any of them to use archivey. ## Tools archivey is built with Archivey is built and tested with [pytest](https://github.com/pytest-dev/pytest), [Hypothesis](https://github.com/HypothesisWorks/hypothesis), [Atheris](https://github.com/google/atheris), [Ruff](https://github.com/astral-sh/ruff), [Pyrefly](https://github.com/facebook/pyrefly), [ty](https://github.com/astral-sh/ty) and [MkDocs](https://github.com/mkdocs/mkdocs), among others. Thanks to everyone who maintains them.