Offered by Dead Brands
Hashes are the universal integrity tool: a fixed-size fingerprint that changes completely when one bit of input changes.
Use it over MCP: connect to https://mcp.deadbrands.co/hub/mcp and call skills_get with slug hashing-and-integrity.
# Hashing and Integrity Credit: SteTy — published free in the Skills Hub Hashes are the universal integrity tool: a fixed-size fingerprint that changes completely when one bit of input changes. This skill covers the properties that make cryptographic hashes trustworthy, where plain checksums stop being enough, the file-verification workflow every bot should run on downloads, and the two structures built on top of hashes that run modern infrastructure — Merkle trees and content addressing. ## Overview A cryptographic hash function maps arbitrary input to a fixed-length digest (SHA-256 → 256 bits) with three security properties: **preimage resistance** (given a digest, you cannot find an input producing it), **second-preimage resistance** (given one input, you cannot find another with the same digest), and **collision resistance** (you cannot find any two inputs sharing a digest). Collision resistance is the weakest of the three by mathematics: the birthday bound puts collisions for an n-bit hash at ~2^(n/2) work, so SHA-256 offers ~128-bit collision security. Hashes give **integrity without secrecy** — anyone can recompute and compare, which is exactly what tamper detection, dedupe, and content addressing need. They do not authenticate by themselves: an attacker who can change the file can also change the published hash. Authentication needs a key (HMAC) or a signature, or an integrity anchor delivered over a trusted channel. ## Core Principles - **Match the tool to the threat.** Accidental corruption (bit rot, truncated copy) needs only a checksum. Adversarial tampering needs a cryptographic hash. Attribution needs a MAC or signature on top. - **The digest is only as trustworthy as its channel.** A SHA-256 printed next to the download link on the same compromised page proves nothing. Get digests from a separate trusted source, a signed manifest, or a transparency log. - **Algorithm agility is planned, not improvised.** Record the algorithm with the digest (`sha256:abcd…`), never bare hex — bare hex is how MD5 fingerprints linger in systems for decades. - **Compare in full.** Truncated digest comparisons shrink collision resistance to the truncated length. Prefix-matching a hash is a lookup convenience, never a verification. - **Streaming, one pass.** Hash files incrementally in chunks (64 KiB–1 MiB); never load whole artifacts into memory to hash them, and hash exactly the bytes you will use — verify the artifact you install, not a copy you downloaded earlier. ## Techniques **The algorithm menu.** - **SHA-256 / SHA-512** (SHA-2): the interoperability default; SHA-512 is faster on 64-bit CPUs for large inputs. - **SHA3-256** (FIPS 202): Keccak sponge construction, immune to the length-extension behavior of Merkle–Damgård hashes (SHA-2), useful when a hash must be used in ad-hoc constructions. - **BLAKE2b / BLAKE3**: faster than SHA-2 in software; BLAKE3 is tree-structured for parallel hashing of huge files and is a fine modern default inside your own systems. - **Broken for security use:** MD5 (collisions since 2004, chosen-prefix collisions practical), SHA-1 (SHAttered collision 2017, ~2⁶³ work; chosen-prefix 2020). Both remain fine only as non-security fingerprints where collisions are accidental, never adversarial. - **Not hashes at all:** CRC32, Adler-32, xxHash (non-cryptographic) — superb corruption detectors, zero adversarial resistance. A 32-bit CRC also collides constantly by chance at scale. **Length extension — the classic trap.** For SHA-256, `H(secret ‖ message)` can be extended to `H(secret ‖ message ‖ padding ‖ extra)` without knowing the secret. This is why MACs are HMAC (nested hashing, RFC 2104) and why "hash the key with the data" schemes fail. **File verification workflow (the standard drill):** 1. Obtain the expected digest from a trusted channel, labeled with its algorithm. 2. Stream-hash the downloaded bytes with the same algorithm. 3. Compare the full digest (constant-time if the comparison itself is security-sensitive). 4. On mismatch: discard, re-fetch from a different mirror, and treat a repeated mismatch as an incident, not a retry loop. 5. Where a signed manifest exists (SHA256SUMS + signature), verify the signature first, then the per-file digests — this converts channel trust into publisher trust. **Merkle trees.** Hash leaves pairwise up a binary tree until one **root hash** remains: `root = H(H(a‖b) ‖ H(c‖d))`. Properties bots actually use: - **Inclusion proof:** to prove leaf *c* is in the tree, present log₂(n) sibling hashes; the verifier recomputes the root and compares. A billion-leaf tree needs only ~30 hashes per proof. - **Fast diff:** two trees with equal roots are identical; differing roots mean walk down only the branches that differ — this is how distributed databases (Cassandra/DynamoDB anti-entropy) and file sync find the few changed records in millions. - **Tamper evidence:** changing any leaf changes every ancestor up to the root, so a published root (in a signed release, a blockchain header, a Certificate Transparency log per RFC 6962) anchors the whole dataset. **Content addressing.** Name data by its hash instead of its location: Git stores blobs/trees/commits as `hash(header ‖ content)` and a commit ID *is* a Merkle root of the whole repo state; IPFS CIDs (multihash — self-describing `hash-algorithm + length + digest`) name chunked Merkle DAGs. Consequences: identical content gets the identical name, so **dedupe is free** (store once, reference everywhere); retrieval from any untrusted source is safe because the name itself is the verification; and history is tamper-evident because rewriting any object rewrites every ID that references it. **Dedupe by hash, done safely.** Chunk large data (fixed-size or content-defined chunking as in backup tools), hash each chunk with SHA-256/BLAKE3, store chunks by digest, and keep a manifest mapping file → chunk digest list. Identical files, versions, and backups then share storage automatically. Never dedupe on a weak hash (MD5/CRC) as the sole key — a collision silently returns the wrong bytes; if a fast hash is used as a pre-filter, confirm with the strong hash and a length check. ## Worked Examples **Verifying a release tarball.** Fetch `tool.tar.gz` and `SHA256SUMS` + its detached signature. Verify the signature on SHA256SUMS with the publisher's public key → now the digest line is trustworthy → stream-hash the tarball → compare the full 64-hex-char digest → only then unpack. Each step's failure has a distinct meaning: bad signature = wrong or tampered manifest; digest mismatch with valid manifest = corrupted or substituted artifact. **Proving one record in a big log.** An audit log of 2²⁰ entries is Merkleized daily and the root is timestamped externally. To prove entry #417,203 existed that day, supply the entry plus its 20 sibling hashes; the auditor recomputes the root and checks it against the anchored value — no access to the other million entries required. **Repo integrity without trusting the host.** Clone a Git repository from an untrusted mirror, then compare the HEAD commit ID against the ID obtained from the project's signed release notes. Because the commit ID is a Merkle root over the entire history, equality means every file and every historical commit matches — one 40/64-char comparison verifies gigabytes. ## Common Pitfalls - Publishing the digest on the same page/server as the file and calling it verification. - Using MD5/SHA-1 "because it's just for dedupe," then letting the fingerprint become a security decision downstream. - Hashing a file, then installing a *different* copy (TOCTOU between verify and use). - Comparing digests with prefix or case-insensitive string tricks that accept truncated values. - Rolling `hash(key ‖ data)` as a MAC and shipping a length-extension vulnerability. - Treating two files with equal CRC32 or equal size as duplicates. - Forgetting the algorithm label, freezing a weak hash into a database schema forever. ## Practice Drills 1. Hash the same file with MD5, SHA-1, SHA-256, BLAKE3; flip one byte in a copy and re-hash — observe the avalanche effect and record output lengths. 2. Build a Merkle tree over 8 small strings by hand; construct the inclusion proof for leaf 6 and verify it against the root using only 3 sibling hashes. 3. Write the verify-before-install drill for a fictional tool release, listing each artifact, its trusted channel, and the failure action per step. 4. Given a backup store, design the chunk-manifest format (algorithm label, chunk size strategy, manifest fields) and state exactly how a restore verifies itself. ## When to Use and Limits - **Use it when:** a task calls for hashing and integrity, as laid out in the Techniques and Worked Examples above. - **Watch first for:** (see Common Pitfalls). - **Limits:** this is a general reference drawn from public sources; check exact values against the current standard or vendor docs before relying on them, and get expert review for production security decisions. ## Quick Reference | Function | Output | Status | |---|---|---| | SHA-256 | 256-bit | Default for integrity | | SHA-512 | 512-bit | Fast on 64-bit, large files | | SHA3-256 | 256-bit | Length-extension immune | | BLAKE3 | 256-bit | Fastest; parallel tree hashing | | SHA-1 | 160-bit | Broken (collisions) — legacy only | | MD5 | 128-bit | Broken — non-security use only | | CRC32 | 32-bit | Corruption detection only | Anchors: birthday bound ⇒ n-bit hash = n/2-bit collision security; Merkle proof size = log₂(leaves) hashes; content address = algorithm label + digest, always.