What a File Hash Can Promise

A digest can tell you whether a copied file still matches an earlier reference. It cannot provide the source, history or meaning of a reference you never verified.

A hard disk and its USB external enclosure
In this story 5 sections

Two copies of a folder can look identical in a file browser. The same names appear in the same order. A picture opens with the same mountains at the edge and trees at the center. The first difference may be in something you cannot see in the image at all, like a line of metadata that a program rewrote when it saved the file. A damaged spot might also be present in material that the program does not display until later. A useful integrity check answers a question that a quick look cannot: does the new file contain the same sequence of bytes as the one I intended to copy?

A cryptographic hash function takes data and produces a digest of a defined length. NIST specifies algorithms such as SHA-256 in its Secure Hash Standard. Given the same sequence of bytes and the same algorithm, you get the same result. A different sequence typically gives a widely different-looking value. Keeping an expected digest for a file and calculating a new one gives you a fairly economical basis for comparison: you can keep the small reference and check a much larger item against it when needed. An archive can use that for a few photos or for collections whose individual files would take far too long to open by hand.

A name can change with none of the bytes

Take a local file whose content you control. Make a second copy under a different name and keep both without opening and saving them in a program that might change them. A normal hash calculated over just the contents should match for the two files. The different names tell the file system where to find them, not what the two sequences contain. If you need to preserve names, paths and other context as part of an archive, those things deserve separate records. An integrity check over file bytes does not secretly include every property that an application might display beside the file.

The inverse case is a file with the same name and size but different content. Change one character in a plain text file to another character with the same size in its encoding. Many details in a file browser might remain identical. Its hash should change. That tells you much more about the contents than comparing the two sizes, though it still says nothing about the location of the change. The same property makes a hash less convenient for a person whose question is what changed? Use a comparison tool that understands the relevant kind of data when that difference matters.

View the hard disk in the picture as one possible carrier, alongside a second disk, a file server or other storage. A disk can keep a sequence faithfully even if the photograph has no way to show what that sequence is. Conversely two quite different pieces of equipment can produce identical file contents. That independence is why an archive can verify that a copy survived a move to another device without asking what the devices look like or how they organize their storage internally.

Establish the reference when it has a purpose

An archive often starts by recording accepted data when it arrives. A file name or path, the algorithm and the expected digest can appear on a line in a manifest. Other information can record where the file came from and when the reference was made. The Library of Congress discusses keeping and comparing this kind of information as part of digital preservation. If a package arrives with a digest from the sender and the archive calculates a matching one, the comparison is useful evidence of consistency with that transfer record. It does not establish everything about how the sender got the content.

The time at which the reference was made matters. Suppose a home video was copied in May and the expected hash was not calculated until September. The September digest can give a baseline for later checks. It cannot retroactively certify that the May transfer included all the original material. A reference obtained on the original copy before moving it answers another question. Write down which copy you hashed and under which directory or version. A date on its own can fail to distinguish two quite different uses of the same number.

There may also be more than one file the archive wants to preserve. The original image and a smaller public derivative are each legitimate objects for verifying their respective contents. They are not supposed to match one another. Capture a distinct digest for each and record the relationship, rather than reporting a failed comparison whenever a viewing copy differs from the original. The hash can establish that a derivative has stayed as it was originally prepared, while separate documentation explains why someone created it and what was changed to do so.

A good match is conditional evidence

A fixed-size result cannot uniquely distinguish every possible arbitrary input. Different inputs can share a digest in principle. Modern cryptographic hashes are designed to make particular ways of finding such overlaps computationally difficult, with the security of the actual algorithm in view. For checking a file copy under ordinary conditions, a match with a suitable algorithm is strong evidence of the same contents. It would be inaccurate to treat the small value as a reversible or perfectly unique form of the file. The original cannot be reconstructed from that value.

More often the weakest part of the check is the status of the expected value. A match against a reliable published reference is useful. A value generated from an unknown attachment after it arrived proves nothing about whether the sender gave you the right attachment. When verifying a file offered by another party, get reference information through a trusted source and make sure it is for the named edition or version. A digest by itself carries no binding assertion that a person or organization endorsed it. A system for authenticated publication or signing can provide additional evidence that an isolated string does not.

There is another practical way for a match to mislead. An application may calculate a value over a folder package while the comparison value belongs to an individual file inside it. If the two do not agree, there was no meaningful expectation they would. This is especially common when an archive has both compressed and unpacked forms. The file boundaries, algorithm and version have to agree before the outcome can be interpreted. Record those three facts in a useful order. The exact text of the digest is the final item on that line, after the choices that tell another person what it is for.

The package needs a list as much as a digest

A manifest with a hundred listed files may help detect a hundred altered files. It may also help detect one that simply did not get copied. A digest calculation will not run over a file that is absent and give you a neat comparison for it. Enumerate the expected paths and compare them with the set of items received, then check the contents of those that are present. Do the same for unexpected files a tool created as a side effect. A check that reports every tested item as good can be quite accurate while leaving one of those questions unanswered.

Keep a completed report distinct from the manifest used to generate it. The report says which comparison was made, on what date and what was found. The manifest provides the basis for comparison. They can be useful to different people after a restoration, and one need not overwrite the other.

Handle disagreements without rewriting history

Suppose a routine check finds that a saved image no longer matches. Keep the existing manifest, and do not replace the reference with the newly observed value just to make the report clear. Compare against known separate copies, look at the recorded date and note whether an intentional process could have produced a new version. A mismatch is an observation about the two values and the items from which they were obtained. It can justify further inquiry without yet establishing whether there was damage, an unexpected saving operation or confusion of files.

A clean recovery should leave the archive with a record of what happened. If a copy was restored using another verified source, the new check can say that. If a change was intentional, record a new version rather than erasing the earlier digest that carries different evidence. Keep the reference somewhere it can be recovered and checked even if the storage holding one copy of the original fails. In a small personal archive, this can matter as much as having a second drive: the extra copy is more useful when you have a reasonable way to establish that it still contains the intended files.

The mountains in the two pictures may look identical again after all this work. That outcome is compatible with a match or mismatch, depending on what the bytes contain and how the application renders them. A good record makes the question clear enough that the outcome means something either way. You can use visual comparison for what changed in the view, and a known hash for whether the file at this stage is the file the reference was made to identify.

Further reading