Checksums for video offload: xxHash vs MD5, and what an MHL file is
A checksum shows that a copy of a file matches the original. MultiSyncer Studio computes it for every file on the card and in every copy and compares them. For video xxHash is the most convenient: in our test it is more than 10 times faster than MD5, so the check barely slows the copy down.
What a checksum is
A checksum (hash) is a short string of letters and digits that the app computes from the entire contents of a file. If even one bit of the file changes, the checksum comes out different. So the checksum is computed twice: from the file on the card and from its copy on the drive. If the checksums match, the copy is the same as the original.
Simply dragging files in Finder or Explorer computes no checksum: a file can be copied with an error and still look fine. More in the article “Memory card offload on set: two copies and checksum verification”.
Which checksum to choose
| Checksum | What it is | When to choose it |
|---|---|---|
| xxh128 | A modern fast checksum of the xxHash family, 128-bit: an accidental match is even less likely | The default in new MultiSyncer installs; fits almost always |
| xxh3 | Just as fast, shorter checksum (64-bit) | If that is what your team uses |
| xxHash64 | The usual DIT standard | When that is what your team or client uses |
| MD5 | An old, widespread checksum, noticeably slower to compute | If the client requires it |
| SHA-1 | An old, widespread checksum, slower than xxHash | If the client requires it |
The app puts it this way: “xxHash64 is the usual DIT standard. xxh3 and xxh128 are newer and faster. MD5 and SHA-1 if the client requires them.”
How fast the checksums are
We measured the speed with the same code MultiSyncer uses: a 1 GB file in memory, three runs.
| Checksum | Speed, MB/s |
|---|---|
| xxh128 | about 9,500 |
| xxh3 | about 9,400 |
| xxHash64 | about 8,600 |
| SHA-1 | about 2,500 |
| MD5 | about 800 |
What this means on set. xxHash computes more than 10 times faster than MD5 and is usually faster than the card can be read and the drive written. So with xxHash the copy speed is limited by the card and the drives, not by the checksum. MD5 (about 800 MB/s) can slow down fast cards such as CFexpress. If that happens, the app tells you that “the checksum holds it back” and that xxh3 and xxHash64 are faster than MD5 and SHA-1.
What an MHL file is
MHL is a list of the files in a copy with their checksums. It sits next to the copy, and anyone can use it to check the copy later: on another computer, in the edit room, a month later.
MultiSyncer writes two lists next to every verified copy:
- ASC MHL v2: the history in the
ascmhlfolder inside the copy, with a chain of checks; - MHL 1.1: a file in the older format next to it.
Check the copies later. In the day panel each card has a “Check copies” button: the app reads all copies of the card again and checks them against the checksums (MHL).
How it works in MultiSyncer Studio
Choose the checksum in the project setup
Section “Copying and folders”, line “Checksum”: xxHash64, xxh3, xxh128, MD5 or SHA-1. New installs default to xxh128.
Copy the cards
Every file is copied to the selected drives, from one to four. Then comes the “Check” stage: every copy is read back and checked against the card by checksum.
If the checksum does not match
The app reads and copies again; no half-read files are left behind. If the copy still does not match, the app reports “Checksum mismatch”.
Format only what is verified
“Safe to format” appears only with two verified copies on different drives. A cloud folder does not count as a copy.