Detecting Duplicate Tracks With a Fingerprint I Wrote Myself

A friend handed me a 2,100-track folder from an old laptop. Half of it was the same 300 songs ripped twice, once at 128 kbps and once at 320, with slightly different file names. File hashes were useless; tags were a mess; duration matching flagged 40% of the library because so many tracks were the same length. I wanted something that said “these two files are the same recording” without trusting anything I had typed into a tag field.

I used a cheap acoustic fingerprint: mono, 8 kHz, one spectrogram per ten-second window, and a hash of the strongest local peaks. Nothing clever. The whole thing runs on ffmpeg plus a short Python script.

import subprocess, numpy as np, hashlib

def pcm(path, rate=8000, seconds=10):
    raw = subprocess.check_output([
        'ffmpeg', '-v', 'quiet', '-i', path, '-t', str(seconds),
        '-ac', '1', '-ar', str(rate), '-f', 's16le', '-'])
    return np.frombuffer(raw, dtype=np.int16).astype(np.float32)

def peaks(path, bands=24, top=6):
    x = pcm(path)
    win = 2048
    frames = [x[i:i+win] for i in range(0, len(x)-win, win//2)]
    spec = np.abs(np.fft.rfft(np.array(frames), axis=1))
    edges = np.linspace(0, spec.shape[1], bands+1).astype(int)
    out = []
    for lo, hi in zip(edges[:-1], edges[1:]):
        band = spec[:, lo:hi].mean(axis=1)
        for idx in np.argsort(band)[-top:]:
            out.append((int(idx), int(lo)))
    return sorted(set(out))

def fp(path):
    h = hashlib.sha1()
    for f, b in peaks(path):
        h.update(f'{f}:{b};'.encode())
    return h.hexdigest()

Then I grouped by fingerprint and kept the largest file in each group, printing the rest as candidates. On that folder it found 640 duplicates and left 12 false positives, all live versions of the same song that happened to open with the same ten seconds.

The reason this works is that the first ten seconds of a recording carry the arrangement, not the bitrate. A 128 kbps MP3 and a 320 kbps copy of the same master produce nearly identical peak positions once you throw away everything above 4 kHz. The cost is that quiet intros, spoken-word tracks, and anything with a fade-in will collide or vanish. I cap the comparison at ten seconds because longer windows made the run slower without finding more duplicates; two minutes per file was the point where I stopped caring.

Where it breaks: remasters, radio edits, and anything with a different intro. A 2011 remaster of an album I own from 1994 will pass as a duplicate about 60% of the time in my testing, which is why I print candidates instead of deleting automatically. Live bootlegs are the other trap; the 12 false positives were all from the same tour. And if a file has more than a few seconds of silence at the head, shift the window before hashing or you will match every track that starts quietly.

What I still do not have a good answer for: how do you tell a remaster from a re-rip of the same master without listening to both? If you have a rule that survives a whole shelf of 90s reissues, I would like to hear it.