We're in the second week of Wildcard Month, this time looking at the process of what I think of as data triage. I'm in the middle of an ongoing data triage project, so the subject is fresh in my mind.
Although the ideas this time apply most directly to situations where you have multiple hard drives and layers upon layers of potentially overlapping backups, the same concepts work in other situations involving data organization. But you might want to tweak certain steps.
Wildcard Week 2: Data Triage
Imagine for a moment that you have a collection of hard drives pulled out of the dozen or so different personal computers you've owned over the last 20 years.
Each time you upgraded your computer, you brought the old drive(s) along into the new machine. Each time the technology changed enough to be incompatible, you did your best to copy the stuff from the old tech onto the latest tech first, ending up with folder names like "OldLaptop2012\Archive3" with little rhyme or reason.
Sometimes, you had to keep working with some of the files from the old backups. When this happened, sometimes you copied the old data to a new location first. Sometimes, you moved it instead of copying it. And sometimes, you just used it right where it was in the haphazard backup folder structure.
The result is something like this:
- A stack of 20 hard drives
- A few dozen sharpie-labeled CDs and DVDs
- A random assortment of mostly unlabeled USB flash drives
- The knowledge that you have between one and six digital copies of everything
- The knowledge that probably 90% of what you have is not important enough to keep
- No easy way to differentiate between the important and unimportant or to organize the valuable stuff
This is the situation I ended up in--mostly because I allowed myself to fall into it without taking steps to avoid it.
To get out of this situation, we need to engage in a process I call "data triage." I use the term triage because it communicates the reality that certain portions of our data might never make sense to spend time on. Prioritize the things that matter most, and only move on once you finish those tasks.
Identify, Filter, Organize
Some of the earliest Tidy Bytes material describes the IFO method for cleaning up your data: Identify, Filter, Organize. In short:
- Identify: Where is it?
- Filter: Can I get rid of it?
- Organize: Where should it go?
The data triage process mainly concerns the first two steps. If you have a stack of old physical media (hard drives, CDs, DVDs, flash drives), you already know where your data is in one sense. Still, you need to put it somewhere to filter through it efficiently. While you can process one device at a time start-to-finish, this is slow and painful--especially if you know you have overlapping backups spread across multiple drives.
So, what's the alternative?
Data Triage Drive
Because storage continues to get significantly cheaper over time, the most convenient solution is to obtain a dedicated hard drive specifically for performing data triage. As of late 2024, you can get an 8-terabyte (TB) mechanical hard drive for around $120. This won't be as fast or power-efficient as the latest SSD/NVMe storage technology, but it's significantly cheaper and will work perfectly well for data triage. As a bonus, if you have something like a Synology NAS device with an empty drive bay, you can add this drive to it once you finish the triage process!
Now, I use the 8TB size as a reference point; you may have far less data than that, so something like 1TB or 2TB would be enough; or, you might have far more data than that, so you need 2-3 times as much capacity. If you add up the maximum capacity of your old drives, that will be your upper limit on what you might use; however, most drives are only partially full of data. The goal is to get whatever drive(s) you need to contain everything you want to filter through simultaneously.
If you realize later that you can't fit everything you wanted to, you can either pre-delete some content more aggressively (see below) or else go through this process twice, once for each set of data. The second time will be much faster, and twice is still much better than doing the entire process for each drive.
The drive(s) you get can be internal (SATA interface) or external (USB interface), according to your preference. Internal drives will likely be faster for some of the data searching and deduplication processes that must happen over potentially millions of files, but either will work well enough.
Centralize Everything
Once you have your drive(s) ready to use on the computer you'll be using for data triage, the next step is to copy (NOT move) the original data over to the triage drive. The exact steps look different depending on whether you're using Windows, macOS, or Linux, but this is the process:
- Pick the next unprocessed storage media device (hard drive, CD, DVD, flash drive, etc.)
- Physically label it if it doesn't have a label--even something generic like "Drive1" is fine, as long as it's unique
- Create a folder in the top level (root) of your triage drive to match the label you made for this media
- Copy (again, NOT move) the entire contents of the original media to the named folder on the triage drive
- Disconnect/remove and set aside the original media
- Repeat these steps until you run out of media
Preparing the triage process like this has two significant advantages over trying to process one drive at a time start-to-finish:
- You can deduplicate the entire dataset in one pass (with the right software)
- If something goes wrong, the original data is still intact
Advantage #1 saves a lot of time in the long run. Advantage #2 provides a backup for peace of mind. It's why we always copy instead of moving the data at first, even though the ultimate goal is to wipe and repurpose (or discard) all of the old media. But that only happens at the end of the triage process.
One important note: if your physical media is on the brink of mechanical failure--more likely for older drives--you might want to create a raw image duplicate of the entire drive first using a tool like R-Drive Image from R-TT Tools. This requires more technical knowledge, but it's possible. Feel free to contact me if you have questions about this process.
Pre-Delete Pass
Before you set the automated deduplication tools to work, it's usually worthwhile at this stage to use your computer's file explorer tool to quickly browse through some or all of the data you've just copied and delete anything obviously unimportant. Doing this can be more or less valuable depending on whether you've held onto the entire contents of each drive (including the operating system and installed programs) or whether your backups primarily contain only your personal data like documents and photos.
The main reason to do this is that it speeds up the deduplication process, and you won't have to do it later. If you have eight different hard drives, each with a copy of Windows installed, most of that operating system data will be identical and irrelevant. Cleaning it out beforehand takes only a few minutes if you know what to look for.
The big things you can usually remove safely reside outsidepersonal user folders. On a Windows PC, for example, the "\Windows" and "\Program Files" (as well as "\Program Files (x86)" folders contain the generic part of the operating system and installed programs. You can delete these folders and their contents without consequence unless you have specific reasons to keep them.
(BIG IMPORTANT NOTE: DON'T DELETE THESE FOLDERS FROM DRIVES IN COMPUTERS THAT ARE STILL IN USE. THE ABOVE COMMENT APPLIES ONLY TO OLD DRIVES THAT CAME OUT OF COMPUTERS NOBODY IS USING ANYMORE.)
Most personally important files usually reside somewhere inside one main folder on the drive, depending on your operating system:
- Windows: \Users...
- macOS: /users/...
- Linux: /home/...
Those folders will also likely have plenty of irrelevant stuff inside, but that's not something to worry about now.
Deduplicate Everything
After you've gathered all the data from all your storage media into a central triage drive and (optionally) performed a pre-delete pass, it's time for the main event: deduplication. This is especially important if you, like me, have accumulated layers of archived data rolled forward across a dozen computer upgrades. There's probably a lot of redundant data.
The good news is that software tools exist to address precisely this problem. One of the more popular options for Windows is called Duplicate Cleaner from Digital Volcano, a UK-based company. It's a powerful program and does its job exceptionally well. Although it's not free, a free trial is available, and the paid version is not prohibitively expensive: a $45 one-time purchase. Considering the amount of time it's likely to save--dozens of hours, at least, if you have lots of data--it's easily justifiable. For Mac users, one option is Duplicate Finder from Nektony. This is fundamentally similar to the previously mentioned application for Windows.
As it happens, I've written before about using deduplication software; for more detail, check out this Tidy Bytes post.
At this stage, you should be able to instruct whatever duplicate-cleaning software you've chosen to run across the entire triage drive and just let it go. Depending on how much data you've prepared, this could take anywhere from 15 minutes to many hours.
Once the process is complete, the deduplication software will likely recommend that you delete all but one copy of every file it finds. Since you still have the original media as a backup (right?), you should be safe to follow this recommendation. Again, depending on how much data there is, even the deletion process could take quite a bit of time--though it will be faster than the duplicate detection step.
That's it! You've finished the data triage process. The remaining files still need to be organized, but you have a clean collection devoid of duplicates. You can move this deduplicated data collection to a more permanent home (like a NAS or other backup drive) or leave it on the triage drive for further cleanup and manual organization.
Also, once you're satisfied that the data is sufficiently triaged and safely stored somewhere that a single catastrophic failure (like a triage drive crash) won't eliminate the only viable copy, feel free to dispose of the original media.
No more files spread across decades' worth of aging tech! Well done!
I know data triage is one of the more tech-heavy topics, so I'd love to field any questions that you might have about it. Just use the contact form on the Tidy Bytes website.
Have a great week, and happy data-taming!