Did you know you had around 200,000 jpeg files on your machine before it wen't kaput? No, neither did I...
You have your data back now, but it's missing all context. My data was important to me. I wanted it back very badly, and I was willing to put in the elbow grease to get it back properly. Note that this process took me around 10 days to complete, and I still find bits of data called 'Recovered_*_file...' to this day.
Windows Explorer will be your friend in this battle against thousands upon thousands of files, so keep things organised to try and avoid any mistakes. Use this opportunity to restructure your PC in a better way; something I had been meaning to do on and off for years.
Adobe Bridge
Windows Explorer works, but it can be slow. Adobe Bridge on the other hand, is a very robust, simple replacement of Explorer.exe. Much like Explorer, you can use Bridge to preview files and folders, but it's much better at it. It can show you a thumbnail of a much wider ranger of file formats. It is much faster to load folders full of thousands of files, and its batch processes for moving files around and renaming them is fantastic. It is also completely free! The renaming function will be your best friend here. It gives you every option under the sun as to how to rename things, and is especially useful with large numbers of files that are all related, such as holiday photos etc.
dupeGuru
A strange side effect of recovering data is that a lot (but not all?) of files, especially photos I found, will have a duplicate or 3 somewhere else. I've had the issue of having duplicate files in the past, but always just tried to track them down and delete them. However when working with so many files, it becomes impractical. Thus, I found this online: dupeGuru. Get it, its free, and a massive time saver! Literally point it at a folder full of photos you're pretty damn sure you've just spotted dozens of duplicates in, tell it to search 'Pictures' (it can do other stuff, but this was by far the most useful to me), and launch it off! It scans the structure of every file and compares them, leaving you with a hit-list of files it knows are 100% copies of eachother, and some others its around 96% - 99% sure on. My suggestion is to arrange the list by %, and manually check the 96% - 99% ones. For me, the 96% ones were close matches but not dupes, so it's your call on those. The 99% sure ones often turned out to be 100% right, so I ended up just spot-checking most of them and telling it to remove the rest. Processing the files is easy. I'd select them all to begin with, then uncheck the ones you don't agree are dupes. You cannot directly delete the files from dupeGuru, probably just in case. Instead, you can move them to a new folder, and then it's down to you if you really do delete them.
The Results
These two applications together allowed me to figure out 99% of the work, and 10 days later, everything was back on track. A heads up on this whole process though, many files were still lost forever. Windows wiped out files I'll probably not notice for years to come until I go looking for X. Other files were successfully restored, but were actually corrupt. This has only happened with .jpg files for me so far. I guess the format can be lenient, so it's still readable but actually scrambled when you look at it. These files would display with green and pink bars and artefacts across them. In these cases, you simply had no other choice other than to delete them. A few .png and .gif files I had looked fine and opened in Windows Fax Viewer or Windows Photos, but Photoshop said something was wrong with them. I expect to be finding these for years to come such is their nature. However they are fully recoverable. Simply open them in Paint. Nice simple, shitty Paint. It seems to open them just fine, then simply resave them, overwriting the file or saving as a new file altogether. After this, they are as good as new!
Looking Forward
Back up everything. There's plenty of backup software out there these days, but ASCOMP's Backup Maker and ASCOMP's Synchredible are both brilliant shouts. They are both fully featured and free. The only time you have to pay for them is if you're a business or want user support, but Google is your friend 99% of the time. Personally I now have a spare 2TB drive in my PC that does nothing but store backups. I started with Backup Maker, but actually found Synchredible to be what I was actually looking for. Read into both of them at your leisure, I go from there. Basically if I add, remove, or edit anything Synchredible is looking at, it will automatically update a backup of it on my other disk every time Windows starts up again. It's great!
Nothing will beat cloud storage though. Think of it like this: worst case scenario, your house burns down or everything gets robbed. Your PC is gone. No matter how many drives you had backing up data, no matter how mnay external drives you had in your drawer next to your pc, all your data went with them. Cloud services mean you could travel to the other side of the world at a moments notice and still have access to your files you need to work. The catch for me and many others is simply uploading that data to begin with. Even uber-fast internet in the UK is slow at uploading unless you're a company that pays for this. It's simply not something providers prioritise for the standard user. But once it's up there, uploading a file here and there at the end of each week is nothing in comparison.
Dropbox is a tad expensive these days, but it used to be my go-to. It always felt faster than OneDrive back when that was in its infancy, but this is no longer the case. OneDrive will give you 1TB of online storage and Office 365 for those that want it for £8 p/m. Not a bad deal. Google Drive will give you double the storage (2TB) for the same price, but no Office 365. I went with Google in the end because all I need is space, and it made sense seeing as my internal backup drive is also 2TB, and so is my external hard drive; all could fit duplicates of the data on them if necessary.
Never rely on a hard drive to not fail, and backup now while it's still on your mind and before it's too late.
Saturday 25 April 2020
Recovering Lost Data from a Hard Drive
With Windows thinking it'd be funny to trick me into wiping my data backup to use as a repair disk, and then overwriting my main data drive with a new copy of Windows 10 no one asked it to do, I was in dire need of some recovery software to at least attempt to salvage my data.
Recovery Software
I had never really looked into this before. I was only partially aware that it could even be done in theory, but it turns out there's a fair few bits of software out there for this. Some better than others of course.
My first point of call was to use the free software Recuva. I ran it in quick-mode first time find that it found a bunch of Windows related files (no doubt from Windows being installed on top of my data). I launched a deep scan and continued researching online. I found people saying that Recuva takes far too long and only recovers some data and not all file formats. With it still running and saying it would take around 8 hours to complete, I went to the highest rated paid software: Stellar Data Recovery.
Running this took about half the time and threw up results it could show me on its interface as it was going. Recuva was simply a loading bar, giving no indication that it was indeed finding things. Stellar's interface also has a preview ability to look at image files on the fly. I did notice very few of them would preview, but figured this was because it was still scanning at the time.
Stellar finished it's scan, as did Recuva. Stellar did find a lot more, and I had more trust in it purely because I could see what it was finding along the way. Despite this, exporting found data from both ran into the same issue. No recovered files would open. I looked at Stellar's paid options, and the high-end version included auto-repair tools for images and other document types. It wasn't cheap at around £100 though, and I was sceptical, so I Googled around the issue. Read the comments to this article by Stellar Data Recovery. Essentially, hundreds of people had the same issue: all the files recovered just would not open. Stellar's response to this was that they were clearly corrupt before you even used data recovery software, a notion that enraged many people in the comments and had many more pointing out what a load of bollocks that was after having paid £100 for software that essentially showed them their old data but wouldn't give it back to them. Needless to say, I decided against joining that pool of people who spent their money to gain nothing back, and called it a lost cause.
Success - miniTools!
I had one more piece of software that I found called miniTools. It had good reviews online also, so before finally accepting that everything was lost and I'd just have to get used to it, I installed this and gave it a whirl. Like Stellar, it could show you what it was recovering as it found them. As soon as it started, it began finding thousands and thousands of files. As it did so, it keeps them in a folder called 'RAW' data. Within that folder are a bunch more folders, each one titled with a file format such as ,jpg, .tif, .png, and so on. I headed into the JPEG folder to find thousands being listed, and each one could be previewed! I could see my photos, my screenshots, everything! The final test was to export a bunch of the recovered files and try to open them. Minitools lets you recover files while it is still scanning, which is cool. All photos opened just fine! For the first time in 2-3 days of testing, I could actually see fully restored files coming up! I was so happy!
The catch? Minitools lets you restore up to 1GB of data for free, then tells you to pay for the software to get more (£80 or so) The difference here is that you can restore files on the fly for free, and they actually worked. This is where the others fell down for me, so I really highly recommend this software! Of course £80 isn't exactly cheap either, but it proved that it would work beforehand, and knew it had me by the balls if I really did want back what it had just found.
The Process of Recovering
There's a few things to note about this process that I've learned along the way. Once miniTools had finished scanning my drive for recoverable data, it found around 5TB worth on a 3TB disk that was never even full... go figure? What happened becomes more apparent as you go on restoring data. There are essentially 2 sets of your data. One set looks just like your data did. It's all in the folder structures correctly, has all the correct metadata present, and so on. Granted, bits of this data were still missing after all this (installing Windows would have definitely overwritten some data for sure). However, if you restore this data, you'll quickly run into that good old issue of nothing opens. Not a single file. I'm not sure exactly how this is, but it makes a bit of sense. This data is all the metadata and heading data of everything on the disk, but none of it is functional. What you need is the RAW data.
In miniTools, after the scan is complete it tells you how many partitions of data it found. For me, it was about 12. I'm not sure what they all are, but most had random crap in them that I didn't recognise as mine at all. The first few had files and folder structures I recognised, but they didn't open. The very last partition is called RAW Files. This is what you need.
Selecting the file formats you care about, you can tell them to export to a selected drive. Note that if you haven't paid for miniTools yet, you can do so now without having to wait hours for it to rescan the drive again. If the file format you were looking for isn't there, you can change which file formats miniTools should try to find in its options. For example, .txt files are off by default. Turning this on however found over million .txt files I wasn't aware existed!
It should be noted that some file formats are not available for recovery. I work as a 3D artist, so I can only cast light on graphic formats that it couldn't find really. For the most part it was very good! It can discover Maya files, Max files, Photoshop files, and Illustrator files to say the least. But it couldn't discover .indd files, InDesign files. I found this a little odd seeing as they are part of the same family as Photoshop and Illustrator, but I assume there must be something more to them that makes them difficult to read in some way?
Another note that might be very helpful for general computer users alike, is that miniTools differentiates between jpeg files, and jpegs files taken specifically from a camera. This made finding holiday photos and such far more easier! It does this by reading files that have metadata attached to them; something that every camera will automatically do to photos you take with it.
Recovered_jpg_file_68752.jpg
Get used to seeing the above. Every file recovered will be named in a similar way. Every file will say it was made today. Every file will have no context of where it was or what it was used for. The hard part begins now, but hey, you got your data back at least!
Recovery Software
I had never really looked into this before. I was only partially aware that it could even be done in theory, but it turns out there's a fair few bits of software out there for this. Some better than others of course.
My first point of call was to use the free software Recuva. I ran it in quick-mode first time find that it found a bunch of Windows related files (no doubt from Windows being installed on top of my data). I launched a deep scan and continued researching online. I found people saying that Recuva takes far too long and only recovers some data and not all file formats. With it still running and saying it would take around 8 hours to complete, I went to the highest rated paid software: Stellar Data Recovery.
Running this took about half the time and threw up results it could show me on its interface as it was going. Recuva was simply a loading bar, giving no indication that it was indeed finding things. Stellar's interface also has a preview ability to look at image files on the fly. I did notice very few of them would preview, but figured this was because it was still scanning at the time.
Stellar finished it's scan, as did Recuva. Stellar did find a lot more, and I had more trust in it purely because I could see what it was finding along the way. Despite this, exporting found data from both ran into the same issue. No recovered files would open. I looked at Stellar's paid options, and the high-end version included auto-repair tools for images and other document types. It wasn't cheap at around £100 though, and I was sceptical, so I Googled around the issue. Read the comments to this article by Stellar Data Recovery. Essentially, hundreds of people had the same issue: all the files recovered just would not open. Stellar's response to this was that they were clearly corrupt before you even used data recovery software, a notion that enraged many people in the comments and had many more pointing out what a load of bollocks that was after having paid £100 for software that essentially showed them their old data but wouldn't give it back to them. Needless to say, I decided against joining that pool of people who spent their money to gain nothing back, and called it a lost cause.
Success - miniTools!
I had one more piece of software that I found called miniTools. It had good reviews online also, so before finally accepting that everything was lost and I'd just have to get used to it, I installed this and gave it a whirl. Like Stellar, it could show you what it was recovering as it found them. As soon as it started, it began finding thousands and thousands of files. As it did so, it keeps them in a folder called 'RAW' data. Within that folder are a bunch more folders, each one titled with a file format such as ,jpg, .tif, .png, and so on. I headed into the JPEG folder to find thousands being listed, and each one could be previewed! I could see my photos, my screenshots, everything! The final test was to export a bunch of the recovered files and try to open them. Minitools lets you recover files while it is still scanning, which is cool. All photos opened just fine! For the first time in 2-3 days of testing, I could actually see fully restored files coming up! I was so happy!
The catch? Minitools lets you restore up to 1GB of data for free, then tells you to pay for the software to get more (£80 or so) The difference here is that you can restore files on the fly for free, and they actually worked. This is where the others fell down for me, so I really highly recommend this software! Of course £80 isn't exactly cheap either, but it proved that it would work beforehand, and knew it had me by the balls if I really did want back what it had just found.
The Process of Recovering
There's a few things to note about this process that I've learned along the way. Once miniTools had finished scanning my drive for recoverable data, it found around 5TB worth on a 3TB disk that was never even full... go figure? What happened becomes more apparent as you go on restoring data. There are essentially 2 sets of your data. One set looks just like your data did. It's all in the folder structures correctly, has all the correct metadata present, and so on. Granted, bits of this data were still missing after all this (installing Windows would have definitely overwritten some data for sure). However, if you restore this data, you'll quickly run into that good old issue of nothing opens. Not a single file. I'm not sure exactly how this is, but it makes a bit of sense. This data is all the metadata and heading data of everything on the disk, but none of it is functional. What you need is the RAW data.
In miniTools, after the scan is complete it tells you how many partitions of data it found. For me, it was about 12. I'm not sure what they all are, but most had random crap in them that I didn't recognise as mine at all. The first few had files and folder structures I recognised, but they didn't open. The very last partition is called RAW Files. This is what you need.
Selecting the file formats you care about, you can tell them to export to a selected drive. Note that if you haven't paid for miniTools yet, you can do so now without having to wait hours for it to rescan the drive again. If the file format you were looking for isn't there, you can change which file formats miniTools should try to find in its options. For example, .txt files are off by default. Turning this on however found over million .txt files I wasn't aware existed!
It should be noted that some file formats are not available for recovery. I work as a 3D artist, so I can only cast light on graphic formats that it couldn't find really. For the most part it was very good! It can discover Maya files, Max files, Photoshop files, and Illustrator files to say the least. But it couldn't discover .indd files, InDesign files. I found this a little odd seeing as they are part of the same family as Photoshop and Illustrator, but I assume there must be something more to them that makes them difficult to read in some way?
Another note that might be very helpful for general computer users alike, is that miniTools differentiates between jpeg files, and jpegs files taken specifically from a camera. This made finding holiday photos and such far more easier! It does this by reading files that have metadata attached to them; something that every camera will automatically do to photos you take with it.
Recovered_jpg_file_68752.jpg
Get used to seeing the above. Every file recovered will be named in a similar way. Every file will say it was made today. Every file will have no context of where it was or what it was used for. The hard part begins now, but hey, you got your data back at least!
Losing Everything (Almost...)
Windows is a wonderful OS when it works, but at times it really does piss me off. I recently brought a new SSD with the intent to clone over my current Windows drive and carry on as if nothing happened, just now with 1TB of space on C: instead of 256GB. Little did I know, this was going to be the start of a long, horrible process!
Failing to Cloning Windows
My new SDD arrived (along with 2 new HDDs that would replace some older storage drives in my PC that were starting to show signs of failing) arrived. I chose to use Macrium Reflect to clone my current Windows drive. It's free and very easy to use, minus a few basic hard drive limitations that I had to relearn. I had forgotten how much of a ball-ache moving, scaling, and merging partitions was, but it was all do-able. Long story short, this new clone of my drive simply would not boot. Choosing it as the primary drive via bios would indeed launch into it. Hell, even Windows acknowledged that it should be doing something. That something turned out to be infinite loading wheel next to a cursor over a black screen...
Googling on the subject, it appeared I wasn't alone. As many pointed out, while you'd think it should just be as simple as that, it rarely ever is. My line of thought quickly moved over to creating a disk image, and going from there. However this would have resulted in some specifics that I wasn't up for, mostly pertaining to 90% of my applications currently installed on a separate drive, and I wanted them all together back on C: drive.
Fresh Installation of Windows 10
My last resort suddenly became my only resort; reinstall Windows from scratch onto my new 1TB SSD. So I went to Microsoft's page to download the external disk media to allow me to reinstall Windows 10. The media had to be on a min 16GB external USB drive (or CD, not an option here though). The biggest memory stick I had was 8GB, so I decided to use my 2TB external drive. As part of the process of making this disk suitable for Windows to install from it had to format it, completely wiping my file backups. I took note of this, but didn't give it any further thought beyond that.
I Googled around to find what I should be expecting during this installation. I removed my current Windows drive, plugged in my 2TB external drive, and restarted. I manually told bios to boot from my 2TB ext. drive, starting the process to reinstall Windows. The process began. The screen went to black, then came up with Windows saying it was repairing itself. This was one of the ways I saw it happening when Googling around, so I left it. It took an hour or two, in which time I walked away and watched Tiger King on Netflix.
Upon my return, I saw that the loading bar had gone back down to a lower percentage than it was when I left it. I Googled and read that the process often failed first time causing a reboot, thus launching itself back into the same process. I left it, and this time sat with it while it got to the finishing stages to see if it said anything before rebooting. The process had finished repairing and was rebooting to continue into Windows OS.
As the system rebooted, I watched it try to relaunch into my ext.HDD to 'repair Windows' once again. I stopped it just before this stage, and restarted, this time telling bios to launch from my new 1TB SSD, the first drive on the list in my machine. It launched, had a think, and finally started up, all fresh and new!
Staring Back At Me
Once I was into my fresh install of Windows 10, it didn't take long for me to spot the difference. My 3TB main storage drive was called 'System F:', and had a 2nd copy of Windows 10 on it... Somehow Windows had installed on my new SSD, and then installed on my main data drive, overwriting everything on it! You can imagine the sudden butt-clenching experience of thinking, "nah, it's probably gotten mixed up, I'll just restart it and make sure all drives are plugged in correctly, etc". It's worth noting that I actually had some silly number of drives showing up now where Windows splits any drive it's on into 3 partitions.
The computer restarted, and indeed even Macrium Reflect agreed; my main data drive has been wiped to make way for a copy of Windows no one asked for. Not only that, but my main backup of that data on my 2TB ext. HDD had also been wiped to temporarily use it as a system repair drive. Sure, I had some shit floating around on a mix-up between Google Drive and Dropbox, but uploading always took so long I never bothered to properly back everything up that I would be sad to lose; and damn, if I wasn't suddenly realising how sad I really would be!
Moral of the Story is...
Always, always, always backup files and folders that are important to you! Not only should you back them up on another drive, you should have cloud versions backed up too. It's really not that much effort to do, and you never think you'll need it, but all it takes is for you to sit down at your PC one day and realise one of your drives has just stopped working. That's it, done; gone. In today's age things like photo albums are often held digitally; they sure were for me. I work as a 3D artist too, so all of my work is on my PC (I actually had this all backed up online thankfully). Then there's those things you do on spur of the moment that you wouldn't even think about until they are gone. I'm talking about screenshots from games! Essentially photos of memories taken from virtual worlds you've explored, often times with your friends and people from all over the world that you've had contact with now. They seem trivial, but in reality they can be just as important to keep as holiday photos! Take stock of everything you have on your computer and figure out what you would be sad to miss if something were to go wrong. Then make a contingency plan, and execute it now. Don't wait.
Failing to Cloning Windows
My new SDD arrived (along with 2 new HDDs that would replace some older storage drives in my PC that were starting to show signs of failing) arrived. I chose to use Macrium Reflect to clone my current Windows drive. It's free and very easy to use, minus a few basic hard drive limitations that I had to relearn. I had forgotten how much of a ball-ache moving, scaling, and merging partitions was, but it was all do-able. Long story short, this new clone of my drive simply would not boot. Choosing it as the primary drive via bios would indeed launch into it. Hell, even Windows acknowledged that it should be doing something. That something turned out to be infinite loading wheel next to a cursor over a black screen...
Googling on the subject, it appeared I wasn't alone. As many pointed out, while you'd think it should just be as simple as that, it rarely ever is. My line of thought quickly moved over to creating a disk image, and going from there. However this would have resulted in some specifics that I wasn't up for, mostly pertaining to 90% of my applications currently installed on a separate drive, and I wanted them all together back on C: drive.
Fresh Installation of Windows 10
My last resort suddenly became my only resort; reinstall Windows from scratch onto my new 1TB SSD. So I went to Microsoft's page to download the external disk media to allow me to reinstall Windows 10. The media had to be on a min 16GB external USB drive (or CD, not an option here though). The biggest memory stick I had was 8GB, so I decided to use my 2TB external drive. As part of the process of making this disk suitable for Windows to install from it had to format it, completely wiping my file backups. I took note of this, but didn't give it any further thought beyond that.
I Googled around to find what I should be expecting during this installation. I removed my current Windows drive, plugged in my 2TB external drive, and restarted. I manually told bios to boot from my 2TB ext. drive, starting the process to reinstall Windows. The process began. The screen went to black, then came up with Windows saying it was repairing itself. This was one of the ways I saw it happening when Googling around, so I left it. It took an hour or two, in which time I walked away and watched Tiger King on Netflix.
Upon my return, I saw that the loading bar had gone back down to a lower percentage than it was when I left it. I Googled and read that the process often failed first time causing a reboot, thus launching itself back into the same process. I left it, and this time sat with it while it got to the finishing stages to see if it said anything before rebooting. The process had finished repairing and was rebooting to continue into Windows OS.
As the system rebooted, I watched it try to relaunch into my ext.HDD to 'repair Windows' once again. I stopped it just before this stage, and restarted, this time telling bios to launch from my new 1TB SSD, the first drive on the list in my machine. It launched, had a think, and finally started up, all fresh and new!
Staring Back At Me
Once I was into my fresh install of Windows 10, it didn't take long for me to spot the difference. My 3TB main storage drive was called 'System F:', and had a 2nd copy of Windows 10 on it... Somehow Windows had installed on my new SSD, and then installed on my main data drive, overwriting everything on it! You can imagine the sudden butt-clenching experience of thinking, "nah, it's probably gotten mixed up, I'll just restart it and make sure all drives are plugged in correctly, etc". It's worth noting that I actually had some silly number of drives showing up now where Windows splits any drive it's on into 3 partitions.
The computer restarted, and indeed even Macrium Reflect agreed; my main data drive has been wiped to make way for a copy of Windows no one asked for. Not only that, but my main backup of that data on my 2TB ext. HDD had also been wiped to temporarily use it as a system repair drive. Sure, I had some shit floating around on a mix-up between Google Drive and Dropbox, but uploading always took so long I never bothered to properly back everything up that I would be sad to lose; and damn, if I wasn't suddenly realising how sad I really would be!
Moral of the Story is...
Always, always, always backup files and folders that are important to you! Not only should you back them up on another drive, you should have cloud versions backed up too. It's really not that much effort to do, and you never think you'll need it, but all it takes is for you to sit down at your PC one day and realise one of your drives has just stopped working. That's it, done; gone. In today's age things like photo albums are often held digitally; they sure were for me. I work as a 3D artist too, so all of my work is on my PC (I actually had this all backed up online thankfully). Then there's those things you do on spur of the moment that you wouldn't even think about until they are gone. I'm talking about screenshots from games! Essentially photos of memories taken from virtual worlds you've explored, often times with your friends and people from all over the world that you've had contact with now. They seem trivial, but in reality they can be just as important to keep as holiday photos! Take stock of everything you have on your computer and figure out what you would be sad to miss if something were to go wrong. Then make a contingency plan, and execute it now. Don't wait.
Subscribe to:
Posts (Atom)