Recovering Lost Data from a Hard Drive

With Windows thinking it'd be funny to trick me into wiping my data backup to use as a repair disk, and then overwriting my main data drive with a new copy of Windows 10 no one asked it to do, I was in dire need of some recovery software to at least attempt to salvage my data.

Recovery Software

I had never really looked into this before. I was only partially aware that it could even be done in theory, but it turns out there's a fair few bits of software out there for this. Some better than others of course.

My first point of call was to use the free software Recuva. I ran it in quick-mode first time find that it found a bunch of Windows related files (no doubt from Windows being installed on top of my data). I launched a deep scan and continued researching online. I found people saying that Recuva takes far too long and only recovers some data and not all file formats. With it still running and saying it would take around 8 hours to complete, I went to the highest rated paid software: Stellar Data Recovery.

Running this took about half the time and threw up results it could show me on its interface as it was going. Recuva was simply a loading bar, giving no indication that it was indeed finding things. Stellar's interface also has a preview ability to look at image files on the fly. I did notice very few of them would preview, but figured this was because it was still scanning at the time.

Stellar finished it's scan, as did Recuva. Stellar did find a lot more, and I had more trust in it purely because I could see what it was finding along the way. Despite this, exporting found data from both ran into the same issue. No recovered files would open. I looked at Stellar's paid options, and the high-end version included auto-repair tools for images and other document types. It wasn't cheap at around £100 though, and I was sceptical, so I Googled around the issue. Read the comments to this  article by Stellar Data Recovery. Essentially, hundreds of people had the same issue: all the files recovered just would not open. Stellar's response to this was that they were clearly corrupt before you even used data recovery software, a notion that enraged many people in the comments and had many more pointing out what a load of bollocks that was after having paid £100 for software that essentially showed them their old data but wouldn't give it back to them. Needless to say, I decided against joining that pool of people who spent their money to gain nothing back, and called it a lost cause.

Success - miniTools!

I had one more piece of software that I found called miniTools. It had good reviews online also, so before finally accepting that everything was lost and I'd just have to get used to it, I installed this and gave it a whirl. Like Stellar, it could show you what it was recovering as it found them. As soon as it started, it began finding thousands and thousands of files. As it did so, it keeps them in a folder called 'RAW' data. Within that folder are a bunch more folders, each one titled with a file format such as ,jpg, .tif, .png, and so on. I headed into the JPEG folder to find thousands being listed, and each one could be previewed! I could see my photos, my screenshots, everything! The final test was to export a bunch of the recovered files and try to open them. Minitools lets you recover files while it is still scanning, which is cool. All photos opened just fine! For the first time in 2-3 days of testing, I could actually see fully restored files coming up! I was so happy!

The catch? Minitools lets you restore up to 1GB of data for free, then tells you to pay for the software to get more (£80 or so) The difference here is that you can restore files on the fly for free, and they actually worked. This is where the others fell down for me, so I really highly recommend this software! Of course £80 isn't exactly cheap either, but it proved that it would work beforehand, and knew it had me by the balls if I really did want back what it had just found.

The Process of Recovering

There's a few things to note about this process that I've learned along the way. Once miniTools had finished scanning my drive for recoverable data, it found around 5TB worth on a 3TB disk that was never even full... go figure? What happened becomes more apparent as you go on restoring data. There are essentially 2 sets of your data. One set looks just like your data did. It's all in the folder structures correctly, has all the correct metadata present, and so on. Granted, bits of this data were still missing after all this (installing Windows would have definitely overwritten some data for sure). However, if you restore this data, you'll quickly run into that good old issue of nothing opens. Not a single file. I'm not sure exactly how this is, but it makes a bit of sense. This data is all the metadata and heading data of everything on the disk, but none of it is functional. What you need is the RAW data.

In miniTools, after the scan is complete it tells you how many partitions of data it found. For me, it was about 12. I'm not sure what they all are, but most had random crap in them that I didn't recognise as mine at all. The first few had files and folder structures I recognised, but they didn't open. The very last partition is called RAW Files. This is what you need.

Selecting the file formats you care about, you can tell them to export to a selected drive. Note that if you haven't paid for miniTools yet, you can do so now without having to wait hours for it to rescan the drive again. If the file format you were looking for isn't there, you can change which file formats miniTools should try to find in its options. For example, .txt files are off by default. Turning this on however found over million .txt files I wasn't aware existed!

It should be noted that some file formats are not available for recovery. I work as a 3D artist, so I can only cast light on graphic formats that it couldn't find really. For the most part it was very good! It can discover Maya files, Max files, Photoshop files, and Illustrator files to say the least. But it couldn't discover .indd files, InDesign files. I found this a little odd seeing as they are part of the same family as Photoshop and Illustrator, but I assume there must be something more to them that makes them difficult to read in some way?

Another note that might be very helpful for general computer users alike, is that miniTools differentiates between jpeg files, and jpegs files taken specifically from a camera. This made finding holiday photos and such far more easier! It does this by reading files that have metadata attached to them; something that every camera will automatically do to photos you take with it.

Recovered_jpg_file_68752.jpg

Get used to seeing the above. Every file recovered will be named in a similar way. Every file will say it was made today. Every file will have no context of where it was or what it was used for. The hard part begins now, but hey, you got your data back at least!

Losing Everything (Almost...)

Windows is a wonderful OS when it works, but at times it really does piss me off. I recently brought a new SSD with the intent to clone over my current Windows drive and carry on as if nothing happened, just now with 1TB of space on C: instead of 256GB. Little did I know, this was going to be the start of a long, horrible process!

Failing to Cloning Windows

My new SDD arrived (along with 2 new HDDs that would replace some older storage drives in my PC that were starting to show signs of failing) arrived. I chose to use Macrium Reflect to clone my current Windows drive. It's free and very easy to use, minus a few basic hard drive limitations that I had to relearn. I had forgotten how much of a ball-ache moving, scaling, and merging partitions was, but it was all do-able. Long story short, this new clone of my drive simply would not boot. Choosing it as the primary drive via bios would indeed launch into it. Hell, even Windows acknowledged that it should be doing something. That something turned out to be infinite loading wheel next to a cursor over a black screen...

Googling on the subject, it appeared I wasn't alone. As many pointed out, while you'd think it should just be as simple as that, it rarely ever is. My line of thought quickly moved over to creating a disk image, and going from there. However this would have resulted in some specifics that I wasn't up for, mostly pertaining to 90% of my applications currently installed on a separate drive, and I wanted them all together back on C: drive.

Fresh Installation of Windows 10

My last resort suddenly became my only resort; reinstall Windows from scratch onto my new 1TB SSD. So I went to Microsoft's page to download the external disk media to allow me to reinstall Windows 10. The media had to be on a min 16GB external USB drive (or CD, not an option here though). The biggest memory stick I had was 8GB, so I decided to use my 2TB external drive. As part of the process of making this disk suitable for Windows to install from it had to format it, completely wiping my file backups. I took note of this, but didn't give it any further thought beyond that.

I Googled around to find what I should be expecting during this installation. I removed my current Windows drive, plugged in my 2TB external drive, and restarted. I manually told bios to boot from my 2TB ext. drive, starting the process to reinstall Windows. The process began. The screen went to black, then came up with Windows saying it was repairing itself. This was one of the ways I saw it happening when Googling around, so I left it. It took an hour or two, in which time I walked away and watched Tiger King on Netflix.

Upon my return, I saw that the loading bar had gone back down to a lower percentage than it was when I left it. I Googled and read that the process often failed first time causing a reboot, thus launching itself back into the same process. I left it, and this time sat with it while it got to the finishing stages to see if it said anything before rebooting. The process had finished repairing and was rebooting to continue into Windows OS.

As the system rebooted, I watched it try to relaunch into my ext.HDD to 'repair Windows' once again. I stopped it just before this stage, and restarted, this time telling bios to launch from my new 1TB SSD, the first drive on the list in my machine. It launched, had a think, and finally started up, all fresh and new!

Staring Back At Me

Once I was into my fresh install of Windows 10, it didn't take long for me to spot the difference. My 3TB main storage drive was called 'System F:', and had a 2nd copy of Windows 10 on it... Somehow Windows had installed on my new SSD, and then installed on my main data drive, overwriting everything on it! You can imagine the sudden butt-clenching experience of thinking, "nah, it's probably gotten mixed up, I'll just restart it and make sure all drives are plugged in correctly, etc". It's worth noting that I actually had some silly number of drives showing up now where Windows splits any drive it's on into 3 partitions.

The computer restarted, and indeed even Macrium Reflect agreed; my main data drive has been wiped to make way for a copy of Windows no one asked for. Not only that, but my main backup of that data on my 2TB ext. HDD had also been wiped to temporarily use it as a system repair drive. Sure, I had some shit floating around on a mix-up between Google Drive and Dropbox, but uploading always took so long I never bothered to properly back everything up that I would be sad to lose; and damn, if I wasn't suddenly realising how sad I really would be!

Moral of the Story is...

Always, always, always backup files and folders that are important to you! Not only should you back them up on another drive, you should have cloud versions backed up too. It's really not that much effort to do, and you never think you'll need it, but all it takes is for you to sit down at your PC one day and realise one of your drives has just stopped working. That's it, done; gone. In today's age things like photo albums are often held digitally; they sure were for me. I work as a 3D artist too, so all of my work is on my PC (I actually had this all backed up online thankfully). Then there's those things you do on spur of the moment that you wouldn't even think about until they are gone. I'm talking about screenshots from games! Essentially photos of memories taken from virtual worlds you've explored, often times with your friends and people from all over the world that you've had contact with now. They seem trivial, but in reality they can be just as important to keep as holiday photos! Take stock of everything you have on your computer and figure out what you would be sad to miss if something were to go wrong. Then make a contingency plan, and execute it now. Don't wait.

Railroad Shed Project Update

Been a long while since I posted on my blog again!

Lately I have been working on my older 'Railroad Shed' project, adding models to it to make it a clearer environment, and re-texturing the whole thing in Substance Painter. I have tried to keep a very similar cartoon-like style to the textures, while adding a lot more detail in. The now also fully utilise texture maps, not just the diffuse unlit colour as before. Naturally, this changes the style a little, but not much. Here are some newer test renders.

The issues I have now is trying to find a way to create an actual environment out of the work I have, instead of just rendering it on a grey background and calling it a day. Ideally I don't want to be adding much more work to the project, and am eager to move on to the next plan, but it's difficult to come up with a good set of camera angles that can make the scene look like its part of a larger world.

Here is what I have so far in Maya. The floor and steps have been added and will need to be textured if they stay. I may look at adding bushes behind the shed to block out the nothingness that is seen there, but bushes in themselves could be tricky to do well. Any extra work added needs to be easy to complete quickly, and not detract from the main models shown so far.

More Drawings

More 2D artwork. Most of these were done using Photoshop CC 2018's new brushes and tools, taking me away from Corel Painter again. I especially love the brush smoothing setting; it was one of the Painters biggest advantages over Photoshop to me.

Update

久しぶりですね。

Working at AO.com in Manchester has been a great step forward for me, and some of the best days of my life so far. I'm independent from my family now, living with my best friend in Manchester, and doing what I love: 3D graphics!

I've had little free time to truly undertake my usual projects, though its not stopped me drawing as of late. I have picked at my Bell 407 helicopter project in 3Ds Max every now and then, but haven't made any serious progress as of yet. I do however plan to change this in the near future!

Here are some 2D bits I have been working on:

Suga - BTS

Photoshop CC

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Photoshop CC

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Photoshop CC

Not so happy with this piece, so it never really reached completion.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Photoshop CC

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Corel Painter 2017, Photoshop CC

Life Up In The Air

So I've been quiet recently due to big life-changing events taking place. I have a new job as a 3D Artist at AO, which has required me to relocate my life to the big city of Manchester! I'll be house-sharing for a while until my friend is ready to move up here too, and then we'll get an apartment together. Until then, I'm in a strange grey area of my life, hoping everything turns out well!

Substance Designer - Learning The Next Generation of Texture Methodology

I have started to pick up Substance Designer recently, discovering its awesome power of procedural texture generation. The idea is that it can create textures from scratch without any need for external texture files, and it can make pretty complex patterns if you know how. I'm still learning, and have a way to go yet, but I'm building a clear picture of how it works it seems.

This is a tiled floor texture, complete with Albedo, Normal, Roughness, and Metalness maps. The texture is entirely procedural generated, with no external image files being used of any kind.

Here is it's creation graph:

The brilliant advantage of this is the ability to create textures that have infinite iteration possibilities, all at the push of a slider or two; or indeed however many you set up. For example, you could make a pristine metal wall, and use a set of sliders that dynamically age it with weathering and rust as required. From here, you could make an entire texture set that shows an environment ageing over time, without any real extra effort taken to paint the texture up.

I will soon incorporate this method into the Gas Station scene, probably only to the extent of texturing something like the floor, but I'll see how it goes.

Gas Station - Texture Update

During further development of the Gas Station scene I have corrected most of the issues with the normal maps by using cages when baking. Apparently if you use hard edges on the low-poly model, only baking normal maps with cages will produce nice results in xNormal.

The texturing is 99% done in Substance Painter, with some Photoshop involved in areas where it made more sense to use it.

I plan to use Substance Designer in this scene as well, which I will start to properly learn soon.

Gas Station Update

This is just a quick update on the Gas Station project. I may add a roof and some more items, but otherwise, the main part of it is done. I have fully modelled the high-res meshes and the low-res meshed now, and xNormal has created normal maps from each of them. The results can be seen below.

On closer inspection however, you can see that the normal maps are far from finished, and this is the main part of conducting this project; to learn how to correctly bake normal maps on hard-surface models.

You can clearly see in this surface that there is a black line striking through the normal map where it should look smooth like the high-res version.

This error is all over the model in different areas, though not everywhere in what appears to be the same situation, so I am looking into this.

----------------------------------------------

Update:

http://polycount.com/discussion/81154/understanding-averaged-normals-and-ray-projection-who-put-waviness-in-my-normal-map/p1

This covers everything I needed to know. :)

Gas Station Random Project

Recently I have been working on a Gas Station. Didn't really plan to do this, but it's already pretty far on in development. Here are some screen shots:

The model is built in high-res with the aim of its surface data being baked down to a low-res game mesh for the final scene. With the gas pump and some items around it being finished, I have already started to construct the low res geometry, which the last image shows.

Also, just a random update on the helicopter model from my previous post; this model is still being made, and has progressed a bit from where it is shown there, but not by enough for me to bother posting any news on it yet I don't think. I also have an urge to make a kitchen scene for some nice interior design renders, but I probably shouldn't start a 3rd project on the side....