Michael's Guide to Computer Data Storage

From MKRD.info Wiki
Revision as of 14:26, 9 July 2011 by Mkrdwiki (talk | contribs) (Created page with " Michael's Guide to Computer Data Storage By Michael LoneWolf, [http://www.mkrd.info/ www.MKRD.info] rev D 23Feb11 © 2011 Michael LoneWolf, [http://www.mkrd.in...")

(diff) ← Older revision | Approved revision (diff) | Latest revision (diff) | Newer revision → (diff)




Michael's Guide to Computer Data Storage



By Michael LoneWolf, www.MKRD.info


rev D 23Feb11



© 2011 Michael LoneWolf, www.MKRD.info. No distribution allowed.




Which Hard Drive to buy?



There's a bewildering number of choices out there for hard drives, too much for the consumer to know what to choose. However, I will share advice which will not only save you money, but protect your data as well, which is ten times as important.



Let us start. Firstly, cost is irrelevant. Any capacity can be purchased for any cost, low or high. The only time cost enters the equation is if you buy from a retail store. Therefore, do not buy from a retail store. You'll pay too much. I will help you find an affordable hard drive for your budget, so that you will be able to afford two hard drives, often for less than one retail hard drive would cost. Why two? You'll find out shortly! To find a cheap hard drive, shop around. Do not try just one website, try other large reputable online resellers as well. Use Google Shopping, to find out how low you can buy that hard drive for, but by only from reputable sources.



Secondly, capacity is irrelevant. For almost the same price you can buy a 500GB hard drive, or a 2TB one. How much storage should you get? As much as possible. Don't ever limit yourself by amount of storage – it will grow very quickly as time passes. I went from 60GB hard drive to 250GB, to 750GB, to 1TB, to several 2TB hard drives. however, there is one factor that increases with increasing capacity – how much data you can lose at a time. Almost noone realizes that while you can buy a 500GB and a 2TB hard drive for almost the same price, losing 2TB data is very different from losing 500GB of data. In general, buy the biggest hard drive you can find on your budget so that you will never have to worry about adding a second one or moving your data to a larger hard drive in the future.



So what does matter when purchasing a hard drive? Surprisingly, it is factors that few people know about, and factors that are often omitted from online descriptions.



Reliability. Most important factor when buying a hard drive, it is nonetheless hard to be calculated.

  1. Brand can be used somewhat to protect yourself. Take a look at all of these factors that go with a brand:
    1. Past history – has the company had a history of hard drive failures? For example, many people, myself included, were affected by the Seagate "brick" drive symptom. The company did little to explain or remedy the complete and irreversible failure of the hard drive. Therefore, I will never again buy from Seagate.
    2. What is the return policy of the manufacturer? This is why buying online from reputable sources with original warranty intact is a must. Both the seller and the manufacturer must have a good warranty policy.
    3. How new is this hard drive technology?
    4. How new is the company to market? Is it one of the established manufacturers or a no-name company?
    5. Where and how are you buying your hard drive? Is it from a no-name website that you do not know where they get their hard drives from? Is it in original manufacturer's packaging? Has it been refurbished? How well will it be packed for shipping? Badly packaged hard drive will either arrive DOA or will fail some time in the future.
  1. Do not purchase refurbished hard drives, period.

Cache size. This factor is the single biggest one for hard drive performance and reliability. Small cache will significantly slow the transfer rate and decrease reliability and increase power consumption and heating due to hard drive constantly doing random "seek"s. Go with largest cache, currently 64MB.



Transfer rate. If you care at all how fast you can move that huge amount of storage (for secondary storage) or how fast your system is going to be (for primary OS storage), this is another factor you need to find out. Watch out for idiotic rates like "Up to 3000 Mb/sec" or "300 MB/sec" this is the transfer rate of the cable between the hard drive and your computer, irrelevant in affecting the hard drive performance. You need to look for "sustained write" specs or similar. This most likely can only be found from the manufacturer's website, or another (more intelligent) online reseller. This sustained transfer rate will be significantly less than the "300MB/s" rate, and that is how to tell them apart.




Table of recommended hard drives



I have put together the following table of recommended hard drives. In general, most affordable hard drive from each category was chosen (based on capacity). While the exact part numbers may no longer be on sale, or sold at all, a web search by model of the hard drive will bring up equivalent hard drives in each category. Shop around to find the best deal.


Solid State Fast Speed High Reliability Conventional High Capacity

OCZ Agility 2 OCZSSD3-2AGT90G, OCZ OCZSSD3-2VTX90G Vertex 2 WD VelociRaptor WD3000HLFS WD RE4 WD2503ABYX WD Caviar Blue WD1600AAJS WD WD20EARS Caviar Green
Price / Size $180 / 90GB ($2/GB) $150 / 300GB ($0.5/GB) $70 / 250GB ($0.28/GB) $36 / 160GB ($0.23/GB) $90 / 2TB ($0.045/GB)
Transfer rate 250 MB/s 126 MB/s 128 MB/s 126 MB/s 110 MB/s
Cache
16MB 64MB 8MB 64MB
RPM
10,000 RPM 7200 RPM 7200 RPM Intellipower
Reliability Low (wears out, 3 year warranty) Medium (5 year warranty) Medium (testing, 5 year warranty) Low Low
Link Link1, Link2, Link3 Link Link Link Link


The following are some things to consider while looking at the table:

  1. First and foremost, the hyped VelociRaptor hard drive has the same transfer rate as just about any other hard drive. As it is so heavily advertised (and understood) by “performance” seeking people to be much faster than other hard drives (not the least due to its 10k RPM speed), that is simply not true. The only things in which this hard drive is better than conventional drives are: seek time (time for HD to find a random file), its built-in heatsink, and higher reliability. However, simply defragmenting your hard drive (with a third-party utility such as UltimateDefrag if necessary) will erase that difference.
  2. Secondly, for a modest price increase, but a large reliability increase, you can purchase “enterprise class” hard drives from major manufacturers, such as WD RE4 above. This is second best thing (besides bringing the temperature down) that you can do to increase reliability.
  3. SSD drive gives you twice the transfer rate for four times (or more) the cost. It is up to you whether you want to justify the cost of purchasing a SSD drive.
  4. There is little sense in buying anything less than several terabytes' worth of storage for a second hard drive as the $100 prices for such drives are simply ridiculous. So buy second one as a backup. Note: Unless you are using an OS such as Windows 7 or similar, you may be unable to use all of the 3TB or 4TB capacity of a single large drive. In my opinion, you should stay away from most recently introduced hard drives for reliability reasons, and stick to 2TB drives with their well known behaviors.

Remarks:

  • Note: It is highly recommended to use SSD drives only with Windows 7 and comparable OSes that understand specifics of SSD (such as not doing defrag on an SSD drive)
  • The SSD chosen is about the fastest (transfer rate) and cheapest in the industry. I have researched the most affordable SSD storage.

You have just received your hard drive, now what?



Don't plug it in just yet!



If you bought the hard drive from a retail store, mark and save the receipt, and save all packaging.



If you have bought the hard drive online, collect all purchase orders and order confirmations that you have, print them out, and save along with packaging. Inspect packaging for damage. When opening note how well the hard drive was packaged. If it is colder outside than it is inside, allow the hard drive to gradually warm up to house temperature.



If it is an internal hard drive, shut the computer down. If there is a power switch on the power supply itself, turn if off but leave the power cable plugged in (for ESD grounding). Install the hard drive securely. Touch the metal case of the computer often to discharge ESD. Plug in cables securely. Be careful to not unplug other cables, with SATA cables on other hard drives being ridiculously easy to pull out accidentally.



On first power up, you must listen to noises that the drive makes. If there is a single loud click, multiple clicks, or a loud power-up whine, then the hard drive either does not have enough power, or is defect. For USB drives, low power is often the cause. For internal drives, any clicks should make you very suspect.





Do I need additional cooling?



So many sources now sell all kinds of gizmos for your computer, with additional cooling being the most popular. There are heatsinks, fans, and even cooled bays for hard drives. Do you need any of that? You don't – if your intent was not to cram as much power into your computer as possible. Do not install hard drives on top of another. Leave one bay between them empty. If there is airflow from the front of the case over your hard drives, they will do fine.



If you made your mind up on purchasing cooling, avoid cooled bays. These are often crappy at best, with bad thermal transfer between the hard drive and the metal casing, and with an underpowered fan that clogs and whines and breaks and drives you insane. A solid chunk of metal as a radiator over the hard drive is a better cooling method. In general, avoid adding more and more fans to your computer enclosure blowing in random directions.





How many hard drives for my computer?



You will need at least two hard drives for any desktop computer. Why two? Because the Operating System must be separated from your data. Having a single hard drive for both OS and data is bad for many reasons, with some being:

  1. An OS crash and OS reinstall will likely wipe your data
  2. You need to have a faster hard drive for OS, and a reliable one for your data
  3. Most viruses and malware only infect the OS drive
  4. OS is hard to transfer from one hard drive to another, while you may need to upgrade your storage HD over time. Separating the two will not affect one from the other
  5. OS hard drive is accessed about every second for memory page writes. This significantly decreases its reliability
  6. In the future, you may want your storage HD to be hot-swappable or removable, or to be placed in a RAID array
  7. In case of HD crash (it will happen), you don't want a crash to affect both your ability to use your computer, and to regain access to your data
  8. Whenever you will cycle power to the computer (ACPI hard restart by holding down the power button, power loss, computer freeze, etc) there will be data loss. If your computer wasn't constantly writing to the secondary storage HD during power loss, the damage will be constrained to your primary HD.

In general, buy a high-performance, high-transfer-rate, low capacity hard drive for your primary (OS) storage, and biggest capacity, reliable, and power conserving hard drive for your secondary (Data) storage. Keep the primary hard drive 120GB or so, bigger only if needed. Only keep your "My Documents" files in there. Files that you access often, constantly, or that need fast access to should be kept on the primary hard drive. Files that are downloaded, or shared, or being accessed all the time should be on your primary hard drive. Like on a desk, you keep everything you need often at arm's length.



Secondary hard drive is your filing cabinet. You need to get up and walk over to access it. It contains a huge amount of data neatly organized. It is physically secure. It is accessed rarely. But data there is relied upon being there, easy to find, whenever you need access to it. Also, go for "green" hard drives for secondary storage. Not for the "save the earth" BS, but because many of them run quieter, cooler, and slow down or turn off when not accessed for some period of time.




How to significantly increase the reliability of your hard drives



How to drastically increase the reliability of your hard drives:

  1. Use a UPS.
  2. Do not play games on the computer you use to store large amount of data on. Games crash constantly and increase the case temperature significantly.
  3. Control northbridge temperature. Lockups where your computer starts responding completely – not even the mouse moves, but no BSOD is displayed are often due to overheating, especially of the northbridge.
  4. Do not ever shut down your computer by pressing down on your power button for several seconds. If your computer is extremely slow to respond, tell it to shutdown, try to open the task manager, select the option to shutdown from the task manager, and give the computer as long as it needs to very slowly shut down.
  5. Make absolutely sure your data is separate from your OS hard drive
  6. Move your page file to another hard drive. This may even be a third hard drive, or a PCI solid state drive with very low capacity. Your OS does page writes several times a second, along with writing to temporary files locations (several of them). This wears down your hard drive significantly, and ensures that at any event of power loss, the hard drive was always writing somewhere. Errors in the page file and temporary files are most often what results after a sudden power loss.

How to significantly increase system speed



To significantly increase system speed and throughput,

  1. Use two conventional Hard Drives in a striped RAID configuration.
  2. RAM is always the speed limiting factor. Use the fastest RAM your motherboard will support, and use at least 4GB.
  3. Defragment your hard drive(s).

Software for your Hard Drive



Often, when you first plug in your hard drive, it needs to be initialized or formatted. This not a process you click "next, next" on. Carefully research what the process will do and what options it takes.




"SMART" software is most often a load of BS as utilized. In theory, it will monitor your hard drive real—time for temperature, error rate, and impending crash. In reality, most SMART software is crap, and gives you a false sense of security. Anything you trust in providing you with reliability must be researched by you as thoroughly as needed. Real SMART software must always run in background, inform you of unsafe temperatures, inform you of disk errors, detect approaching disk failure, warn on physical hard drive errors (seek failure, power up length failure, page write failure, etc), and warn of RAID errors (if implemented). I will present a report on real SMART software in the future.




Backup is a must, not an option. When you buy a hard drive, buy another one with same capacity or larger. The hard drive should be from a different manufacturing method than the primary one you bought (different manufacturer, or different family from same manufacturer, etc). Have the second hard drive as a backup. Most people are only aware that storage these days is ridiculously cheap. They are not aware that losing a ridiculous amount of data has become ridiculously easy. Cost of the hard drive is irrelevant, cost of having a large amount of data in safe-keeping is what is relevant. And you never, never want to have to pay for data recover, as costs of several thousands of dollars are typical. Now do you see why a hundred bucks for a hard drive is not relevant?



Sadly, most backup strategies are one of the following:

  1. Poorly understood. RAID being an example.
  2. Not fault—proof. If a power loss corrupted data on one hard drive in your computer, a poorly implemented RAID solution of wiring to two hard drives at the same time most likely damaged the backup as well.
  3. Have data kept on a primary desktop which gets hard restarts, BSOD, freezes, and people play games on.
  4. If manual, being time—intensive and being ultimately abandoned.
  5. Only blindly saving whatever it is on the hard drive being backed up, while you will likely need that hard drive you overwrote, send to recycle, bin, or one you need a previous version of.
  6. Over—hyped. With purchased software being most typical. Paying for a third—party software very likely gave you a false notion that backup is easy.

Backup is hard, it requires commitment, it requires research and investment. The only time you can trust some software to do the backup for you is when you are absolutely able to do the job yourself manually, but a piece of software will do the same thing you would do, only with also saving you time and being regular about it.



Additionally, no one single method of backup is sufficient. Not RAID, with it being too technical compared to how hard to use it is, not copy paste, with it not being a valid strategy, and certainly not paid third party software, with it being dump and over—hyped.




Hot plugging. We were all told that SATA supports hot—plugging (or hot removal) – when you don't need to turn off your computer to remove or replace a hard drive. Guess what, more BS. There's one program that does this, called HotSwap!




Locked files (files that cannot be deleted or renamed, or that prevent the parent folder from being modified or deleted) are easily remedied with the very useful Unlocker.




Indexing and “desktop search” (searching within files) is best with Copernic Desktop Search. It is the only search program that I know of that searches within Open Source files like Open Office odt, and others. Micro$oft's Desktop Search does not.




Encryption. There are two kinds of encryption – good and inadequate. Most encryption and data protection methods are inadequate. In analogy, they are a padlock. It is visible and impressive, but only lasts as long as a guy with big bolt cutter comes along. And some protection software is a padlock that you can break with your bare hands (built-in Operating System methods being a good example, and anything from Micro$oft). There is good encryption software like TrueCrypt, however if you forget or lose the password, it is physically impossible to recover the data. In this case, the data is so secure not even you will be able to gain access to it. Again, a backup of the password, volume header (for TrueCrypt), and a backup of contents must also be made.






Do not forget utilities to fix shortcuts, eliminate duplicate files (NoClone), to pack and unpack files (WinRAR, and there's a guide to it coming from me in the future), to save a DVD (DVD Decrypter) to hard drive, and CD/DVD emulation (PowerISO, Alcohol 52%).




Additionally, for certain situations you might want to print out the list of files in a directory. This software is called by many names, “file tree printer”, “directory listing print” being two of many. I have found that simply running the command from a “cmd” window


tree /F /A >filename.txt


or


dir /B /S >filename.txt


is sufficient.




Rather than carrying around your private information, or copying it back and forth, research remote access. If you carry around a laptop, or want access to files stored on your home computer from another location, don't copy them and carry them around. Remote access, once researched and set up, is much more intelligent than multiple copies of same files all with different modification dates.




Similarly, if you do have to access same files from more than one device or location, file synchronization needs to be researched and setup. I can only note that most software is insufficient or dumb for the task.




Do not forget to setup remote access to your bookmarks (favorites). Unfortunately, all of the above three tasks I have attempted to research without finding anything of value. Watch out for reports in the future, to be placed on my website, about topics above.




Dealing with data loss. Loss is inevitable in our lives, be it the previous version of the file, file you cannot locate, a single deleted file or directory, overwritten data, or crash of the whole hard drive. It is how you deal with the loss what's important. STOP doing everything. WRITE DOWN what happened prior to loss, and what you have lost. RESEARCH recovery strategies. RECOVER to another hard drive (not to the same one you are recovering data on). PREVENT same from happening ever again.





Other Storage Devices



USB memory sticks – are ridiculously cheap and are too large in capacity. Why? Because carrying around more than 16GB of data on a memory stick just makes it that longer to copy or transfer that data, that much easier to lose the data, that much more private information that can be lost, and that many duplicate files to synchronize. Buy something cheap from a reputable online seller, that is less than 16GB, and synchronize/offload your data often. Also, mark the memory stick as yours on the outside, with a file inside, and include your email and return for reward information. Do not include your name or phone number if you store private data on the device. You can also encrypt data on a memory stick, or run Portable Apps on it.




External Hard Drives – are a good alternative to adding an internal hard drive, however:

  1. As mentioned before, if they make clicking noises, they either do not have enough power or are about to fail. If it is powered from a power adapter that you plug in the wall, and the hard drive makes either a single loud click or a series of clicks, move your data to another hard drive immediately and return the device.
  2. Do not turn off an external hard drive by turning off the power switch or unplugging it. Always “safely remove” the drive from Operating System (or shutdown your computer first), and then power the device down.
  3. 2.5 inch laptop drives that you have lying around, as well as 3.5 inch desktop drives can be easily converted to an external drive by buying a cheap external hard drive enclosure/case.
  4. For cheap and no—name external devices, cooling and sound insulation is often inadequate. Fans on these units often start to whine or fail outright.

About the Author



Michael LoneWolf is the administrator, designer, and writer for the www.MKRD.info information website, commenting on many different areas, and doing his best to document experience and knowledge he comes across.



Professionally, Michael works in electronics engineering, and website services areas. He specializes in providing services for websites – converting old static html websites to a modern design, rescuing hacked websites, and website administration services.



He can be reached thru the online contact form at http://mkrd.info/contact1.html.