Why is this important to Joe IT dude? Because the performance of any Compute System is largely based on the slowest item ; The Disks. Even the best disks today are still millisecond response devices. This means that the nanosecond Quad core CPU and all that great cache and memory is waiting at the same speed as the Disk. Yes many optimisations and clever engineering techniques make it look like the system is responding but that is sheer slight of hand. Disk is 1000 times slower than CPU. So at some point the System will prove you a liar to the users. "Don't worry its the fastest we can get for the money" you said shortly thereafter your standing in the CFO's office saying "we need more because those pesky users are "using" the system too hard!
So Why does the machine you over spec'd so carefully gradually grind to a halt? Well the basic answer is until Win 2008 came along the OS providers and Hardware teams had absolutely no reason to resolve the basic problem that the on disk data started at an arbitrary spot on the disks. That spot is called Sector 64. Why ? Who knows and honestly who cares! The Problem like most things in IT is just to configure the system to avoid the inherent bottleneck. You may like to note that not all OS's and hardware designs have this problem. Pretty much just windows.
How did this happen and what can we do ? It is a classic example of Disjointed Technology. It happens when your Disk / RAID / HBA presents a Disk LUN to your WIN OS that starts a nice round 64 Sectors from the start of the disks actual origin. This results in poor performance and 30>40 % more Disk operations, cache fragmentation and even more! More Power More Tape More Trouble for all of us.
Users and Application developers are disbelieving. Surely no one would design nor implement a computing system that has this built-in Flaw ? of course not !
Microsoft know it and the Storage Vendors say it's an OS Problem.
In the posts that follow we'll demystify the process of designing out this issue and look at the way Disk operations are being "designed" in modern SAN and multi-path IO systems.
So what have we done for 20 years ?
We've been pretending that the Symptom is the cause. We tell users to "defragment their drives" as If it's some how their fault that their files are framented. Of course we don't tell them that this shouldn't happen and that the OS is infact prone to it and has attempted to "workaround" usign things like the NTFS MFT
Fragmentation has become a common parlance term and while this is an "end user" issue It hides the real engineering sins of the industry behind a "user" initiated symptom + re-mediation process. 'DEFRAG THE DISK' rings in every user's ears and of course it does make a great difference but in a world of google and electronic fuel injected vehicles DEFRAG'ing a computer disk is an anachronism akin to tuning carburettor's or sorting your mail inbox.
It wasn't until Windows 7 that the real impact of Fragmentation was recognised openly by them as a real every day issue. After all shouldn't the users be upgrading BEFORE this issue arises ? Windows XP's life after death has proven that business model faulty in may ways. However to their credit they have done soemthing !
Steven Sinofsky has posted transparently in the Engineering Windows 7 Blog detailing enough to show they are aware and interested and provides this graphical view of the CPU to disk IO performance gap which shows the problem they face. Disk perforamnce has just not kept up with everything else, exposing the limited design scope of NTFS.The graph shows just how quickly Moores Law applied to CPU is outpacing the Disk Industry.
If you just need the answer NOW, then jump here and please come back with your comments.
The Answer
Disk Best Practices for Win 2008 ( SQL )
