Seriously, I have wondered this my entire 21+ year career as I watched IT admins blow money on equipment they “needed” at 10x the cost of things that would work equally as well.
I’m obviously generalizing here, and I don’t want to sound like a bigot, but in general… I’ve always regarded the “IT guys” as the “Jocks” of the computer/tech industry. These guys get paid boatloads of cash to administer servers and perform backups… things that you could kinda train monkeys to do these days… while the true “nerds” of this industry are the engineers, the electrical engineers, the software engineers, and the firmware engineers… the socially awkward, perpetually lonely kids who get belittled and abused by their corporate masters. But, maybe my social commentary is too over-the-top…
But lets face is, the IT guys plug wires into servers and switches and sit on their asses watching progress bars all day while they partition hard drives and install OSes… they wait for backups to complete… they wait for RAID arrays to finish syncing. They are overpaid… the gear-heads… they drive BWMs and play golf… few really know jack about computers other than how to play World of Warcraft or write a couple of bash scripts on their shitty Linux distros, yet most laymen will lump an IT guy in the same category as a software engineer or an electrical engineers, while they enjoy the social status of being considered “smart” while the ladies like to run their fingers through their luxuriously long brown hair and beards. But face it, ladies, these guys, in the engineering world are actually at the lowest end of the totem pole in the eyes of true computer geniuses. The real geniuses are the guys who are too busy thinking about algorithms to comb their hair in the morning, pound Big-Gulps for breakfast, lunch, and dinner, and know damn well that no matter how many firewalls the IT-bros have put up, there’s no way in hell that your corporate network will ever be “secure” without their express compliance and consent because your shit is about 20 keystrokes away from being compromised by anyone half as smart as them, any time of the day.
One of the things that always drove me nuts about IT guys, is that they have this zeal for spending money on worthless garbage. They spend $2000 on $50 equipment, and $10,000 on $1500 equipment. As a software guy, I feel like I’m a bit more in touch with the pulse and capabilities of the equipment I use, and I like to get my money’s worth.
So today I was looking for a new backup solution. I wanted to find the cheapest per TB storage solution for backing up our servers at night and wanted to find the current best option. I had heard that there were some new advancements coming in LTO storage tapes and wanted to see if the market had actually moved in on the technology. Supposedly a company had figured out how to cram 185GB on a single inch of tape and hundreds of TB on a single tape cassette. That would be beautiful to have. Unfortunately, the biggest LTO tape storage on the market right now is the LTO-6 standard, storing only 2.5TB on a cassette. Per cassette, the price is supposedly pretty low at around $20 per cartridge, which would be great, as long as the tape drives could be found at a reasonable price… but a jukebox would be preferable.
I looked up LTO-6 tape drives and to my SHOCK, I found that the cheapest, single cassette LTO-6 drive is around $1,700. You can buy hard drives at about $30/TB right now. For the price of the cassette drive alone, I could get 56TB worth of hard-disks, enough to hold 22.4 cassettes worth of data that I would NOT have to swap out by hand. There’s just no mathematical justification for using LTO-6 for anything right now, and my conclusion is that the only people who are buying them have no friggen clue what it is they are buying or are simply buying them to be compatible with some other standard.
There’s a lot of wasted money in this industry. The truly innovative companies out there understand this and are creative enough to come up with better ways of doing things. Google, for example, has been known to fill its data centers, not with name-brand expensive servers, but with cheap, hand-built servers, complete with custom DC power supplies and hacked-together battery backup systems. They literally just attach a 12V battery that you can buy at radio-shack to the back of their power supplies, foregoing expensive name-brand battery backup systems that might cost them millions of dollars in favor of simple, common sense solutions with cheap, expendable motherboards.
I was at a company in the past where I was forced to install my apps on servers that cost $10,000. This would be all great if I really NEEDED a $10,000 server to do the company’s business, but I didn’t. But the sales people sold the executives on the idea that all these servers were “more reliable”. That paying $600 for a tiny SCSI drive vs. $50 for a BIG non-SCSI drive was justifiable because the expensive ones wouldn’t fail…. that the servers weren’t going to fail because they had redundant back-planes, redundant network cards, redundant disks… redundancy.
Another popular argument for overpriced gear was that the overpriced gear somehow “out performed” the cheap gear. But one thing I pointed out, yet failed to convince the executives of, is that the power of your average computer was doubling every 6-months, therefore a cheap computer 6-months from now was going to out-perform this expensive beast they are buying, so unless it was truly important that we were 10% faster than every other computer on the planet for the next 4-6 months, there was no justification for buying a system on the merits of “performance”.
Our server collection grew to the point where I had an entire room full of $10,000 servers at my disposal to do whatever I wanted for alpha and beta testing before we’d deploy to our real data centers around the world. There were maybe 10 racks, floor-to-ceiling with various name-brand servers, with expensive load balancers, and expensive network switches with a dedicated H-Vac system for the room. Despite paying 5-6x what the computers were worth for each unit, those computers weren’t any more reliable than the desktops we built for ourselves from components. Despite having “dual back planes”, the HD controllers failed, the SCSI drives failed, the network cards failed, the systems failed at the same rate if not a higher rate, than any other computer in the office, including the cheap budget ones we’d give to the mail-room or data entry folks.
“High end” equipment belongs to more of a “boutique” industry. They are high-price, low-volume products, which may translate into less time spent on QA, drivers, and compatibility. They sell far fewer high-end systems than they do systems for regular joes and in-fact, the regular joes likely have higher standards of quality as they want their systems that they use for playing Call-of-Duty to be rock-solid and reliable, or they want their money back. If 1 of 100 servers in my data center has a problem, I probably wouldn’t get too emotional about it… I’d just shut it down and let the other systems pick up the slack. It probably wouldn’t even be worth sending back to the manufacturer.
So why the hell are companies routinely spending $10,000 on $2,000 systems? I have begged for an answer to this question for years.
These days, a $10,000 system still does not have any real advantages over a $2,000 system. Go to Dell.com right now and choose an “entry-level” server that starts out at $10,000… and you’ll find that $10,000 gets you the entry-level enterprise system. Out of the box, it comes with just 64GB of RAM, just 2 Xeon CPUs that you pay 10x the price for over Core i7s, just a regular old 1GB ethernet adapter, and a pitiful 300GBs of hard disk space. Certainly you can pimp it out with more RAM (up to 6TB) and more CPUs, but at a HIGH cost. You’ll find that it becomes a $50,000 or even a $100,000 system very quickly. I must admit, however, that I would love to get my hands on a $100,000 system… just to “play around” with. Would it outperform 100-cheap computers though? Probably not. Would it out-perform 1,000 of these new “Micro” computers that fit on a USB stick for $100? Probably not, but maybe, depending on the application and scaling model. You’d have to have a pretty serious database to justify buying one of these things…. and you better hope that it has a stellar warranty.