I’ve been fighting backups for a while now and not very confident in our mixed bag of Symantec Backup Exec server agents and data sets.  Having all the Exchange data is one thing.  Rebuilding and restoring an Exchange server is another.  Same goes for SQL.  I was planning to replace my Symantec agents backup scheme with a virtual friendly system in 2012 but Christmas came early this year and I was able to tackle the project sooner than expected.  I looked at a couple of different products including VDR which is kind of the moped of the VMWare backup world,  Symantec’s Virtual backup solution, Acronis which had a lot of features and did most of what I thought would be required and Veeam which seems to be the industry leader (if their marketing material is any indication VMware thinks so).

I liked the Acronis product because of the virtual appliance setup and considering I went through the hard work of getting everything virtualized I would have liked to continue the trend, but in the end Veeam won out.  I already had the server hardware with some locally attached storage for online backups and it was easier to just build an independent backup server than manage a software iSCSI solution and the Acronis appliance.

I installed Veeam Backup and Recovery v5 and then not two weeks later they released v6 so I upgraded.  To give you an idea of the architecture, I have an older IBM xSeries 3650 with two dual core processors and 19Gb of ram attached to three 2TB external drives.  The 6TB of drive space is managed as three separate drives and backup jobs are assigned to each.  This is my online backups and they stay in the data center.  With the deduplication features in Veeam B&R I can keep more than 20 restore points per virtual machine.

I set up the jobs as incremental with synthetic fulls created on Wednesday and Friday as well as transforming the full backup chains to rollbacks.

What that means is that on Mondays  and Tuesdays a backup job creates an incremental backup (a small file of deltas since the last full).  On Wednesday it does an incremental backup and then rolls the incrementals into a synthetic full.  It doesn’t take a full backup that uses disk but rolls up a full from the previous incrementals.  At the same time it transforms the incrementals into rollbacks.

.vib files are incrementals

.vkb files are full backups

.vrb files are rollbacks

In this scenario I would have the following files at the end of the backup cycle:

Monday – 1 .vib file, 1 .vbk file and multiple .vrb files from the previous backups
Tuesday – 2 .vib files, 1 .vbk file and multiple .vrb files from the previous backups
Wednesday – 1 .vbk file and multiple .vrb files from the previous backups
Thursday – 1 .vib file, 1 .vbk file and multiple .vrb files from the previous backups
Friday – 1 .vbk file and multiple .vrb files from the previous backups

On Saturday there’s a job that copies all of the files in those backup jobs to the Drobo FS which has three 2TB disks in an array that can be removed and replaced for offsite storage.  It usually takes about 24 hours for all of the data to copy from the online storage to the offsite storage and since it’s not taking resources from the production servers it’s easily maintained even if the job time increases.

We store disks offsite for four weeks and then rotate them back into the data center effectively giving us an seven to eight week retention period.  For our purposes that’s enough but this model is easily scalable simply by purchasing more sets of disks.  If my calculations are right, I will probably be able to maintain two full backup sets per offsite effectively doubling our retention period but we haven’t gone through a full cycle yet so time will tell.

Overall I’m really pleased and feel much more confident that we could recover in a reasonable amount of time.  There are still other things that Veeam B&R can do that we haven’t explored yet like virtual labs and replicas.  I look forward to playing with those soon.  I’ll let you know how it goes.

100% virtualized.  It only took a little over two and a half years but about a week ago I shut down my final physical production server.  The last hold out was some profile based home directories that I needed to move to our file server VM.  The new Windows 7 “always available offline” threw a monkey wrench in moving the laptop user’s folders until I could get them reconfigured but now it’s done.

Now, that said, I still have a physical backup server with some locally attached drives for online storage.  We recently acquired Veeam Backup & Replication 5.0 (which will be covered in another post) and determined that best practice would be to use a separate server for it.

Along with that, I finally went through the easy but time consuming process of standardizing all of the naming conventions for the workstations.  I even cleaned up and standardized all of the user names and did a little house keeping in the active directory organization units.

This is the fifth iteration of VMWare that I’ve worked with. Starting back in the 2.0 days with direct attached storage, moving through 3.5, 4.0 and 4.1 with the ESX product and vCenter Server and finally yesterday, moving to ESXi 5.0 and vCenter server 5.

I had hesitated to make the move to the latest product in my production environment while testing the upgrade path in the lab. Moving from ESX to ESXi had been somewhat a taboo subject prior to the recent release and basically meant that you got to rebuild. However with 5.0 there was a clear upgrade path and it was really a non-event except for one little glitch on one of the hosts.

First I updated my vCenter server using the 5.0 source media and followed the next, next, finish steps through the upgrade with no problems. I had taken database backups across the board prior to starting the process. I was quickly able to see my 4.1 hosts just as I left them.

After completing the vCenter upgrade I turned to the first host and migrated all of the virtual machines off to the other two. I popped the ESXi 5.0 disk in and restarted the host. Again, it was pretty much a next, next, finish affair and I was able to reconnect the host in vCenter.

I then followed the procedure with the 2nd host with the same result.

When I got ready to upgrade the 3rd host I ran into a snag however. The virtual machines had a hard time migrating using vmotion and four of them hung between 55% and 66% completed. I waited for over an hour for them to finish but ended up needing to restart the VMWare management services with:

service mgmt-vmware restart

After the service restart I was able to migrate all the VMs off the third host and update it with no problems as well.

After updating all three hosts and reconnecting them into vCenter I removed the HA properties, refreshed and added it back to give the servers a chance to configure HA under ESXi 5.0 and do an election.

The most exciting thing about the process was moving from the vSphere 4.0 Advanced SKU to the 5.0 Enterprise SKU. This means I have the full power of DRS now instead of just vmotion. I set that up too with some rules so both my DC’s don’t live in the same place, two of my SQL servers don’t fight for resources and my Exchange edge server and mailbox server aren’t on the same server. The third rule isn’t really a big deal I don’t think and I hesitated to add it but I figured it wouldn’t hurt either.

DRS did some magic and moved some servers around spreading the CPU and memory love across the three hosts. After a day with living with the new environment, so far, so good!

Windows 8 first impression… clunky. Granted it’s a dev release and has a long way to go but it’s not intuitive for a working environment and if I have to go through and uninstall games from the UI prior to pushing it out to desktops in the office it’s going to be a tough sell for business migrations.

I’ll play around with it a little more but will probably need to wait for a later beta or possibly the RC to make a final call.

We’re in the midst of a file server migration that will hopefully:

• Reduce the amount of old / un-needed data and generally clean things up
• Organize what’s left using a meaningful system
• Create “owners” that can be consulted when something can’t be found
• Give responsibility for keeping things organized to the “owners” of the data

During this transition I decide to use the nifty Group Policy Preferences that showed up in Windows 2008.  They’re powerful and can replace the old login.bat / ifmem.exe scripts that you’ve been using since the NT 4.0 days by allowing you to change settings according to group membership.  The cool thing is these settings aren’t persistent.  For instance, if you wanted to change every computer’s power settings to match a company standard but still give individuals the ability to change the settings to something that fit their personal work habits, you can tell the policy to set the setting once and never change it again.

Here’s an overview from Microsoft themselves: An Overview of Group Policy Preferences

There’s a lot of blog information and technet information on the preferences out there but there’s one thing I couldn’t find an answer for when I tried to map drives using the GPO preferences.

When you are selecting a group to target, under the Common tab and the Targeting button, you MUST use the Browse button to select the group.  When you do this, it shows the SID for the group under the group and will apply the policy correctly.  If you don’t do this, it won’t apply the policy no matter how many repadmin /syncalls and gpupdate /forces you do.

If you don’t see the SID it ain’t going to work!

Symantec just released it’s Backup Exec 2010 R3 version and it’s just in time to me to rebuild my backup environment. We were using an older version of the software on a stand alone box with eSATA attached storage. Unfortunately about a month ago we ran into a snag where the amount of data we backed up in a week wouldn’t fit on the single 2TB drives we used for media.

I’ve been looking at some new devices and decided to buy a Data Robotics Drobo FS.  This little NAS style box has 5 bays, a gig ethernet port and some nifty “any size drives in a protected array” magic.  The plan is to use this device with 2TB commodity drives as the backup media.  When you shut the device down and remove one set of drives you can slide another set in and configure it with the same share parameters and you’re off to the races.  It works pretty good too.

The only thing that you have to remember is to take inventory of the new drives from the media options in Backup Exec and make sure the devices are “online” according to the software.  The share names have to be the same as well.

The device is slower than the eSATA drives that I used to use but not enough to cause problems.  We don’t have a huge amount of data to back up nightly and our windows on the weekend are plenty big to manage.  I’m looking into some ways to speed it up a little though and will be doing some experimenting.

We haven’t been through a retention cycle yet but once we are I’ll update this post to let you know how it does and if there are any other pitfalls along the way.

I was quickly running out of drive space on the C: drive of one of my Windows 2008 Servers, running in a virtual machine, and started looking into options to increase it’s size. It used to be in the old days of 2003 you would have to jump through hoops to make this happen. This included adding the space with vdisk manager and then using a third party app like Partition Magic or a Knoppix disk with GPart to rebuild the drive. It was time consuming and always freaked me out a little to let that tool start messing with the partition tables. It worked, but I could always see something going south quickly.

With 2008 and vSphere all you have to do is edit the VM in the vSphere Client, add the space and click OK (while the machine is running even). Then go into storage manager on the 2008 server and rescan the disk. The unallocated space shows up at the end of the drive you just added space to. From this point you can right click on the C: partition, click extend and add the unallocated space to the drive. Less than five minutes and a lot more confidence inspiring task than the old method.

Data is interesting and cool.  Even data that you might never use is interesting.  Many of the new cars out there have computers SYNCing and OnStaring and communicating.  The services are becoming more and more prolific.  But what about those of us that drive older vehicles?  We don’t have to be left out.

I recently bought a Bluetooth OBD connector like this:

It connects into the standard OBD port on my recently purchased 2003 Civic and transmits OBDii data using bluetooth.  Coupled with an Android device running an application called Torque, I can now read fault codes as well as real time diagnostic data from the car.  Torque also uses GPS data off the Android device as well as accelerometer data to give more data points in the application.  After collecting all those data points, you can download them in CSV formats or upload the information directly to a website to be viewed online.  The online site keeps a record of each trip and you can look at the data in various graphs.

Right now I think the Torque developer is concentrating on the Android side of the application.  The application is geared toward racing performance but it also has conservation and diagnostic uses that are coming to the forefront. The website side of the project leaves something to be desired but I can see how it could quickly become a bear that would be too big to wrestle.  Making the website fit for performance junkies, average joes like myself and hyper – milers may be overwhelming.  I’m hoping I can get the code and roll my own site with my cars information soon.

How great would it be to have a website that I could pull up on my iPad when I go to my mechanic giving him the information about coolant temperature or oxygen sensor data for years, then be able to say, “Something changed here.  We need to look at this.”  It would greatly reduce the cost of diagnoses from my mechanic and could possibly even let me fix it myself.  The device will even let me clear the codes after the repair. No more, “The engine light came on again when I went through a puddle.”

I could see where this would be great data for fleet vehicles.  Not only is it giving coordinate information but maintenance information as well.  Lots of possibilities to track information across many cars in a central location.

Good job and kudos to Torques’ developer.  Thanks.

By the way, my wife calls my obsession with this kind of stuff geeky.  I can live with that.

I’ve been taking courses for years at a company called New Horizons in Charlotte, NC. The value of the classes can be seen, not only through the course material but also through the interaction and discussions with other people in the class. Well, you can count that out now, and me for that matter.  I won’t be using them anymore.

I discussed the value of having a course with both the instructor and the students in the classroom with my account rep before signing on to take a SQL administration course. I was told, yes, there will be other people in the class with you. The instructor will be there in the classroom. When I got there, it’s an online learning classroom and I’m the only person taking SQL administration in the class. This means not only have to deal with learning the course work but have to figure out a convoluted interface to communicate with the instructor and other students and get saddled with a cheap headset, not to mention, no peer discussions during breaks.  To top it off, there are two guys sitting on either side of me taking a VMWare course and a guy down the way taking a Windows 7 course.  All three courses aren’t coordinating breaks so someone is getting up to go to the bathroom or leaving for lunch or other distracting actions throughout the day.

If I hadn’t had the discussion and directly asked the sales associate about the details of the class I wouldn’t be writing this. But I did. So I had spent the money and planned to be out of the office for a week so I can take a week long webinar.  “This is the way the industry is going,” I was told.  “Even if the instructor is in the building, he’ll be in a studio and the rest of the class will be in the classroom.  We had complaints from the remote students that more time was spent with the classroom students.”  Really?  You think?  That’s the whole point of taking a class.  You communicate with the teacher, learn the curriculum and have discussions with your classmates.

Out of eight or so people in the class I was the only person who asked questions for clarification of points the instructor was talking about the first day.  I was the only one that expanded into real world scenarios that wasn’t covered in the book.  When I asked what other people did in a certain backup scenario, nobody answered.  Why? Because they didn’t have to.  They were safely at Internet’s reach and didn’t have to respond where, in a normal social classroom, a discussion would have taken place.  I’m not there to learn to take a test and get a certification.  I’m not there to just read the book and do the labs.  I’m there to learn how to setup and administrate a product and get as much knowledge about how to do it in a real world environment as possible.  That includes talking to everyone in the class to get the extra knowledge their experience affords.  You can’t do that over a OLL interface.

Maybe I’m too much of an old curmudgeon and set in my ways but I can’t see how this environment is better for learning the material.  It’s definitely better for the instructor, he or she doesn’t have to answer extra questions or interact with students during breaks and after classes.  It’s better for the training business because they can maximize their instructors because they aren’t limited to a geographic location.  But I don’t care about the instructor or the training business.  I care about the instruction and knowledge I get out of the training.

Next time maybe I’ll just watch some online videos and buy a book.

It’s been a pretty good week for a small potatoes working man IT professional like myself. I’ve accomplished retiring a host and installing a new host in my VMware environment. I’ve finalized restructuring a storage array to include about 7TB of new space and provide head room for things like snapshots. To top it off, I was given the chance to talk with one of the knowledgeable elite of the virtualization industry and record a podcast with him for his blog.

It’s somewhat of a coup really. Who wants to hear about our little three host environment anyway? Well, Mike Laverick of RTFM Education obviously thinks there’s something to be heard from the SMBs of the world.  It was a fun and interesting conversation for guys like myself.  If you would like to check it out you can subscribe to Mike’s podcast through iTunes (mp3 or video) or just check it out from his website.

Mike and I met a few years ago at the VMware conference here in Charlotte, NC in 2009 where he was one of the keynotes as well as a panelist in one of the breakout sessions.  It just so happened I sat with him at lunch and we visited a little there.  In 2010 we were able to visit again at the same conference and then I took a trip to the UK on business where we had a chance to meet once more.  Quite a few of the worlds problems were solved that night.  It’s unfortunate that we couldn’t remember the solutions the next morning due to the pints we consumed.

Mike is also a musician and had an interest in my instruments.  Here’s a picture of the two of us with his Blues Bowls.