One of the problems I’ve faced over the past few years is maintaining a mail archiving system that had a mind of it’s on, an inch thick administrators manual and an interface and tools that worked sometimes but not others.  I’m in the process of moving 1.8M emails from a SQL based GFI Mailarchiver system to a Jatheon PlugNComply appliance.  It’s no small task to get your emails out of a GFI system by the way.  I ended up breaking the job down in to quarters at first starting back in 2007 when the system was turned on.  It didn’t take long to realize that GFI couldn’t pull more than 100K messages out of the database at a time without choking so I ended up shortening my blocks of time to two months which led to about 36 iterations of a painful search process.

Now that I’ve pulled the data the Jatheon is slowly ingesting it at a rate of about 300K messages a day.  I’m coming up on six days on Friday.  Then I get to convert and ingest some old .pst files before putting the GFI beast to bed.  What a glorious day that will be.

Everyone’s seen the “Three Card Monte” card trick.  It’s where the guy has three cards, he shuffles them around, you inevitably get pushed into picking the wrong card and losing your money.  In the version I’ve been through I didn’t even get to lay my money on the table before the card, Postini in this case, was wisked away.

It all started with Symantec.  We decided to switch anti-virus software due to some problems with viruses walking around Symantec’s Enterprise Protection like it wasn’t there on three workstations this past year.  Each time I ended up using Kaspersky’s rescue disk to clean up the mess.  When we made the move, we had to give up a long standing relationship with Symantec’s suite of products and re-buy what I thought was a spam/email antivirus product (Symantec Mail Security for Microsoft Exchange) at a significant cost per seat.   It so happens that the spam portion of the product is an add-on which costs even more so we ended up scrapping Symantec altogether and trying to find a more cost effective, less nickle and dime pinching solution.

Postini came to mind since I know a few shops that use it and being a GMail user, I’m somewhat familiar with it’s performance.  I started communicating with a Google salesperson which ended up being a chore in itself and found out that because I was only trying to cover 70 seats, their $12 Google Mail Security product wouldn’t reach their minimum annual expenditure limit to even do business with them.  I thought about it for a while, made a proposal to my boss for the next tier and we decided to pay a nickle more, at $25 per seat and go first class.  I emailed the sales guy and he responded that Google had decided to discontinue both the Google Mail Security and Google Mail Continuity products and if I wanted to get the benefit of the spam and anti-virus protection I would have to purchase Google Apps at $50+ per seat.  Wow, all I needed was the service from a $12 account and now all that’s available is a $50 account.  Amazing.

In the end I headed back to the drawing board to find a reasonable solution.  I’m not ready to commit our email to “The Cloud” yet; maybe next year, but I will submit our email to be filtered in the cloud and release some of the burden of our edge server.  I found that Barracuda Networks has a significantly less expensive alternative to filtering viruses and spam in a SaaS type service.  I’ve been familiar with their appliance products in the past so we’re going to give them a try.  Hopefully they won’t decide to pack up their briefcase stand and move on like the other guys.

I’ve been fighting backups for a while now and not very confident in our mixed bag of Symantec Backup Exec server agents and data sets.  Having all the Exchange data is one thing.  Rebuilding and restoring an Exchange server is another.  Same goes for SQL.  I was planning to replace my Symantec agents backup scheme with a virtual friendly system in 2012 but Christmas came early this year and I was able to tackle the project sooner than expected.  I looked at a couple of different products including VDR which is kind of the moped of the VMWare backup world,  Symantec’s Virtual backup solution, Acronis which had a lot of features and did most of what I thought would be required and Veeam which seems to be the industry leader (if their marketing material is any indication VMware thinks so).

I liked the Acronis product because of the virtual appliance setup and considering I went through the hard work of getting everything virtualized I would have liked to continue the trend, but in the end Veeam won out.  I already had the server hardware with some locally attached storage for online backups and it was easier to just build an independent backup server than manage a software iSCSI solution and the Acronis appliance.

I installed Veeam Backup and Recovery v5 and then not two weeks later they released v6 so I upgraded.  To give you an idea of the architecture, I have an older IBM xSeries 3650 with two dual core processors and 19Gb of ram attached to three 2TB external drives.  The 6TB of drive space is managed as three separate drives and backup jobs are assigned to each.  This is my online backups and they stay in the data center.  With the deduplication features in Veeam B&R I can keep more than 20 restore points per virtual machine.

I set up the jobs as incremental with synthetic fulls created on Wednesday and Friday as well as transforming the full backup chains to rollbacks.

What that means is that on Mondays  and Tuesdays a backup job creates an incremental backup (a small file of deltas since the last full).  On Wednesday it does an incremental backup and then rolls the incrementals into a synthetic full.  It doesn’t take a full backup that uses disk but rolls up a full from the previous incrementals.  At the same time it transforms the incrementals into rollbacks.

.vib files are incrementals

.vkb files are full backups

.vrb files are rollbacks

In this scenario I would have the following files at the end of the backup cycle:

Monday – 1 .vib file, 1 .vbk file and multiple .vrb files from the previous backups
Tuesday – 2 .vib files, 1 .vbk file and multiple .vrb files from the previous backups
Wednesday – 1 .vbk file and multiple .vrb files from the previous backups
Thursday – 1 .vib file, 1 .vbk file and multiple .vrb files from the previous backups
Friday – 1 .vbk file and multiple .vrb files from the previous backups

On Saturday there’s a job that copies all of the files in those backup jobs to the Drobo FS which has three 2TB disks in an array that can be removed and replaced for offsite storage.  It usually takes about 24 hours for all of the data to copy from the online storage to the offsite storage and since it’s not taking resources from the production servers it’s easily maintained even if the job time increases.

We store disks offsite for four weeks and then rotate them back into the data center effectively giving us an seven to eight week retention period.  For our purposes that’s enough but this model is easily scalable simply by purchasing more sets of disks.  If my calculations are right, I will probably be able to maintain two full backup sets per offsite effectively doubling our retention period but we haven’t gone through a full cycle yet so time will tell.

Overall I’m really pleased and feel much more confident that we could recover in a reasonable amount of time.  There are still other things that Veeam B&R can do that we haven’t explored yet like virtual labs and replicas.  I look forward to playing with those soon.  I’ll let you know how it goes.

100% virtualized.  It only took a little over two and a half years but about a week ago I shut down my final physical production server.  The last hold out was some profile based home directories that I needed to move to our file server VM.  The new Windows 7 “always available offline” threw a monkey wrench in moving the laptop user’s folders until I could get them reconfigured but now it’s done.

Now, that said, I still have a physical backup server with some locally attached drives for online storage.  We recently acquired Veeam Backup & Replication 5.0 (which will be covered in another post) and determined that best practice would be to use a separate server for it.

Along with that, I finally went through the easy but time consuming process of standardizing all of the naming conventions for the workstations.  I even cleaned up and standardized all of the user names and did a little house keeping in the active directory organization units.

This is the fifth iteration of VMWare that I’ve worked with. Starting back in the 2.0 days with direct attached storage, moving through 3.5, 4.0 and 4.1 with the ESX product and vCenter Server and finally yesterday, moving to ESXi 5.0 and vCenter server 5.

I had hesitated to make the move to the latest product in my production environment while testing the upgrade path in the lab. Moving from ESX to ESXi had been somewhat a taboo subject prior to the recent release and basically meant that you got to rebuild. However with 5.0 there was a clear upgrade path and it was really a non-event except for one little glitch on one of the hosts.

First I updated my vCenter server using the 5.0 source media and followed the next, next, finish steps through the upgrade with no problems. I had taken database backups across the board prior to starting the process. I was quickly able to see my 4.1 hosts just as I left them.

After completing the vCenter upgrade I turned to the first host and migrated all of the virtual machines off to the other two. I popped the ESXi 5.0 disk in and restarted the host. Again, it was pretty much a next, next, finish affair and I was able to reconnect the host in vCenter.

I then followed the procedure with the 2nd host with the same result.

When I got ready to upgrade the 3rd host I ran into a snag however. The virtual machines had a hard time migrating using vmotion and four of them hung between 55% and 66% completed. I waited for over an hour for them to finish but ended up needing to restart the VMWare management services with:

service mgmt-vmware restart

After the service restart I was able to migrate all the VMs off the third host and update it with no problems as well.

After updating all three hosts and reconnecting them into vCenter I removed the HA properties, refreshed and added it back to give the servers a chance to configure HA under ESXi 5.0 and do an election.

The most exciting thing about the process was moving from the vSphere 4.0 Advanced SKU to the 5.0 Enterprise SKU. This means I have the full power of DRS now instead of just vmotion. I set that up too with some rules so both my DC’s don’t live in the same place, two of my SQL servers don’t fight for resources and my Exchange edge server and mailbox server aren’t on the same server. The third rule isn’t really a big deal I don’t think and I hesitated to add it but I figured it wouldn’t hurt either.

DRS did some magic and moved some servers around spreading the CPU and memory love across the three hosts. After a day with living with the new environment, so far, so good!

Windows 8 first impression… clunky. Granted it’s a dev release and has a long way to go but it’s not intuitive for a working environment and if I have to go through and uninstall games from the UI prior to pushing it out to desktops in the office it’s going to be a tough sell for business migrations.

I’ll play around with it a little more but will probably need to wait for a later beta or possibly the RC to make a final call.

We’re in the midst of a file server migration that will hopefully:

• Reduce the amount of old / un-needed data and generally clean things up
• Organize what’s left using a meaningful system
• Create “owners” that can be consulted when something can’t be found
• Give responsibility for keeping things organized to the “owners” of the data

During this transition I decide to use the nifty Group Policy Preferences that showed up in Windows 2008.  They’re powerful and can replace the old login.bat / ifmem.exe scripts that you’ve been using since the NT 4.0 days by allowing you to change settings according to group membership.  The cool thing is these settings aren’t persistent.  For instance, if you wanted to change every computer’s power settings to match a company standard but still give individuals the ability to change the settings to something that fit their personal work habits, you can tell the policy to set the setting once and never change it again.

Here’s an overview from Microsoft themselves: An Overview of Group Policy Preferences

There’s a lot of blog information and technet information on the preferences out there but there’s one thing I couldn’t find an answer for when I tried to map drives using the GPO preferences.

When you are selecting a group to target, under the Common tab and the Targeting button, you MUST use the Browse button to select the group.  When you do this, it shows the SID for the group under the group and will apply the policy correctly.  If you don’t do this, it won’t apply the policy no matter how many repadmin /syncalls and gpupdate /forces you do.

If you don’t see the SID it ain’t going to work!

Symantec just released it’s Backup Exec 2010 R3 version and it’s just in time to me to rebuild my backup environment. We were using an older version of the software on a stand alone box with eSATA attached storage. Unfortunately about a month ago we ran into a snag where the amount of data we backed up in a week wouldn’t fit on the single 2TB drives we used for media.

I’ve been looking at some new devices and decided to buy a Data Robotics Drobo FS.  This little NAS style box has 5 bays, a gig ethernet port and some nifty “any size drives in a protected array” magic.  The plan is to use this device with 2TB commodity drives as the backup media.  When you shut the device down and remove one set of drives you can slide another set in and configure it with the same share parameters and you’re off to the races.  It works pretty good too.

The only thing that you have to remember is to take inventory of the new drives from the media options in Backup Exec and make sure the devices are “online” according to the software.  The share names have to be the same as well.

The device is slower than the eSATA drives that I used to use but not enough to cause problems.  We don’t have a huge amount of data to back up nightly and our windows on the weekend are plenty big to manage.  I’m looking into some ways to speed it up a little though and will be doing some experimenting.

We haven’t been through a retention cycle yet but once we are I’ll update this post to let you know how it does and if there are any other pitfalls along the way.

I was quickly running out of drive space on the C: drive of one of my Windows 2008 Servers, running in a virtual machine, and started looking into options to increase it’s size. It used to be in the old days of 2003 you would have to jump through hoops to make this happen. This included adding the space with vdisk manager and then using a third party app like Partition Magic or a Knoppix disk with GPart to rebuild the drive. It was time consuming and always freaked me out a little to let that tool start messing with the partition tables. It worked, but I could always see something going south quickly.

With 2008 and vSphere all you have to do is edit the VM in the vSphere Client, add the space and click OK (while the machine is running even). Then go into storage manager on the 2008 server and rescan the disk. The unallocated space shows up at the end of the drive you just added space to. From this point you can right click on the C: partition, click extend and add the unallocated space to the drive. Less than five minutes and a lot more confidence inspiring task than the old method.

Data is interesting and cool.  Even data that you might never use is interesting.  Many of the new cars out there have computers SYNCing and OnStaring and communicating.  The services are becoming more and more prolific.  But what about those of us that drive older vehicles?  We don’t have to be left out.

I recently bought a Bluetooth OBD connector like this:

It connects into the standard OBD port on my recently purchased 2003 Civic and transmits OBDii data using bluetooth.  Coupled with an Android device running an application called Torque, I can now read fault codes as well as real time diagnostic data from the car.  Torque also uses GPS data off the Android device as well as accelerometer data to give more data points in the application.  After collecting all those data points, you can download them in CSV formats or upload the information directly to a website to be viewed online.  The online site keeps a record of each trip and you can look at the data in various graphs.

Right now I think the Torque developer is concentrating on the Android side of the application.  The application is geared toward racing performance but it also has conservation and diagnostic uses that are coming to the forefront. The website side of the project leaves something to be desired but I can see how it could quickly become a bear that would be too big to wrestle.  Making the website fit for performance junkies, average joes like myself and hyper – milers may be overwhelming.  I’m hoping I can get the code and roll my own site with my cars information soon.

How great would it be to have a website that I could pull up on my iPad when I go to my mechanic giving him the information about coolant temperature or oxygen sensor data for years, then be able to say, “Something changed here.  We need to look at this.”  It would greatly reduce the cost of diagnoses from my mechanic and could possibly even let me fix it myself.  The device will even let me clear the codes after the repair. No more, “The engine light came on again when I went through a puddle.”

I could see where this would be great data for fleet vehicles.  Not only is it giving coordinate information but maintenance information as well.  Lots of possibilities to track information across many cars in a central location.

Good job and kudos to Torques’ developer.  Thanks.

By the way, my wife calls my obsession with this kind of stuff geeky.  I can live with that.