Author Archives: Brandon Willmott

About Brandon Willmott

Brandon Willmott is a Global Sr. Solutions Engineer at VMware, 8x VMware vExpert, founder of the New Orleans VMUG and founder of the Baton Rouge VMUG. Brandon has been in IT for over 15 years, and holds AWS SA-A, VCP and CCNA certifications.

Use Case for Invalidating PernixData Read Cache

When sizing an acceleration resource for PernixData FVP, we look to recommend a resource that can contain the working set of data for the VMs using that resource. The majority of the time this isn’t a problem. However, we recently came across a use case where the working set of the VM was so large that the VM used the whole flash device and began garbage collection after just 3 days. Let’s take a closer look at what’s going on and how we found a feasible solution for the customer.
The workload is a batch processing application that runs once a day and churns through a lot of data during it’s processing time. Before implementing FVP, this was taking approximately 2.25 hours. The first day that it ran after installing FVP, the processing time was reduced a noticeable amount as FVP accelerates every write operation in write back but caches blocks for read acceleration as the VM requests them. By the 2nd or 3rd day, batch processing time was nearly reduced by 50% and customer was thrilled. But the 4th day, processing time was creeping back up to around 2.25 hours. It turned out that even with an 800 GB SSD, the drive filled up and was affected by frequent write amplification and garbage collection to take on newly read blocks and service the writes. So what about stringing some SSDs together in RAID0?
This is a very common question that we get so let’s investigate SSD and RAID a bit.
First, it’s important to understand that SSDs don’t fail like HDD. As SSDs near failure, they might continue to service IO but at an extremely high and invariable latency whereas we typically see consistent performance. In turn, a single SSD that is failing will drag down performance of the RAID array because the array can only performa at the latency and throughput of the slowest SSDs.
As a side note, FVP has adaptive resource management. As soon as FVP determines that going to a given flash device is slower than simply using the backend SAN, it will correctly deactivate said flash device from the FVP cluster. This is technology that is built from ground up to deal with the failure characteristics of SSDs (i..e good ssd vs slow SSD vs failed SSD). RAID was never built to make a distinction between a slow SSD and a failed SSD. Being highly available with an impractical-to-use flash device is pointless — that is actually worse than being not available.
The customer is also using blade servers so a larger capacity PCIe card isn’t an option. So the customer has the largest available capacity in SSD format. Currently the customer needed a solution to consistently achieve the reduction in batch processing time.
What can we do?
PernixData FVP ships with some very robust PowerShell cmdlets that allows us to manage the environment and automate all the things. We first considered using PowerShell to remove the VM from the FVP cluster every few days to force the cache to drop but that would also lose all the VM performance graphs. Instead, our solution was to use PowerShell to blacklist the VM every 3 days. The customer is happy with this solution because even on the first run with batch processing and FVP, run time was still better than straight to the array.
What does this look like?
#Setup some variables
$fvp_server = "fvpservername"
$fvp_username = "domain\username"
#TODO: This isn't very scalable for large environment. I know there's a better way.
$vms = "vm01", "vm02"
#A file we're going to use to store the password in an encrypted format
$passwordfile = "c:\path\fvp_enc.txt"

import-module prnxcli

# Authentication and connection strings removed for brevity. 

#Loop through the list of VMs and set each to be blacklisted, wait 30 seconds then add them back to cluster in WB
foreach ($vm in $vms)
{
     write-host "Blacklisting $vm to invalidate cache"
     Set-PrnxProperty $vm -Name cachePolicy –value 1
     Start-Sleep -seconds 10
     write-host “Adding $vm back to FVP cluster in write back"
     Set-PrnxProperty $vm -Name cachePolicy -value 3
     Set-PrnxAccelerationPolicy -WaitTimeSeconds 30 -name $vm -WB -NumWBPeers 1 -NumWBExternalPeers 0
}

Disconnect-PrnxServer

This scenario isn’t very typical but with large data sets, sometimes you need to get creative!

Clone this repo on GitHub to get the full script or fork and make it fancy! https://github.com/bdwill/prnx-invalidate-cache

Goals for VMworld 2014

VMworld 2014 will be my second time at VMworld and because the conference goes by so fast, it’s important to make a list of goals to achieve and work to stick to it. You don’t want to look back on your time at VMworld and say, “I wish…”

My first VMworld was in 2011 and was an amazing experience. I had been using Twitter and been involved with the community for about 2 years and meeting many people I followed and became friends with was a great experience. I had also been a VMUG leader for 2 years and enjoyed the opportunity to meet and mingle with other VMUG leaders.

VMworld 2012 would have also been another first for me as I was honored as a vExpert 2012 and was really looking forward to participating in the vExpert party. However, a category 3 hurricane was bearing down on New Orleans and I didn’t want to leave my family. I hope to join the current ranks of vExpert for Q3 2014 and participate next year.

Now that I’m at PernixData, my time at the conference will be spent completely differently than my first time. As a vendor, you don’t typically get to attend sessions. A fair amount of my time will be spent at the PernixData booth (#1017) demonstrating FVP. So the role as a vendor is going to be a completely different but still exciting. PernixData has built an incredible product and is already changing how datacenter are designed. Not many startups can say that after selling for 1 year. I’m surrounded by such an incredibly talented team and looking forward to spending time with them this week.

So what do my goals for VMworld 2014 look like?
Evangelize PernixData FVP. My primary reason for being at VMworld is to share PernixData FVP with the world so without a doubt, that’s goal #1. PernixData is an incredible product that I really love so it’s very easy and fun. I’ll be at the PernixData booth during the week so stop by and say hi.
Network with VMUG leaders. VMUG has grown tremendously since 2011 and I’m looking forward to meeting new leaders and continuing to learn how to improve the New Orleans VMUG.
Attend vBrownBag sessions. Without the ability to attend vmworld sessions, I’m looking for some avenue to learn while surrounded by so many smart and talented people. The vBrownBag crew has lined up some great speakers for the week and is sure to be a place I spent a lot of personal time. I’ve also recently co-hosted a couple of weekly sessions so it will be great to meet everyone in person.
Network with the VMware community. PernixPros, PernixPrimes, other vExperts, and everyone else in between: I want to meet you. Everyone has a story to tell so I look forward to growing personal and professional relationships.
Blog. I’ve been wanting to do this for years and never set the time aside to do it. With so many new announcements from VMware and partners, it’s the perfect opportunity to kick things off.
Time will fly so make a list of your goals and stick to them!

Enabling Sublime Text for Command Line Use

I have been using Sublime Text 2 on my PC for a few months and now that I have a Mac, I got around to installing it today. I have been used to modifying scripts and code snippets from the command line by using “subl <filename>”. Unfortunately, that didn’t work after installing Sublime. I’ve read other blog and seen people use it so I figured that it was just an error after installation. Let’s fix it!

The first thing we need to do is create a .bash_profile file if we don’t have one already.

touch ~/.bash_profile

Then we can add this nifty alias to our .bash_profile:

alias subl=”/Applications/Sublime\ Text\ 2.app/Contents/SharedSupport/bin/subl”

Finally, reload that bad boy

source ~/.bash_profile

This was a head scratcher but a quick fix.

Enable Software iSCSI Adapter Using PowerCLI

I recently began studying for the VCAP5-DCA (VDCA550) and in the course of administering a 2 host environment, it’s very rare that I have to use PowerCLI. The time it would take to write the script would easily exceed the time to perform the task by hand. But section 7 of the VCAP5-DCA blueprint states that the exam will cover PowerCLI and vCenter Orchestrator (vCO) so as I go work through the blueprint, I intend on becoming very comfortable with many different tasks in PowerCLI.

From what I’ve read about the VCAP5-DCA (VDCA 550 version), the exam environment now contains 5 hosts. Performing the same task on this many hosts in a limited amount of time is a good opportunity to automate as much as possible. I don’t think creating a powershell script for each minute task is a good use of time but over the course of learning PowerCLI, I will post many PowerCLI scripts that coincide with VCAP5-DCA blueprint objectives.

We’ll begin with enabling the software iSCSI initiator on all hosts in the datacenter. If the environment is already setup with NFS storage, there’s a good chance that the the software iSCSI initiator will need to be added to the hosts to connect to iSCSI storage. The script uses a hardcoded username and password since this is a test environment but I wouldn’t recommend doing that in production. To use in your environment, use your username, password, and name of each host. Make sure the name of your hosts matches what’s in vCenter. If it’s in vCenter with a FQDN, mirror the name in the script. Next, open PowerCLI and run the script with .\iSCSI.ps1.

#variables

$vcenter “name”

$username “username”

$password “password”

$listofhosts = “host1”, “host2”, “host3”


#connect to vcenter

write-host “Connecting to vcenter”

connect-viserver $vcenter -user $username -password $password

#Loop through hosts

foreach ($esxhost in $listofhosts)

{

$currentesxhost = get-vmhost $esxhost

write-host “Enabling software iSCSI initiator on $currentesxhost”

get-vmhoststorage $currentesxhost | set-vmhoststorage -softwareiscsienabled $true

}

write-host “Disconnecting from vcenter”

disconnect-viserver $vcenter

Cisco CCNA 640-802 Results

UPDATE: I passed the exam on June 2, 2009

I recently took the CCNA 640-802 for the second time and I wanted to post my performance for each section.

I first took the test on February 19, 2009. I felt about 80% ready for the test when I took it. I didn’t feel that I had a firm grasp on ACLs, NAT, DHCP, EIGRP, or OSPF. But I felt very strong in switching, and basic router operations.

I felt that I was doing ok till around question 13 — it took my by surprise and probably took me 15 minutes to figure it out. I quickly realized that I wasn’t as prepared. Throughout the rest of the test I had a bad feeling.

I finished the exam and it wasn’t that bad — a 778 out of 1000 (825 required to pass). I was pretty impressed with myself. I was a little down but quickly rescheduled for a month later.

Below is the breakdown:

Describe how a network works: 88%
Configure, verify, and troubleshoot a switch with VLANs and interswitch communications: 80%
Implement an IP addressing scheme and IP services to meet network requirements: 33%
Configure, verify, and troubleshoot basic router operation and routing on Cisco devices: 43%
Explain and select the appropriate administrative tasks required for a WLAN: 100%
Identify security threats to a network and describe general methods to mitigate those threats: 80%
Implement, verify, and troubleshoot NAT and ACLs in a medium-size enterprise branch office: 80%
Implement and verify WAN links: 66%

Score: 778 (825 needed to pass)

So, I knew where I needed to improve. Let’s see how I did on March 19, 2009 — a full month of studying 2-3 hours a day.

Describe how a network works: 85%
Configure, verify, and troubleshoot a switch with VLANs and interswitch communications: 66%
Implement an IP addressing scheme and IP services to meet network requirements: 50%
Configure, verify, and troubleshoot basic router operation and routing on Cisco devices: 46%
Explain and select the appropriate administrative tasks required for a WLAN: 100%
Identify security threats to a network and describe general methods to mitigate those threats: 100%
Implement, verify, and troubleshoot NAT and ACLs in a medium-size enterprise branch office: 75%
Implement and verify WAN links: 100%

Score: 776 (needed 825 to pass)

776?! 2 points lower than the first attempt and feeling fully confident and maintaining positive throughout the whole test and finishing with 20 minutes left? Well after losing some sleep last night I believe I identified some things I got wrong and how to make it right. I believe that if I would have gotten the same score on switching and VLANs, I would have passed. As far as scoring so low on the fundamentals of a router — I can’t explain. There’s not much to know — hostname, configuring an interface, setting passwords, vty access, etc.

I really want to retake 640-802 again instead of taking ICND1/2 — If my assumptions on scoring are close to right, I’m probably missing 1-2 too many.

I’m going to think about it over the next few days and make a decision.

If you’re not following me already on twitter, you can do it now. I’m @bdwill.