Taking FlashArray Snapshots with Veeam

In April 2018, Veeam released the Universal Storage API which enabled storage vendors like Pure Storage to create integrations for Veeam with their storage system. At a high level, this functionality allows Veeam to leverage storage system snapshots when performing backups as well as take snapshots of volumes for instant restore of VMs or granular file restoration.

In the initial release of the Pure Storage FlashArray plugin, the ability for Veeam to see and utilize existing snapshots on the FlashArray is unavailable. Additionally, it’s not currently possible for Veeam to take snapshots of all the volumes associated with a Protection Group. Joint customers have expressed the desire for this functionality but development takes time.

In the mean time, I created a script that gives customers the ability for Veeam to create snapshots of all the volumes in a FlashArray Protection Group. This script is designed to be run automatically using Windows Task Scheduler; however, you can run it from a PowerShell command prompt for a quick, one time use.

The most significant use case I created this for was recovering file shares faster if it was encrypted by a malware attack. It’s totally possible to immediately remediate the most extreme case where the whole file share is encrypted by overwriting the volume from a storage snapshot but what if it’s just a user’s home directory or a small subset of the file share?

In the following example, I have snapshots on the FlashArray that were taken by Veeam:


From Veeam’s view:


When selecting a snapshot, you can see each VM protected by that snapshot:

This integration is extremely powerful as it provides instant VM, guest file, and application item recovery from FlashArray snapshots instead of backup.

In a sample test, I recovered a single Windows Server 2016 VM in just over a minute:


Veeam performs this operation similar to how it operates when restoring from a backup, with the exception that it creates a volume on the FlashArray from the snapshot,  presents it to the applicable host, rescans the hosts’s HBA, mounts the volume, and adds the VM to vCenter.


Known Limitations

Currently the first version of this script only supports volume-based Protection Groups. If your Protection Group’s members are hosts or host groups, the script will not work. I anticipate fixing this in an upcoming release as well as adding the ability to specify a volume instead of a Protection Group. Additionally, this script doesn’t limit the number of snapshots taken so please monitor your usage. A future version will address this issue as well.

If you have questions about installing and configuring the Pure Storage FlashArray plugin for Veeam, check out Stephen Owens’ blog posts:


To download the script, head over to the script’s repository on my GitHub page.

Taking My Talents to the Docker Community

After reflecting on my personal goals and the recent announcement that VMUG was joining the Dell Technologies User Community, I’ve decided to step down from the leadership role of the New Orleans VMUG effective immediately and focus my attention on building the Docker community in Louisiana. This hasn’t been a rash decision because of the “acquisition” of VMUG by DTUC but rather an affirming indicator that now is the right time to move on.

I’ve been part of VMUG leadership since 2010 when I started the Baton Rouge VMUG (and sequentially the New Orleans VMUG in 2011). My motivation to start the groups stemmed from my lack of knowledge about virtualization, the desire to learn what others are doing, and what new technologies were coming to the market. I didn’t want to just be a consumer though, I wanted to create an environment where I could learn and then help others who had the questions of their own and start a feedback loop. After 6 years, I can look back and say that I was successful in achieving those goals and now I’m looking for the next challenge.

Why Docker and why now?

I’ve been following Docker since 2014 and have watched the project mature and continue to gain in popularity. In 3 short years it has become the de facto standard for building cloud native applications and a robust ecosystem is forming around it much like with VMware a decade ago. The technology is still in it’s infancy and the population of IT professionals don’t know about it but the number of early adopters continues to rise and drive further development and innovation at a rapid pace. Now is the perfect time to get in at the ground floor and begin building a community.

As an open source project, community is at the heart of what has fueled Docker’s growth. Forty one percent of Docker’s contributors are individuals! That speaks volumes for the commitment that thousands of people around the world have made to improving a project. At it’s core, Docker, Inc. embraces community and local groups have exploded in popularity. As of May 2017, there are 288 groups around the world. I encourage you to take a look at the list of Docker meetup groups and get involved. There’s even an online meetup if you can’t attend one locally.

This is also an exciting opportunity for me to grow as an IT professional and network with a different group of professionals. I’ve been an amateur developer since 8th grade but knew it wasn’t a career path for me but as infrastructure has become more “cloud”-like, my interests have shifted to trying to learn more about applications and how they’re deployed. What platforms are used? What barriers exist to reliably getting stable code into production? What day-to-day challenges do developers face? I look forward to the opportunity to personally grow and being a part of the Docker community will allow that as it is currently geared more towards developers. However, the technology is becoming mainstream and the necessity to enable ops teams to successfully manage containers will be more important. It will undoubtedly lead to an even larger ecosystem and participation by former virtualization engineers as their organizations restructure their environments.

What’s your take on VMUG becoming part of the Dell Technology User Community (DTUC)?

First and foremost, my decision has nothing to do with disliking Dell EMC nor do I have a lot of commentary on what this means for the future of VMUG. My dissatisfaction lies with how the VMUG CEO, President, and Board handled this very important decision that aligns a previously independent community with a corporate run community. I strongly disagree with this path for VMUG and I don’t feel that Dell EMC brings value to a community that has been successfully grown and managed through grassroots efforts. VMUG HQ has greatly veered away from their original objective of supporting leaders to build a successful community for VMware customers. I know my fellow leaders will defend their ability to bring in sponsors and topics that are in the best interest of their local community regardless of the alignment with Dell EMC. I’m uncertain what the future holds for VMUG but I know that communities will stick together no matter what obstacles are presented.

I’m Going to OSCON!

I’m very excited that I’m attending my first OSCON this year, compliments of The Cloudcast. On the April 25, 2016 episode, it was announced that they were giving away two Bronze Level passes to the conference. To win a pass, they asked to send in a personal story about community and why you wanted to attend OSCON. I wanted to share my story with a wider audience:

I’ve had an interest in Linux and coding since 6th grade and did some intermediate level coding in high school, college, and in my career. Finding help online to complete a coding project or successfully getting something like XWindows running in the late 90s was quite difficult. Similarly today, learning new, uncharted, and complex systems such as Kubernetes and Mesos is challenging but the community around it is fanatical and extremely helpful. I find that the community surrounding a technology is a key to it’s success and the success of the business. I’ve found myself gravitating to upcoming technologies that have deeply rooted and committed communities because those are the ones that usually become a mainstay in the industry.

One of my first experiences with such a community was the VMware community via Twitter in 2010. I started down the path of virtualization at my employer and my head was spinning. I happened to find that there was a vast community of virtualization evangelists that shared best practices and experiences and it was awesome to connect with hundreds of people who were there to help others. From there, I learned about local VMware user groups and wanted to participate in one to hear what others were doing and try to learn more. Living in Louisiana, which isn’t a major city/tech hub, there wasn’t a local group. I found tremendous value in the virtual community and I wanted the same experience of learning and sharing in person as well. I reached out to VMware and told them I wanted to start a local chapter of the VMUG. I started the first VMUG in Louisiana in Baton Rouge and a year later, due to it’s success, VMware asked me to form and run a second user group in New Orleans. I’ve been running the New Orleans group for the last 4 years and continue to enjoy learning about how organization’s environments are maturing with the maturation of the ecosystem.

As virtualization has solidified itself as a basic function of the datacenter, I’ve found myself following the maturation of the datacenter and it’s all pointing towards open source software. This has been very refreshing for me personally because it feels that there has been a reset in the datacenter as it begins to reinvent itself. Once virtualization became an anchor of the datacenter, VMware and other ecosystem partners moved up the stack to build upon that foundation. This has brought more end-user related products to the market but has left the infrastructure architect without many new tools with which to innovate. Containers will be a pillar of the next generation datacenter and it’s all fueled by open source software. I’m excited to see how the shift away from enterprise software vendors, which have been to  have rigid and archaic ways of designing software, changes over the next few years. The flexibility of OSS will give organizations a way to consume software through the community which is free and empowered to define how the software or application is best built. The community around these technologies will have a strong influence to guide it in the best direction but the biggest impact of the community is to be engaging with each other to guide newcomers and strengthen the established members.

I look forward to becoming part of this community and meeting new people next week.

9 Months at PernixData: Recapping My Best Career Decision

It’s hard to believe that my one year anniversary at PernixData is in 3 months. I have to say that coming here has been the best career decision I’ve ever made. I’m proud to be a part of the fastest growing software infrastructure company in history and surrounded by highly talented and passionate people. Having come from the customer side, specifically a medical practice, it’s interesting to be part of a software company and be a part of a startup.

PernixData is now a 3 year old company and I previously didn’t think I would ever join a startup. When I considered other opportunities at other young companies in the past, my family always cautioned me on having stability. I stayed at a 30 year old company for 10 years in the name of comfort and stability and I have some regrets about it. Louisiana isn’t busting at the seams with jobs in enterprise IT so the associated risk of joining a startup and losing my job one day weighed heavily on me. The security of an established organization was very comforting but I recognized that my growth was extremely limited due to the size of the organization and their plans for growth. Building the proverbial IT mansion was fun because I left the organization with a solid infrastructure but after the projects were over, the upkeep was minimal and the days became mundane. The decision to move on came down to fulfilling a desire to grow professionally by facing a new challenge outside of out of my comfort zone.

When going to a startup there can be a lot of risk and a lot of reward. Aside from believing in the architecture, what gave me comfort in joining a young company is it’s leaders. Satyam Vaghani and Poojan Kumar aren’t household names but they were both seasoned VMware alum that had a vision and brought along a team of world class developers that could execute that vision.

There’s risk with everything in life, but I opted to set aside my overly cautious feelings about job security and jumped right in because I only see PernixData continuing to grow.

The transition from Director of IT to Systems Engineer was exactly the change I was looking for because I was burnt out with operations. The career change also gave me the opportunity to explore existing, new, and upcoming technologies and understand how they relate to FVP. In my previous role, my experience with various hardware and software configurations were limited because there I felt learning about them wasn’t beneficial because they weren’t relevant to my job nor would we ever need to implement a solution like it. (Note: I recognize this was a terrible mindset to have and have since changed.)

As the Director of IT of a small company, I was responsible for maintaining the infrastructure and I was the manager of a desktop tech. The desktop guy was great at his job but I didn’t have a peer to collaborate with. That’s why starting a VMUG in Louisiana was important to me — I wanted a community of peers to share and learn with about virtualization.

The PernixData SE team is a great example of the peer community that I wanted to be part of. I’m surrounded by around 20 other engineers that have come from various backgrounds such as a fellow IT Director, virtualization admins, VMware instructor, and experienced SEs. Each of us has worked in different verticals and with different applications, hardware, and end users. This diversity allows each of us to bring our unique experiences and contribute them to the team and further develop a highly skilled and technical team. I’m also very proud that over 90% of our team are VMware vExperts.

What has been the most fun about working at PernixData is meeting people across the country and challenging them to re-think how they purchase storage and drive application performance. During the POC process, I love talking about technology with customers, learning about each company’s environments and challenges, and ultimately letting FVP speak for itself. In my opinion, being able to stand behind the product that you sell is ultimately what removes some of the challenges of being in sales and makes it enjoyable.

If you work for PernixData at HQ, there’s a lot of perks but as a remote employee I don’t benefit from but working at home is a great supplement. I don’t know how I could ever go back to working in an office, that’s for sure! Personally, the transition from office worker to teleworker hasn’t been difficult because I talk to quite a few customers everyday and always staying in touch with team members.

On a personal note, my wife is almost always home because she works at night as a registered nurse. For some, working from home while their spouse is there has presented challenges. This hasn’t been the case for us but YMMV! Another perk that I enjoy about working from home is being to take my kids to school occasionally and always being here when they get home from school. Once they’re home, they love to come into my office and keep me company for the rest of the day. It’s not always unicorns and rainbows though, I have to kick them out quite a bit!

Overall, I’m very pleased with how the last 9 months have turned out and always looking forward to the next day.

Shiny new vStuff — Speak at a VMUG NOW!

Having spent the last 3.5 years as a VMUG leader of two different VMUGs and spent time talking to over a dozen other leaders, one issue persists in the VMUG community: lack of customer participation. VMUG recognized this and implemented the Feed4ward program to, “encourage every interested member to share their knowledge at a VMUG local group meeting or User Conference”. Knowledge sharing is what everyone’s there for but most of the time people are nervous about public speaking, don’t think they know enough to discuss topics with others, or they think what they do isn’t that different or interesting. That can all changes now! 

With the release of vSphere 6 on March 12, everything is new to everyone. Not many people have downloaded it in their test/dev/lab environment and (hopefully) no one has deployed it in production yet! There are 11 vSphere ecosystem products that got updated and probably thousands of new features or enhancements to discuss. If you think just an “upgrading to ESXi 6” presentation will be boring, look at upgrading or starting to use one of the other supporting vSphere products such as vRealize Automation or Operations Manager. Maybe you’re a SMB and using or looking to use vSphere Data Protection or vSphere Replication. What was the upgrade or setup process like? How do you manage it? Did you ever have to recover from a backup or replica? Any gotchas? There’s plenty of opportunity now to get started giving back to your local VMUG community. If you want mentoring, look into the VMUG Feed4ward program!

The vSphere 6 documentation ca be found at: https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html.

Take this time to get out in front and start getting familiar with the new features and the associated documentation. Many organizations will look to upgrade once update 1 rolls around (I was in this crowd) which will probably be released in 6 months. Take the lead, become the expert, and be a staple in your local community.

The local VMUG leaders will probably already have a “What’s new in vSphere 6” slot carved out at the next meeting but if there’s a product feature or enhancement you like, love, or have always wanted to see, speak with them about adding a deep dive into that topic. It’s highly unlikely they’ll say no!

Better Documentation with Infrastructure as Code

Creating documentation isn’t fun. I’ve done my fair share in 10 years of administering systems. I’ve written documentation on AD, Exchange, router and switch configurations, VoIP system configuration and operations, and so on. As a one man shop that architected all the systems I ran, I was unsure what level of detail was required. What helped the most was having an outside resource that could review the documentation and try to fix a problem given the information I documented. Whatever question he had, that also had to be added.

What I’m not used to is taking over an infrastructure or application and being tasked to administer it. Even with decent documentation from the previous admin, you really don’t know the environment until you’ve had to fix a problem.

Recently, I tasked myself with taking over and update an internal application when the previous owner left. Because it was a small (but useful) tool, documentation was non-existent. To make the necessary updates to it, I had to spend plenty of time understanding how the application was structured. Once that I was done, I was ready to add my code and begin testing it.

Here’s the problem I was faced with: I don’t want to disrupt the application in production so I need a test environment but I don’t know everything I need to install to match the production application. Sure I could clone the VM, change the hostname and IP address, etc and hack at it that way. But there’s a better tool to tackle this with that will allow me to document the application and build the environment in a repeatable way. Enter configuration management and the concept of infrastructure as code.

Tools such as Puppet, Chef, and Ansible enable this ability. By implementing the concept of infrastructure as code, admins have the ability to provide useful documentation for systems and applications in the environment and also establish a mechanism to stand up additional application components or even provision new hardware. I chose to learn Ansible because I like the fact that the syntax is very simple (YAML), it uses SSH to communicate with the host, and is agentless.

From my investigation into the app, I know I need Ubuntu Linux, PHP, Apache, Postgresql, and Python. With a little command line-fu, I can find out which versions of the software I need and ensure that my configuration specifics those versions to be installed.

By implementing the concept of infrastructure as code, relevant and detailed documentation is provided for you, your team, and those that come after you.


Updating the firmware on a Micron p420m

Update: Micron released B218 in July 2015 to resolve a critical issue with command time outs. Micron highly recommends upgrading to this firmware release.

A common PCIe flash card for PernixData customers to use is the Micron P420m in their environment. It’s a very high performing and cost effective PCIe card and has a variety of applications.

Like all hardware devices, the p420m has firmware that occasionally needs to be updated. To perform the firmware update, we’re going to download the Micron rssdm utility (packaged with the ESXi drivers) on Micron’s site in the Support Pack for Linux and VMware package. As of January 2015, the support pack B145.03 from September 2014 is still current.

The first step to determining which firmware version the card is running is to install rssdm. Put the host into maintenance mode, copy the vib for your version of ESXi to the host, and run the esxcli software vib install -v command and reboot the host.

Once the host is back up, log in and execute the /opt/micron/bin/rssdm -L command to see the firmware version of the card.


As you can see, my card is running firmware version B2100600 and needs to be updated. At the time this article is posted, current firmware version is B2120500. We’re going to copy the new firmware to the host or shared datastore and perform the upgrade.

With the host in maintenance mode and the device removed from the FVP Cluster, copy the B145.03.00.ubi firmware image downloaded from the Micron Support Pack above to a location accessible by the host. The B145.03.00.ubi file will be in the Unified Image folder.

Then execute /opt/micron/bin/rssdm -T /path/to/file/B145.03.00.ubi -n 0


Once it’s complete, reboot the host.

When the host is back up, verify the new firmware is active.


Secure erase the drive by executing: /opt/micron/bin/rssdm -X -n 0 -p ffff

Screen Shot 2015-01-19 at 10.40.10 PM

Note: For those interested, -X is to perform the secure erase, -n is to specify the drive ID, and -p is for password (default is ffff)

After the secure erase is complete, remove the host out of maintenance mode and add the device back to the FVP Cluster.

Intro to Git and the Rise of Social Coding Presentation

Over the Christmas season, I participated in social coding community event called Commitmas. The objective was to learn git and try to use it every day for 12 days. As I wrote in another blog post, Rockin Around the Commitmas Tree, this was a great opportunity to jump start learning to use it with others in the community.

Since then I’ve continued using git on a regular basis for various side projects. I wanted the opportunity to share with the PernixData SE team a little bit about what I’ve learned and why it’s important for us to learn and use git. I feel that learning git is a crucial foundational skill for building and managing next generation datacenters and developing applications. I also like git for daily use to manage versions of documents or presentations.

Putting together this presentation was a bit special for me for two reasons. First, it’s the first technical presentation that I created that wasn’t related to my core competency, VMware vSphere. This stands out to me because it’s representative of my personal journey to be an early adopter of these new technologies. The configuration management/container/PaaS space is still in a very early phase and it reminds me of stories I’ve heard about having to perform vMotion on the command line. That’s always sound so old school to me because I started using vSphere in 2009 when 4.0 was released and it was all well established. I often wonder what this space will look in a few years and will it take less time to mature.

It’s also the first time that I’ve used reveal.js to create a presentation. I’ve had the chance to see a few presentations with done with reveal.js and really like the elegance and simplicity of it. As a technologist, I like the geekiness of creating my presentation with HTML.

Git: Syncing a Fork In It

On the second day of Commitmas, I had a new hurdle to jump — I forked @mjbrender’s repo, submitted a PR, and he accepted my changes as well as changes from @joshcoen and now my fork is outdated. How do I update my fork and local repo with the changes?

To rectify this, we’re going to leverage two git commands: git remote add upstream and git fetch upstream command.

With git remote add upstream, we’re going to specify the repository that we forked. In my case, I did:

git remote add upstream https://github.com/mjbrender/12-days-of-commitmas

A git remote -v verifies that we added the upstream source:

Screen Shot 2014-12-22 at 10.12.15 PM

Then we perform git fetch upstream to copy the original repository (the one we forked) to our local repository (on our dev box):

Screen Shot 2014-12-22 at 10.12.23 PM

Now we just fetched the upstream/master branch/repo and needs to be merged into our master branch. We can switch to our master branch with git checkout master.

Finally, git merge upstream/master to merge upstream/master to our master branch.

Screen Shot 2014-12-22 at 10.12.55 PM

The final stretch! Our local repository is now a clone of the original repository again but we need to push our changes back to our repository on GItHub.

Screen Shot 2014-12-22 at 10.28.16 PM

Done! Join in the Commitmas conversation on Twitter using the #vBrownBag hashtag! Happy committing!

Rockin’ Around the Commitmas Tree


My good friend Matthew Brender started a fun project for the holiday season called 12 Days of Commitmas to encourage everyone to continue to grow over the holiday season by learning or expanding their knowledge with git and coding. These skills will become increasingly prevalent and necessary in 2015 as more sysadmins and organizations implement version control to manage code for configuration management tools like Ansible.

When looking at the skill levels that Matt created, I found myself between beginner and intermediate. In the spirit of code contribution and learning git, and playing with Markdown, I submitted my first pull request on GitHub to expand the skill levels.

My Goal for Commitmas

Because my coding skills are relatively weak (improving them is a 2015 goal), I will contribute documentation to a project on GitHub. Internally on the SE team at PernixData, there are some PowerShell scripts that I will create repositories for in GitLab. I haven’t picked a project on GitHub yet to contribute to but I’m considering something related to Ansible. This will give me a chance to explore the different Ansible projects on GitHub and learn/do more with Markdown. I’m also going to push all of my commits using the CLI as I currently lean heavily on the GUI. I know there’s nothing wrong with using the GUI but I believe that true mastery comes from using the CLI when applicable.

Join in the Commitmas conversation on Twitter using the #vBrownBag hashtag! Happy committing!