Friday, 29 November 2013

Missing VM NIC

The Disappearing Act

A VM went off the network, and actually lost the NIC from within the VM's hardware.

Pouring through logs, (some thanks to LogInsight, more on that later), I discovered in vmware-xx.log:
2013-11-19T07:33:01.246Z| vcpu-0| Powering off Ethernet0
2013-11-19T07:33:01.246Z| vcpu-0| Hot removal done.
ah ha! This shows Ethernet0 was removed via the "Safely Remove Hardware" icon in the Windows system tray.

The solution is to add a new NIC of the same type.

This is something that happens quite a bit as you can read here:

A good idea to prevent this would be to disable HotPlug as mentioned in this KB artile:

LogInsight was able to show some hints, but didn't quite nail it. LogInsight showed:
2013-11-19T07:33:01.841Z esxhost.local Vpxa: [FFE6FB90 verbose 'Default' opID=WFU-e9559ea6] [VpxaHalVmHostagent] 2746: Config changed 'config.extraConfig["ethernet0.pciSlotNumber"].value'

Unfortunately, LogInsight doesn't index the virtual machine logs (vmware.log). If it did, it would have saved me a lot of time.

William Lam has done a great post on how to get your vmware.log into the ESXi syslog,

Search and Destroy

From here I needed to find out who the culprit was. Unfortunately there's no central syslog for our Windows servers, so I had to dig through the event logs. The Security logs are pretty noisey, but I was able to see some users were logged around the time of the incident. No one admitted to accidentally ejecting the hardware, and it wouldn't have mattered if we did identify the individual. Everyone was now aware of it, and knew to be more careful.

Friday, 15 November 2013

Updating powershell help

When installing Powershell v3, it doesn't install any local help files for get-help. If you are upgrading from v2 to v3, the help isn't updated either.

If you are using a proxy, a simple update-help may not work for you.

Do the following:

$webclient = New-Object System.Net.WebClient
$creds = Get-Credential
$webclient.Proxy.Credentials = $creds

This will connect to the internet and download the updated help files.

Tuesday, 12 November 2013

Constant Alarm 'Network Uplink Redundancy Lost'

It's amazing how much is going on when you dig through logs. On this occasion I was looking at  "tasks & events" of a host and noticed a lot of network errors.

Alarm 'Network uplink redundancy lost' on <servername> triggered an action

The error was occurring every 5 minutes. This was made visual with the use of Log Insight. My new favourite tool.

Alarm 'Network uplink redundancy lost': an SNMP trap for entity <servername> was sent
appname  source  hostname  vc_event_type  vc_alarm_type 
Alarm 'Network uplink redundancy lost' on <servername> triggered an action
appname  source  hostname  vc_event_type  vc_alarm_type

I couldn't find anything wrong with this particular ESXi host, vSwitch or uplink. It had the same configuration as all the other hosts in the cluster.

The fix was to go to the top level where the alarm is defined, Edit Settings, disable the alarm, then go back and re-enable it.

After that, the errors stopped appearing.

Reconfigured alarm 'Network uplink redundancy lost' on Datacenters
appname vc_username source hostname vc_event_type vc_details 
Reconfigured alarm 'Network uplink redundancy lost' on Datacenters
appname vc_username source hostname vc_event_type vc_details

Monday, 28 October 2013

VCA Exams Worthless?

VMware released the VMware Certified Associate (VCA) exams at VMworld San Francisco 2013. These are entry level exams, below VCP, aimed at people just getting into virtualisation or not quite ready to sit the VCP exams.

Bloggers are saying great things about the VCA exams. I spoke with a few people at vForum Sydney recently on the new certifications, and I'm glad I'm not the only one to have a different opinion on them.

Having the VCA exams can help build confidence on the way to sitting the VCP exam.

The exam is priced at $120, but there's promotions and discount codes to make it free, so a LOT of people have been doing the exam.

You can attempt the exam at home with no supervision, and the time limit is extremely generous. If you happen to fail, you can immediately re-sit the exam.

There's the potential for people to exploit it and google the answers and keep attempting it until they pass.

Because of all this, I doubt there will be any industry recognition for the VCA exams.

If you are looking to get into VMware and virtualisation, the VCA exams may be a great start and motivator towards VCP, especially now while the discounts are available.

If there was a real cost involved, and a minimum period to wait for re-tries, it may slow down the non-genuine people, but the horse has already bolted on this one. I think VMware have made a bad decision on this one.

[Update: On a recent podcast (VMware Communities Roundtable Podcast – 255 – VCA – Education & Certification), VMware Education encouraged people working together and googling the answers. Effectively an 'open book' test. ]

Thursday, 10 October 2013

My VCAP-DCA experience

Everyone else has done one, so I hope there's something unique that comes out of this.

The Study

I made some notes as I was studying for VCAP-DCA, so these may not necessarily reflect what was on the exam.

I found an awesome ESXTOP poster at No idea how I hadn't come across this before.

I'd never really used esxcli much. I'd cut & paste esxcfg/vicfg lines that I've been hanging on to, but these commands were mainly for ESX3/4. esxcli has been greatly improved with ESXi5.x, so this is what you want to learn. Not everything is available in esxcli, such as setting up SNMP, you need to use vicfg-snmp, as there's no equivalent esxcli command in ESXi5.0.  BUT... there is for ESXi5.1. Remember, the exam is based on 5.0, so you may have to use a variety of methods.

The best command in esxcli, is:
esxcli esxcli command list

This shows ALL esxcli commands. It helps if you get confused with esxcli storage and esxcli core storage.

When using esxcli, don't try and remember the entire syntax, if you type in the namespace and hit enter, it will show you available options. Up arrow and add the next command, hit enter. Repeat until you find what you are looking for.

esxcli storage
esxcli storage nmp
esxcli storage nmp satp

If you do get stuck on any questions, the PDF manuals are there. It's good to know the titles, and a rough idea what content is in each. Although from within Acrobat, you can choose to search an entire folder of PDF's for keywords.I found the search function extremely quick.

Make sure you have a look at the UI simulation of the exam -

I spent a lot of time with Jason Nash and his TrainSignal / Pluralsight training videos. These are amazing. Jason covers everything on the blueprint. I watched a chapter, then went and did it in the lab. Several times! And our APAC vBrownBag mate Alastair Cooke has some great podcasts to watch.

The Exam

I panicked at the beginning of the exam as it said there were 10 questions. But on every other blog, they mention 26 questions! I calmed down after I realised they were SURVEY questions.

At the beginning of the exam, it gives you a run down of servers in the exam environment. I made a list of all the server names and IP's with their passwords on a sheet of paper so I could refer to it during the exam. But on my first attempt to access a host, it failed. It appeared DNS wasn't working, but later discovered the list of server names at the beginning was missing a prefix.

I kept an eye on the time for the first 30 minutes. The next time I saw it was with 30 minutes remaining. Wow, time flies when you are having fun. And to be honest, there was a point when I was really enjoying the tasks. This is why I work in this industry.

I was prepared for latency issues, and for the most part, it wasn't too bad. On ocassions when I had lots of text scrolling on the screen, that window became useless until the output had stopped. Nearing the end of the exam, the latency appeared to be more noticable (closer to midday in .AU).

Like everyone says, time is your enemy. Any lengthy questions I wasn't confident with I skipped over, so I was able to see all the questions, and then come back to them later. There were a few questions I wasn't able to complete in the time frame.

When the exam finished, I was confident on the questions I answered, but there's always stupid mistakes that can lose some points, plus the few questions I didn't answer, so I wasn't if I had worked fast enough to pass.

The Result

After an agonising 11 business days, I got the result.
Congratulations on passing the VMware Certified Advanced Professional 5 –Data Center Administration exam! 

The Summary

If you want to sit the VCAP-DCA exam, I highly recommend the TrainSignal / Pluralsight courses. Subscribe for a month, and watch a chapter every night. Read everyone else's exam experiences, as it will give you an idea of what to expect.

The speed of the search across the PDF's was impressive, but you need to know key words to make it useful.

I've sent feedback to the certification team as it would be great to have a pre-printed sheet containing all server names, IP's and login details. Afterall, they aren't testing our ability to transcribe or remember servernames and login details. I found it difficult to stop typing in IP's from my workplace!

For my day job, I live in the vSphere client, so some of the exam tasks were BAU, but I'm glad I took a few days off to study to cover everything else.

I'm just glad I've got VCAP-DCA out of the way.

Tuesday, 24 September 2013

2013 September Melbourne VMUG

The September Melbourne VMUG was on this week, and it was a ripper.

The agenda was set for:
  • EMC Presentation: Continuous Availability with VPLEX - Berin Smithson
  • VMware Presentation: What's New in vSphere 5.5 - Greg Mulholland
  • Building a Service Provider Cloud Offering - Arron Stebbing
Coming off VMworld only 3 weeks before, we knew there was a lot of great info to pass on to the crowd, and there was a higher than average number of registrations. The room was packed.

It was great to see a lot of first timers, and some familiar faces that only signed up to the VMUG at the Melbourne VMware Series 2013 back in July.

The big news for the VMUG is that we have found a new venue to hold our quarterly meetings. The next meeting on Thursday 21st November will be at the Telstra Conference Centre, 242 Exhibition St, Melbourne. The main entrance is on the corner of Lonsdale St & Exhibition St. Follow the signs for "Conference Centre".

View Melbourne VMUG at Telstra 242 Conference Centre in a larger map

We are really excited about the possibilities this venue opens for the Melbourne VMUG. It has a much larger capacity, and the room can be divided to run 2 concurrent presentations. Hopefully having more content, it will appeal to wider range of people, and for any community presenters, it won't feel as daunting, as they will have a more intimate audience.

In having more content, we need more community presenters for next year. So if you've implemented a solution, or overcome a problem, we'd like to hear about it. You may not think it's worthy, but other users are probably going through the same pain you were, so lets help them out, and you have a chance of winning a VMware training course of your choice.

Recently in the community, Scott Lowe, Cody Bunch and several others have written articles on "Putting the User back into User Group" and offered to help first time presenters in getting over their stage fright and mentor them with their presentations. This is a great idea, so if you would like to talk at the Melbourne VMUG, let us know, and we can help you. I dread getting up in front of a crowd, but i'm going to make it my goal for the next 12 months. C'mon, you can do it too.

I briefly spoke about the social side of VMworld, and how there are great opportunities to network with other people while you're there. There's plenty of content that comes out through blogs and the recorded sessions, but meeting the people there is a great experience. It's amazing to be able to talk to VMware engineers, and find out reasons why they did certain things in a product, or bond with other VMware users trying to solve similar issues as yourself.

Alastair Cooke invited everyone going to vForum in Sydney next month, to come along to VMDownUnderground. This is a great chance to socialise with the APAC VMware community.

We gave away HEAPS of TrainSignal / PluralSight 30 day training vouchers, and at vBeers we gave away VMware Press books, Tshirts, and a Galaxy Tab 3.

Aaron, the winner of a Galaxy Tab 3

The content from the presentations is available on the Melbourne VMUG workspace.

The next Melbourne VMUG is on Thursday 21st November. See you there. Don't forget to follow @mvmug on twitter for more news.

Wednesday, 21 August 2013

Force mount missing datastores

By accident while in Cluster Settings / Datastore Heartbeating, I noticed a datastore wasn't available of one of the hosts.

Trying to mount it from the vSphere client failed with a popup:
Call "HostStorageSystem.ResolveMultipleUnresolvedVmfsVolumes" for object "storageSystem-326" on vCenter Server "vcenter" failed.

The command to force mount a snapshot that is persistent has changed from ESX(i)4.x to ESXi5.0.

The details are at

Use SSH or ESXi Shell and run:
# esxcli storage vmfs snapshot list
# esxcli storage vmfs snapshot mount -l "VMFS_NAME" 

This should mounted the volume successfully.

Sunday, 28 July 2013

Sit the IaaS exam for US$11

After reading the #vBrownBag IaaS study guide from @coolsport00, and exams being 75% off at VMworld, I decided to book the IaaS exam. The exam is usually US$225.

I booked it using the VMworld SF 75% discount code, and thought I'd be cheeky and try and apply the VMUG Advantage 20% discount as well. It worked!  It appears to do the 20% off the ORIGINAL price of the exam.

So at the checkout screen, it was saying the total cost would be US$11.25.

Wow. That's a great discount.

If you're going to VMworld, and have a VMUG advantage, give it a go. Let me know if it works for other exams at VMworld.

Friday, 19 July 2013

Slope info magic number check failed

While using Splunk I noticed 1 ESX host had a huge amount of logs compared with the others in that cluster. Looking into it, every hour there were about 23,000 entries for:

storageRM:  Slope info magic number check failed. On disk 0x0, expected 0x1df5e76.

There was only 1 hit on google from a twitter conversation with @northlandboy & @blomjoh asking if anyone knew what the error was.

I logged an SR for it. But magically the next day, the error stopped appearing in the logs.

The response from VMware Support:
Here is What I could find regarding this error.
A bug in storageRM sets the magic number to zero .Since this magic number is only used for internal data structure bookkeeping so if the reset is due to this bug then there is no affect on storageRM functions and the data is safe . If this error reoccurs frequently even after the reset then there could be a disk corruption.

Thank you for your response.We do have a workaround to stop these messages to occur - However this is not a tested solution.We do not have much information on this at this time,as we have seen this occur very few times and stopping randomly(like in your case) - We have not been able to collect much information and hence we do not have tested solution on this as yet.we do have a bug report filed for ths however.
The workaround outline is as follows -

To reset the magic number and stop the error from coming please following the instruction:
1. Increase storageRM log level on host where failure is seen --
vsish -e set /config/Misc/intOpts/SIOControlLoglevel 5
2. Wait for the error to show up
3. Disable storage I/O control on the datastore which exhibits this problem.
4. Stop storage I/O control on all hosts sharing the datastore
5. Run command /sbin/storageRM -R
* "-R" is a troubleshooting option hence it is not listed in the men page
* Example: "/sbin/storageRM -R /vmfs/volumes/FDLD_VMTEST0508."
6. Start storage I/O control on all hosts
7. Enable storage I/O control on datastore.

The purpose of posting this workaround is for information only. As it says, "this is not a tested solution". If you see this in your logs, log an SR with VMware.

Sunday, 14 July 2013

2013 July Melbourne VMUG

The next Melbourne VMUG is on Wednesday 24th July.

Veeam will be there doing live demos and talking about the upcoming release of v7 of Veeam Backup & Replication. Veeam has to be the most popular (and coolest) backup application.

VMware will be presenting vCAC - vCloud Automation Center. Going by the amount of submissions for VMworld on vCAC, this is going to be the next hottest thing VMware will be pushing.

There will be an awesome community presentation on vCenter Operations by Adam Selwood. He did a popular community presentation at the all day VMUG user conference in February this year.

Prizes Prizes Prizes!!  We'll be giving away a VMware course of your choice and we have several books available.  There may also be a secret twitter giveaway, so be sure to follow @mvmug for more details.

Hang around for vBeers and mingle with the crowd. If you don't know anyone, come and introduce yourself to the VMUG committee.

"How's my driving?".  We're always after attendees feedback. Let us know what YOU are interested in. We are doing this for YOU. Want more VMware Press books given away? Want a question & answer session? Want more free VMUG caps? Or just 'keep doing what you're doing'. Let us know: twitter: @mvmug, email:

Register at the VMUG website -

Hope to see you there.