Saturday, 15 October 2016

Faking vCenter alarms

Larger environments tend to integrate their monitoring and ticketing systems. Some also add automated workflows based on alarms. The problem when setting up these workflows is how do you test the workflow is triggered based on specific alarms?

With thanks to William Lam for the tips, it's possible to trigger specific events that make up alarms. It's not a simple task (for someone who's not a developer), but you can do some trial and error to work out how to trigger each event.

You need to find the event you want to trigger from the API at http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.wssdk.apiref.doc/index-do_types.html. Pretty much anything ending in 'Event'.

A list of events can be viewed with PowerCLI by connecting to vCenter:
$evt = get-view eventManager
$evt.Description.eventinfo

Lets choose HostConnectionLostEvent. The description says "This event records the loss of a host connection".

The PowerCLI script below will trigger the HostConnectionLostEvent event, and in turn triggers an alarm.


# Usual load modules and connections
import-module VMware.VimAutomation.Core
connect-viserver 192.168.10.99
$server = $global:DefaultVIServer
$entity = Get-View (Get-VMHost -Name 192.168.10.240)
$eventMgr = Get-View $server.ExtensionData.Content.EventManager
# Subsitute your event to trigger
$event = New-Object VMware.Vim.HostConnectionLostEvent
# This property is specific to this event, and is required. This domain and user doesn't have to be an existing account.
$event.UserName = "CAGE.LOCAL\Chaos Monkey"
$hostEventArg = New-Object VMware.Vim.HostEventArgument
$hostEventArg.Host = $entity.MoRef
$hostEventArg.Name = "Host-is-a-label"
$event.Host = $HostEventArg
# Sends the event to vCenter
$eventMgr.PostEvent($event,$null)



When trying different events, the main line is:

$event = New-Object VMware.Vim.HostConnectionLostEvent

If you view $event now, you'll see it's properties:
Key                  : 0
ChainId              : 0
CreatedTime          : 1/01/0001 12:00:00 AM
UserName             : 
Datacenter           : 
ComputeResource      : 
Host                 : 
Vm                   : 
Ds                   : 
Net                  : 
Dvs                  : 
FullFormattedMessage : 
ChangeTag            : 

You'll see other events have different properties. Some of these properties are required. It's a bit hit and miss.

Lets try using UplinkPortVlanUntrunkedEvent.

Substitute this line - $event = New-Object VMware.Vim.UplinkPortVlanUntrunkedEvent

Initially you'll get an error:
Exception calling "PostEvent" with "2" argument(s): "
Required property switchUuid is missing from data object of type UplinkPortVlanUntrunkedEvent
while parsing serialized DataObject of type vim.event.UplinkPortVlanUntrunkedEvent

So we are missing a required input of switchUuid. View the properties of $event:

SwitchUuid           : 
HealthResult         : 
Key                  : 0
ChainId              : 0
CreatedTime          : 1/01/0001 12:00:00 AM
UserName             : CAGE.LOCAL\Chaos Monkey
Datacenter           : 
ComputeResource      : 
Host                 : VMware.Vim.HostEventArgument
Vm                   : 
Ds                   : 
Net                  : 
Dvs                  : 
FullFormattedMessage : 
ChangeTag            : 

So add in a value for SwitchUuid:

$event.SwitchUuid = "Fake UUID"

and execute the remaining lines, and it triggers the alarm.


From there it was trial and error to to see which properties were mandatory for each event.

This was a quick hack to demonstrate the ability to trigger alarms for a customer. 

It also has other use cases:
Feeling lonely and on-call? Trigger an alarm and hopefully you'll receive a call.
Convince a co-worker they are causing errors by using their username.

Have fun.

Wednesday, 7 September 2016

ESXi kickstart with Python

I recently spent some time building kickstart files. Something I hadn't done much since 2008. The most noticeable change in the process is I'm older and fatter. I re/learnt a few things, and did my first ever python script.

First up, William Lam has plenty of articles on the topic, and the official vSphere installation doco is pretty good too.

While testing, it's much easier to link to a kickstart file or scripts from a webserver than directly embedded on the boot CD. For a lightweight webserver on Windows, I've used mongoose-free-6.4 available at https://www.cesanta.com/products/binary.

If you need to customise the installation process further, you can use Ash shell commands, or python 2.7 by using the switch --interpreter=busybox | python.

To see which python modules are available, view /lib/python2.7 on your ESXi host.

Mixing busybox and python.

I found it easier to start off with busybox, then call the python script.

%firstboot --interpreter=busybox

# For troubleshooting only
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

wget -O myscript.py http://webserver/myscript.py

chmod u+x myscript.py
/bin/python myscript.py


When testing the whole kickstart file, put sleep 600; in %pre, %post or %firstboot sections, and jump to the console (Alt-F1) to test commands. You can also watch detailed messages using Alt-F12.

Keep in mind you may not be in a full blown ESXi environment, depending on if you're in pre/post/firstboot.


%pre - Runs BEFORE the installation

%post - Runs AFTER installation. No hostd services are running. There is NO root password set and commands like esxcli won't be available. You'll need to use localcli instead.

%firstboot script is running now.
%firstboot - Runs at the very end of the first boot after installation. All services should be available. It's the last thing that runs before the console shows the ESXi host name, IP. Look for "Running 001.firstboot_001".

Deploy multiple ESXi hosts using kickstart and embedded CSV.


In a home lab or remote environment there may not be DHCP/PXE services, or justification for Auto Deploy. For that use case I created a boot CD that would install ESXi and set the hostname and IP based on matching the host's MAC address with a CSV file included on the CDROM. Perfect for where there is no DHCP. If you have DHCP and a webserver available, you can host the CSV there so it can be easily updated.

First, customise an ESXi boot CD from William Lam.

Create the CSV file (HOSTS.CSV) in the format Hostname,MAC,IP,Subnet Mask,Gateway

HostA,00:0c:29:aa:bb:cc,192.168.0.10,255.255.255.0,192.168.0.1
HostB,00:0c:29:aa:bb:cd,192.168.0.11,255.255.255.0,192.168.0.1

Create the python script (SETNET.PY) to check the CSV for the matching MAC address:

#!/usr/bin/python
import os, commands, csv, subprocess

MAC=subprocess.check_output("esxcli network ip interface list |grep MAC",shell=True)

MACADDR = MAC.split()

with open('/vmfs/volumes/datastore1/HOSTS.CSV', 'rb') as f:
        reader = csv.reader(f)
        for row in reader:
                if MACADDR[2] == row[1]:
                        os.system("esxcli system hostname set --fqdn=" + row[0])
                        os.system("esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=" + row[2] + " --netmask=" + row[3] + " --type=static")
print "End of loop"

[This was my first time to use python, so i chose a range of functions that worked at the time. Be wary of python's strict indentation requirements.]

Copy HOSTS.CSV and SETNET.PY to the custom ESXi boot CD just before the mkisofs step in Williams post.

The hard part was getting the files from the CD, to a location available during firstboot. If you're installing to an ESXi disk bigger than 6GB, a VMFS datastore will be created and we can use that for persistent storage.

Include the following in your ks.cfg:

%post --interpreter=busybox

cp /vmfs/volumes/CDROM/BUILD/HOSTS.CSV /vmfs/volumes/datastore1
cp /vmfs/volumes/CDROM/BUILD/SETNET.PY /vmfs/volumes/datastore1

%firstboot --interpreter=busybox

sleep 30;

chmod u+x /vmfs/volumes/datastore1/SETNET.PY
/bin/python /vmfs/volumes/datastore1/SETNET.PY


This gets the ESXi host configured with a hostname and IP where you can join it to vCenter and do the remaining configuration or apply Host Profiles.



Saturday, 6 February 2016

2016 Australian / New Zealand vExperts





The list of 2016 vExperts was officially announced by VMware. This year sees a total of 1374 people awarded the title of vExpert. Australia / New Zealand represent 4.2% of the total vExpert numbers.

This year the vExpert nomination form asked for your country, so lets hope VMware publish that info. So for now, this is a list of names we recognise as from Australia or New Zealand.

This year sees the number of locals grow to 58, up from 48 last year. There was quite a few that made it onto the 2015 second half year announcement.

Melbourne still leads, up 3 from last year, even though 3 people weren't renewed from last year, which means there were 5 people added from last year.

Canberra added their first during the mid year announcements last year, and they've added another since.

Sydney added 3 to their total from 2015, now up to 14.

Contrary to my previous comment about Brisbane, they snuck in 2 more this year.

Adelaide IS on the map. Amin Naserpourven (VCDX 188). He actually made the list last year.

The eastern island of Australia, also known as New Zealand, quietly gained 1, now up to 9.

There's been a lot of noise from the guys in Perth, but that's about it. No change from last year.

It's great to see the community growing each year, and hope to catch up at vForum.

I've picked up a few errors in the previous vExpert blogs, so I might have to go back and update those ones. Stay tuned.

If you see anyone I've missed, let me know.

NameLastTwitterRegion
DavidBarclay@davidbarclay99BrisbaneAdelaide1
NickBowie@nickbowienzNew ZealandBrisbane7
StevenBridle@virtuallyeucCanberraCanberra2
LukeBrown@luke_brPerthMelbourne22
AndrewBrydon@sidbrydonMelbourneNew Zealand9
AndreCarpenter@andrecarpenterMelbournePerth3
LuisConcistre@luisconcistreSydneySydney14
AlastairCooke@DemitasseNZNew ZealandGrand Total58
AndrewDauncey@daunce_Melbourne
DonovanDurand@donovanjdMelbourne
MarkElliott@eggme1New Zealand
FrankFan@frankfan7Melbourne
AndrewFirthSydney
MichaelFrancisBrisbane
DanFrith@penguinpunkBrisbane
RAMESHGEDDAMBrisbane
KevinGorman@Kev_McCloudMelbourne
MatthewHealy@matt232hMelbourne
BorisJelic@Boris_jelicMelbourne
ChrisJones@cpjones44Melbourne
StevenKang@ssbkangNew Zealand
PraveshKhanna@pravesh2012Melbourne
AskarKopbayev@AkopbayevSydney
SanitKumar@sanitkumarNew Zealand
WillyLEESydney
DavidLloyd@davlloydSydney
DavidManconi@dmanconiNew Zealand
WillMansfield@aussiewjmSydney
RyanMcBride@RyanMcBride81Sydney
GregMulholland@g_mulhollandMelbourne
NirajNaidu@mr_champySydney
AminNaserpour@AminNaserpourAdelaide
ScottNorris@auscottnorrisCanberra
JeffO'Connor@JeffOConnorAUSydney
GrantOrchard@grantorchardSydney
AaronParker@stealthpuppyMelbourne
ClintonPrenticeNew Zealand
DavidQuinney@quinney_davidSydney
JahninRajamoni@jahninSydney
SimonSharwood@ssharwoodSydney
KeiranShelden@Keiran_SheldenBrisbane
MannySidhu@MannySidhu2Melbourne
BrettSinclair@Pragmatic_IOMelbourne
AnthonySpiteri@anthonyspiteriPerth
ArronStebbing@ArronStebbingMelbourne
TasTareq@justonetazBrisbane
TysonThenMelbourne
MarkUkotic@oringinalukoMelbourne
RobWaite@rob_waite_ozMelbourne
JonWaite@jondwaiteNew Zealand
JustinWarren@jpwarrenMelbourne
CraigWaters@cswaters1Melbourne
MichaelWebster@vcdxnz001New Zealand
NathanWheat@wheatcloudMelbourne
ShaneWhite@ausvmguyMelbourne
TimWilliams@ymmit85Perth
JeffWong@jumpyjwSydney
TravisWood@vTravWoodBrisbane


A summary of the changes between 2015 & 2016:

IN
StevenBridle@virtuallyeucCanberra
KevinGorman@Kev_McCloudMelbourne
BorisJelic@Boris_jelicMelbourne
ChrisJones@cpjones44Melbourne
PraveshKhanna@pravesh2012Melbourne
AskarKopbayev@AkopbayevSydney
WillMansfield@aussiewjmSydney
TysonThenMelbourne
TravisWood@vTravWoodBrisbane
ClintonPrenticeNew Zealand
WillyLEESydney
OUT
HarshaHosur@harsha_hourMelbourne
JoshOdgers@josh_odgersMelbourne
ShanonOlsson@sfolssonMelbourne