VMware EVC

VM can be vMotioned to new generation hardware, and vMotion back to its original host, as long as the VM has not been powered off on the destination host with new hardware.

When VM is powered off and powered back on new host, the new host EVC mode setting will taken into effect. If VM hardware version is too old , there could be kernel panic, at least for Linux OS. Guest OS might prompt user interaction in VM console to confirm regular boot process, depending on vendor, then guest OS will start successfully. However after this power-back process, VM could not vMotion back to its original host anymore, since EVC setting is working for the VM. EVC compatibility prevents reverse process.

ansible-doc vmware.vmware_rest.vcenter_ovf_libraryitem

Under - deployment_spec section, Ansible-doc does not provide enough detail on `additional_parameters'  as displayed below:

- `additional_parameters' (list): Additional OVF parameters that may be needed for the deployment. Additional OVF parameters may be required by the OVF descriptor of the OVF package in the library item. Examples of OVF parameters that can be specified through this field include, but are not limited to:

• {@link DeploymentOptionParams}
• {@link ExtraConfigParams}
• {@link IpAllocationParams}
• {@link PropertyParams}
• {@link ScaleOutParams}
• {@link VcenterExtensionParams}

This causes vwmare_rest API call to deploy third-party OVF from content library very difficult to fill in all necessary properties in correct syntax.

Link below contains the class detail, but not syntax format. (I understand that a programmer will be able to figure it out)

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/1cd28284-3b72-4885-9e31-d1c6d9e26686/71ef7304-a6c9-43b3-a3cd-868b2c236c81/doc/operations/com/vmware/vcenter/ovf/library_item.deploy-operation.html

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/1cd28284-3b72-4885-9e31-d1c6d9e26686/71ef7304-a6c9-43b3-a3cd-868b2c236c81/doc/structures/com/vmware/vcenter/ovf/property-structure.html

https://vdc-download.vmware.com/vmwb-repository/dcr-public/4f867b1e-c524-4e4b-b79e-4321822b5c70/a3cd39a7-4eaf-4e10-9ad2-589f9dfe1f9a/doc/Com/Vmware/Vcenter/Ovf.html

Link below provide a quick way to get json format via vCenter API explorer. 

https://stackoverflow.com/questions/47912744/propertyparams-when-deploying-vm-from-ovf

Just convert it into yaml format for readability, then use it in Ansible playbook properly.

PS: deploy ovf from content library is much faster than from local ova/ovf file, if you need to deploy it into multiple VM instances, and one time upload is worthy.

UEFI PXE ESXi6 TFTP iPXE

Newer generation hardware default boot mode could be UEFI now, so existing PXE server might not work anymore.

VMware ESXi 6 ISO installation will still work for UEFI hardware, but there are changes for PXE boot process. Here is PDF link from VMware Installing VMware ESXi 6.0 Using PXE

Like always, my favorite blog site from William Lam already posted related blog a while ago about this:
http://www.virtuallyghetto.com/2015/10/support-for-uefi-pxe-boot-introduced-in-esxi-6-0.html
(Thank William again!)

VMware site PDF actually added obvious details which is missing from William: As we will use boot file from /efi/boot/bootx64.efi from installation ISO directly, instead of original pxelinux.0, so we need put boot.cfg file in the same location as mboot.efi. (bootx64.efi is renamed as mboot.efi), otherwise mboot.efi will not find boot.cfg during PXE boot process and throw out fatal TFTP error (15) about parsing boot.cfg. 

Update: Later version of ESXi uses b.b00 as kernel in boot.cfg file, insead of tboot.

We might encounter TFTP error (24) during PXE boot process if we start PXE Boot with a web server

Update: mboot.efi doesn't have http capability for tftp to continue, so we need start iPXE first. iPXE can be obtained from ipxe.org, or just download package to from vCenter server Auto Deploy service. File name inside the package for iPXE is: snponly64.efi. 

We don't have such issue for legacy BIOS, however now it is all about UEFI. If you decide to test out UEFI HTTP chainloading in VMWare VM, you need Burning iPXE into ROM for network adapter. (https://ipxe.org/howto/vmware)

The process is so-called chainload: snponly64.efi is obtained from tftp server, then it is look for mboot.efi from tftp server. Configuration in dhcpd.conf includes lines below:

class "pxeclients" {

   match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";

   next-server ip-of-tftp-server;

   if option client-system-arch = 00:07 or option client-system-arch = 00:09 {

      if exists user-class and option user-class = "iPXE" {

         # Instruct iPXE to load mboot.efi as secondary bootloader

         filename = "mboot.efi";

      } else {

         # Load the snponly64.efi configuration of iPXE as initial bootloader

         filename = "snponly64.efi";

      }

   } else {

      filename "gpxelinux.0";

   }

}

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.esxi.upgrade.doc/GUID-147D7509-EFB1-4391-973F-48B015B85C83.html

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.esxi.install.doc/GUID-91E32FD0-A33C-4302-9FAB-B52B8A5CEFBC.html#GUID-91E32FD0-A33C-4302-9FAB-B52B8A5CEFBC

Tip 1: before you start UEFI PXE boot process, please check UEFI boot order, and make sure PXE is after hard disk, otherwise PXE boot process will repeat after ESXi server reboot :-) (If your ks.cfg file includes reboot after installation)

Tip 2: We use PXE boot for multiple ESXi servers installation purpose, so the easiest way I found out about assigning IP address as you expected via DHCP is to edit dhcpd.conf file for DHCP reservation:

group {
host host-name1 {hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address ip-address1; option host-name "host-name1";
host host-name2 {hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address ip-address2; option host-name "host-name2
...
}

Of course, you need reboot DHCP for this, but you don't need multiple MAC-address-boot.cfg files. dhcpd.conf syntax is really picky, so you will follow the format exactly

VMware AppSpeed

vCenter AppSpeed will no longer be available for new license purchases as of January 3, 2012. However it will continue to be supported through July 3, 2013 in line with General Support Policy: www.vmware.com/support.

Customers who are active on a Support and Subscription (SnS) plan from December 1, 2011 until the vFabric Application Performance Manager entitlement becomes available in January 2012 may exchange their related licenses of vCenter AppSpeed and upgrade to vFabric Application Performance Manager at no additional license cost. This upgrade will take place on a one-to-one basis.

VMware vCenter AppSpeed provides proactive performance management and service-level reporting for services running within virtual appliances. By analyzing the traffic that flows between end users, Web applications, and back-end servers, AppSpeed provides visibility into multitier services and enables rapid identification of performance issues originating from inadequate resource allocation and service problems.

VM security recommendations

• Install Antivirus software
• Disable copy/past between guest OS and remote console, it will overwrite the settings from VMware tools
• Removing unnecessary hardware devices
• Prevent user or process from disconnecting devices: editing .vmx file
• Limiting guest OS write to Host memory: VM advance settings
• Configuring guest OS logging level (number, size, or just disable)

ESX default setuid and setgid applications

Default setuid applications:

• required: pam_timestamp_check, passwd, pwdb_chkpwd, ssh_keysign, su, unix_chkpwd, vmkload_app, vmware-authd, vmware-vmx
• optional: crontab, ping, sudo

Default setgid applications:

• wall optional
• lockfile required for Dell OM, otherwise optional.

To disable setuid/setgid, use chmod a-s/g <path_to_executable_file>

ESX Password complexity

By default, ESX use the pam_cracklib.so plug-in to set the rules that users must observe when creating passwords and check password strength during the creation process.

esxcfg-auth --usecrack=<> <> <> <> <> <>

pam_passwdqc.so provides a greater number of options for fine-tunning password strenth and performs passoword strength test for all users.

esxcfg-auth --userpasswdqc=<N0> <N1> <N2> <N3> <N4> <match>

chage - change user password expiry information

SYNOPSIS

chage

[-m mindays] [-M maxdays] [-d lastday] [-I inactive]
[-E expiredate] [-W warndays] user

chage

-l user

 

DESCRIPTION

chage changes the number of days between password changes and the date of the last password change. This information is used by the system to determine when a user must change her password. The chage command is restricted to the root user, except for the -l option, which may be used by an unprivileged user to determine when her password or account is due to expire. With the -m option, the value of mindays is the minimum number of days between password changes. A value of zero for this field indicates that the user may change her password at any time.
With the -M option, the value of maxdays is the maximum number of days during which a password is valid. When maxdays plus lastday is less than the current day, the user will be required to change her password before being able to use her account. This occurance can be planned for in advance by use of the -W option, which provides the user with advance warning.
With the -d option, the value of lastday is the number of days since January 1st, 1970 when the password was last changed. The date may also be expressed in the format YYYY-MM-DD (or the format more commonly used in your area).
The -E option is used to set a date on which the user's account will no longer be accessible. The expiredate option is the number of days since January 1, 1970 on which the accounted is locked. The date may also be expressed in the format YYYY-MM-DD (or the format more commonly used in your area). A user whose account is locked must contact the system administrator before being able to use the system again.
The -I option is used to set the number of days of inactivity after a password has expired before the account is locked. The inactive option is the number of days of inactivity. A value of 0 disables this feature. A user whose account is locked must contact the system administrator before being able to use the system again.
The -W option is used to set the number of days of warning before a password change is required. The warndays option is the number of days prior to the password expiring that a user will be warned her password is about to expire.
If none of the options are selected, chage operates in an interactive fashion, prompting the user with the current values for all of the fields. Enter the new value to change the field, or leave the line blank to use the current value. The current value is displayed between a pair of [ ] marks.  

NOTE

The chage program requires shadow password file to be available. Its functionality is not available when passwords are stored in the passwd file. 

vCenter user role

No Access: default role, except for root and vpxuser
Read-only
Administrator default for vpx user and root

Virtual DMZ

a Virtual switch contains two firewall server and other application server

• one firewall server with multi-home address is connected to intranet
• the other firewall server with multi-home address is connected to internet
• this virtual DMZ is essentially a extranet

Customizing VMware HA Behavior

VMware HA attributes:

• das.isolationaddress[1..9] you can specify up to 10 isolation address. Default isolation address is the gateway of service console network
• das.usedefaultisolationaddress (ture|false, true by default)
• das.failuredetectiontime host monitoring, 15 seconds. If there are extra isolation address, increase value to 20 or greater.
• das.failuredectioninterval 1 second by efault
• das.defaultfailovehost takes precedence over failover host name in the policy. Only one candidate.
• das.isolationShutdownTimeout 300 second for host isolation VM to be shutdown will be powered off
• das.slotMemInMB if used, slot size is memory reservation plus VM memory overhead in the cluster.
• das.slotCPUInMHz if used, slot size is CPU reservation plus VM CPU overhead in cluster
• das.vmMemoryMinMB VM memory reservation value, 0 by default
• das.vmCPUMinMHz VM CPU reservation is not zero or not specified, the default value is 256MHz
• das.iostatsinterval VM Monitoring reset after 120 seconds. 0 will disable VM Monitoring.

Changing the attribute value in bold, requires disable/re-enable HA to take into effect.

VM Monitoring

VM Monitoring restarts individual VMs if their VMware Tools heartbeats are not received within a set time. You can configure the degree to which VMware HA is sensitive to such non-responsiveness.

If VM Monitoring is enables, the VM Monitoring service using VMware Tools envaluates wheret each virtual machine in the cluster is running by checking for regular heartbeats from the VMware Tools process running inside the guest. If no heartbeats are received, this is most likely because the guest operating system has failed or VMware Tools is not being allocated any time to complete tasks. In such a case, the VM Monitoring service determines that the virtual machine has failed and the virtual machine is rebooted to restore service.

The VM monitoring also monitors VM's I/O activity. If no heartbeats are received with the failure interval, the I/O stats interval is checked. The default I/O status interval is 120 seconds. If there is no disk or network activity has occurred for the VM, the VM is reset.

das.iostatsInterval

To avoid restting VM repeatly from non-transient errors, by default, virtual machiens will be reset only three times during a certain configurable time interval. After that, VM HA will not make further attempts to reset VMs within specified time (reset period). Number of reset and Reset period could be configure by sensitivity respectively.

Heartbeat Failure Interval  between host and VM:

• high: 30 second, reset period: 1 hour
• medium: 60 second, reset period: 24 hour
• low: 120 seconds, reset period: 7 days

VMware Admission Control

vCenter server use admission control to ensure that sufficient resource are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected.

There are three type of admission control:

• Host, ensure that a host has sufficient resources to satisfy the reservations of all virtual machines running on it.
• Resource pool, ensures that a resource pool has sufficient resources to satisfy the reservations, shares, and limits of all virtual machines associated with it.
• VMware HA, ensures that sufficient resources in the cluster are reserved for virtual machines recovery in the event of failure.

Only VMwar HA could be disabled, however without it, there is no assurance that VM will be failed over after a host failure. Here is the use case for disabling VMware HA:

• DPM test
• VUM like automatic process
• maintenance process

 

Host Isolation response

Host network isolation occurs when the host is still running, but it can't any longer communicate with other hosts in the cluster. by default settings, if a host stops receiving heartbeats from all other hosts in the cluster for more than 12 seconds, it attempts to ping its isolation address. If tis also fails, the host declares itself as isolated from the network.

When thie isolated hot's network connection is not restored for 15 seconds or longer, the other hosts in the cluster treat it as failed and attempt to fail over its VMs. Since the VM files are locked by isolated host, and will won't proceed. The default action is to leave VM powered on isolated host.

To use the Shut down VM setting for host isolation response, VMware tools need to be installed on the VM. This option could reserve the VM state. If Shut-down take longer than 300 seconds or other defined time, VM will be powered off.

das.isolationShutdwonTimeout

If you ensure network path redundancy, host network isolation should be rare.

VM restart priority

VM restart priority determines the relative order in which VM are restarted after a host failure in the cluster.

such virtual machines are restarted sequentially on new hosts, with highest priority first with resource available at the cluster. It is likely that VM with lower restart priority won't be restarted on new hosts when there is resource contention in the cluster.

If the default restart priority of a VM is disable, it will stay on the failed host, and won't be powered on at all.

Host failure detection

HA agents communicate with each others and monitor the liveness of the hosts in the cluster. This is done by heartbeat exchange. By default, every second. If a 15-second period elapes without the receipts of heartbeats from host, and the host cannot be pinged, it is declared as failed. The VMs were running on failed host will be failed over to others hosts in the cluster with most available unreserved resource. (CPU and memory)

A cluster can have up to 5 primary node, and 27 secondary node.

VCB Proxy

A Windows application that provides a LAN-free fiber Channel or iSCSI-based backup solution that offloads the backup processing to a dedicated physical server.
VCB takes advantage of the snapshot functionality of ESX server to mount the snapshots into the file system of the dedicated VCB server.

Once the respective VM files are monted, entire VM or individual files can be backed up using third-party tools. VCB scripts integrate with serveral third-party back-up solution to provide a means of automating the backup process.

• Turning off automount before connecting to SAN
• Using one HBA, and disable other HBAs if any
• Add path at environmental variable for easy command-line
• Licensing VCB from client

V2P

http://www.vmware.com/support/v2p/

• Only covers the details from Windows OS
• Not supported by VMware

For Linux environments, the imaging tool (CloneZilla Live) is a reasonable choice.
Ghost 11.5 or above also support vmdk file to disk
Acronis True Image

Linux V2P

(Credit to: http://pleasedonttouchthescreen.blogspot.com/2011/08/linux-v2p.html)

Virtualization may be the best thing since the microprocessor itself, but sometimes you may have to backtrack your own footsteps.
Here are the steps to convert a Red Hat 5.5 linux vm running on an ESX to a physical box:

Using a disk imaging tool, dump the vmdk of the vm to the internal disk of the target machine.Boot the physical box with the Red Hat CD in rescue mode
Do not mount any partitions, but select "continue" to jump into the shell.

Mount the restored partition:

#chroot /mnt/sysimage

edit the modules list and put every driver the physical machine needs.
you can see which drivers are needed by looking at modprobe.conf from another machine that has the same hw as the one you are V2P-ing.
if you don't have another machine, you can use the same target machine: just before imaging it with the source vmdk, install it from scratch with the Red Hat CD and take note of the resulting modprobe.conf

#vi /etc/modprobe.conf

here is the one i used as a reference (an HP machine)

options bnx2x disable_tpa=1
alias eth0 bnx2
alias eth1 bnx2
alias eth2 tg3
alias eth3 tg3
alias scsi_hostadapter cciss
alias scsi_hostadapter1 lpfc
alias scsi_hostadapter2 usb-storage

then you will have to rebuild the initrd image to include the needed drivers.
note that kernel version number must match the one used on your vm.

/sbin/mkinitrd -v -f /boot/initrd-2.6.18-164.el5.img 2.6.18-164.el5

and reboot.
at boot time you will probably get some errors on X initialization because the graphic card has changed: just answer yes at the reconfigure option and select 800x600 as default resolution.

P2V V2V

 VMware converter is the background tool to completed the task: P2V assistant + VMware importer; disk size could be changed during the migration process

• Live/hot Migration

1. Guided Consolidation at vCenter : VMware Capacity Planner Service as the background process (Start the live migration process from consolidation plan by running VMware converter as tasks)
2. Just use the import Machine option from vCenter client with VMware converter plugin
3. From stand-alone VMware converter to select the physical box to be migrated.

• Cold Migration

       Conversion of a powered off physical box
       Need VMware vCenter converter boot CD from VMware on Download Drivers and Tools (VMware vCenter Server 4.0 Update 4 and modules->Show details)
       Cold clone is ideal for server with frequent updated data

V2V also use the VMware convert to convert also the support VM format from VMware or other third-party to desired VM format for ESX or VMplayer/Fusion/workstation.

I/O Analyzer

http://labs.vmware.com/flings/io-analyzer

I/O Analyzer is a virtual appliance solution by VMware that provides a simple and standardized way of measuring storage performance in a VMware vSphere virtualized environments. I/O Analyzer automates the traditional storage performance analysis cycle and reduces the performance diagnosis time from days to hours.

Features

• Integrated framework for storage performance testing
• Readily deployable virtual appliance
• Easily configure and launch I/O tests on multiple hosts
• View integrated performance results at both guest and host levels
• Easily export performance data for off-site analysis