VMware vSphere Quick Boot with ESXi 6.7

With release of vSphere 6.7 many enhancement to existing features and new features came. One of the efficient feature vSphere Quick Boot is available and I will be discussing about this feature in this post.

vSphere Quick Boot is an innovation by VMware and this feature helps on restarting the ESXi hypervisor without rebooting underlying  physical host . Quick Boot eliminates the time-consuming device initialization and self-testing procedures shortening the time. Not only the reboot , Quick Boot supports patching and upgrades of vSphere hosts through vSphere Update Manager.

This functionality is only available for hosts that are running ESXi 6.7 with compatible Hardware .Even if your hardware is compatible with Quick Boot and you are running a legacy version of ESXi, this feature will not be available. If the system is incompatible with vSphere 6.7 Quick Boot, it will perform a regular reboot.

 Supported Hardware Platforms

Currently Dell and HP are the OEM hardware vendors supports this feature and you can find the supported servers details on below links

With vSphere 6.7 a built in script available to test the hardware compatibility with ESXi Quick Boot.

Script Path - /usr/lib/vmware/loadesx/bin/loadESXCheckCompat.py

Detailed Information available on VMware KB

Verify the Host Compatibility

Login to ESXi as root from putty or Console

Verify the ESXi version and run the script

Configure vSphere Quick Boot 

Configuring VMware ESXi 6.7 Quick Boot functionality must be done through VMware Flex client and Upgrades or Patching Can be done though HTML Client

Navigate to Home > Update Manager

Select the vCenter Server and navigate to Manage - >  Host/Cluster Settings

You can see Quick Boot is not enabled , Click on Edit button go to configuration option.

Select the Enable Quick Boot option by adding Tick on the box and Click OK

After Enabling  , Verify the Quick Boot Status  .

 

 

 

 

 

 

 

The Update Manager UI will present Quick Boot as an option for servers that support the feature, however for some servers this option will not be available and your host will perform regular reboot rather than a Quick Boot.

You may see this issue if:

  • Host platform is not supported.
  • Host is configured to use a TPM.
  • passthru devices configured for VMs on your host.
  • vmklinux drivers loaded on your host.
  • Other non-certified drivers loaded on your host
  • Quick Boot is disabled in the Update Manager UI.

More vSphere 6.7 Posts

Refer VMware Documentation for More details .


VM Backup Failing with No Proper Error Details on EMC NMC

In this post I am going to share how to troubleshoot and fix the VM backup failing issue on EMC networker with out any proper error on NMC

Check the Below images we are getting error as below one of the Daily Backup is failed

Click on the Failed Work flow and select show details and you don't see any error on the Failed VMs Tab . It means Backup is failed due to some issue but no proper information

Only Info you will get it will show how many VMs failed over there but no other information , also if you check the policy group you will find the number of successful backed up vms will be same .

From this we can understand that  policy is looking for some other VM which is not available on the group . This is normally happening when if you remove a VM from vCenter or Delete and not removed from backup policy group .

Lets have a look how can we  find out which VM is casing the issue .

Navigate the Policy Folder , in my case below is the location

D:\Program Files\EMC NetWorker\nsr\logs\policy\VMWare VProxy\VMWare Daily 02\

Open the Latest Policy raw Log  , Log ending with .raw file extension , Here it is Backup_132361.raw

And you can find the virtual machine UUID which is casing the issue

Next You have to update the policy Group with the VM which is not available on group as shown below  and you can see next shchedule of the backup will not show this error.

Note :- If there is multiple VMs with same error  , you have run the above command with all VMs UUID

Issue

nsrvproxy_save NSR ERROR Unable to find selected VM work item with UUID="<UID>" in vCenter, the work item will be skipped , Unable to find selected VM work item with UUID "UUID" in vCenter, the work item will be skipped.

Cause

The virtual machine that was protected by the VMware Protection Policy has been deleted from the vCenter inventory. The NetWorker nsrdb database stores the virtual machine by UUID and performs a lookup of the virtual machine at the start of the backup workflow.

Resolution

This behaviour is by design. The nsrpolicy command line utility can be used to update the group that's assigned to the failing policy:
nsrpolicy group update vmware -g *GROUP_NAME* -O "UUID"

Reference


Virtual Machine Backup and Restore Consideration

Snapshot is a critical component of VMware backup. it’s very important that you understand the role of the VMware snapshot and how VM backup and restore works.

Basic operation

The process of VM backup begins with the backup software taking a VMware snapshot of the VM and it performs two main tasks.

First quiescing of the virtual machine and flushes the disk contents out of the file system cache. This control both the OS and application versions of Microsoft Volume Shadow Copy Service (VSS) safeguards that files may be locked and in use inside Windows are released in such a way that a full backup is possible to occur. If there is an issue with any third-party providers or the VSS service itself, the snapshot operation may fail it will effect backup operation.

Here VMware Tools have a major role over here, VM Tools used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.

How Tools helping

When using the VMware Snapshot Provider service in VMware Tools, you start the process of creating a new Volume Snapshot Service (VSS) snapshot inside the guest operating system (OS). All the registered VSS writers receive the request and prepare applications for backup by writing transactions from the memory to disk. When the VSS writers finish this operation, they report to the VMware Tools Service via the VMware Snapshot Provider that the job is complete and the system is ready for a snapshot.

You can check the registered VSS writers by running command inside the operating system vssadmin list writers

Second, the VMware snapshot unlocks the files that make up a VM from the file system. When a snapshot is involved, each virtual disk receives a snapshot delta like “vmname-00001.vmdk”. From this point onwards, all disk changes accrue in the delta files, which grow in increments of 16 MB. This give the permission to the files that make up the VM, such as the VM’s configuration file (VMX) and virtual disk files (VMDK), free to be archived.

Without the snapshot involvement, the files would be locked by the ESX server that “owns” that VM when it is powered on. The situation is similar to when you try to copy or move a file which already accessed by an application.

How Snapshot effect your Virtual environment

s we know there are a number of challenges associated with the use of snapshots. The longer the snapshot keeps, the larger the delta files and snapshot size will be increased with VM data. This will lead to performance impact and it will consume you all free space available on datastore. Also it will lead to an outage of your virtual machine which is having snapshot as well as other virtual machines resides on the datastore due to insufficient space.

One of the other main concern is how the backup solution will handle snapshot removal. Once the backup job has completed, an instruction will be sent either vCenter or directly to the ESX servers to remove the snapshot from the virtual machine. Any failure on communication from the backup system to the management layer of vSphere will result an “orphaned” snapshots after backup Job.

Most of the backup solution will create a log and alert on same to notify that there is failure on snapshot removal. And real scenario backup solution is not that much intelligent to identify those orphaned snapshot and remove those on the next backup runs.

Restoration

There is multiple option we will get with backup solutions for restoration, mainly there are two methods used by backup solutions

First, the backed-up VMDK files are taken from a shared location accessible to the ESX hosts and “hot-added” to the VM to which they need to be recovered. The result is that the virtual machine will have a new drive added to it while it is powered on. This appears as new drive with “ X “ letter , and this allows the application owner of the VM to restore files using Windows Explorer. Also some solution will allow you to view the file from the restore window and you can select from there and perform restore.

Second, the entire VM will be restored as a new virtual machine to original location or different location, later using Storage vMotion to relocate the restored VM to its rightful location.

Note: - Now days vendors are coming with more options such as instant, point in time recovery etc .

One of the important thing we have to consider in virtual machine backup is backup type such as Full or Incremental.

Full virtual machine backup processing stores a backup copy of all virtual disk images and configuration information for a virtual machine.

Advantages

With full virtual machine backup processing, you get faster data movement than a file-level backup.

Disadvantages

  • Backups are not granular.
  • Full virtual machine backup operations enable a complete restore of a virtual machine, but they take more time and more server space than a file-level or incremental backup.

An incremental backup of a virtual machine backs up all changes that occurred since the previous backup of the virtual machine, whether the backup was a full backup, or another incremental backup.

Advantages

Incremental backup processing backs up changes to virtual machines between full virtual machine backups.

Disadvantages

  • The size of incremental backups can increase if you do not run a full backup regularly.
  • It is inefficient to restore data from incremental backups because the process must automatically complete the following tasks:
    • Restore the most recent full backup.
    • Restore each incremental backup up to the specified recovery point.

In virtual machine backup mainly forever incremental backup is the best or not? Find out from below mentioned advantages and disadvantages.

Advantages

  • Periodic full virtual machine backups are no longer necessary.
  • This method reduces the amount of data that goes across the network.
  • This method reduces data growth because all incremental backups contain only the blocks that changed since the previous backup.
  • No comparison with the backup target is required since only changed blocks are identified.
  • Impact to the client system is minimized.
  • The length of the backup window is reduced.
  • Data restore operations are simplified.
  • This method optimizes data restore operations.

Disadvantages

  • If one or more of the progressive incremental backups is corrupted on the server, you might not be able to fully recover a virtual machine. To ensure that you can fully recover a virtual machine, periodically run a full virtual machine backup.

 


EMC vProxy Error disabling storage migration, retrying, error: VDDK Error

This post is relate to virtual machine backup failing on EMC network environment .

Error 

NVP: Error disabling storage migration, retrying, error: VDDK Error: 20005: The operation has been disabled by the guest operating system.

Few backups fail with error 'operation has been disabled by the guest operating system':

Cause

vProxy attempts to disable VM relocation before backup. However, the vCenter server has already been set with option to disable VM relocation. This option can be set by user or an application like a previous backup using NVP. If this is set by NVP- the flag should be cleared at the end of backup, however a bad termination to backup may cause this cleanup procedure to fail.

First We have to verify this issue by flowing steps

  • Log in to vProxy as root
  • Change to bin directory
    cd  /opt/emc/vproxy/bin
  • Run below command to load library path:
    source ../unit/vproxy.env

Note :- You should d have access to vCenter through SSH  , else enable that from vCenter before proceeding .

  •  Check the VM ID which is failing from Logs  , here it is VM-92

  • Check if the migration is disabled with 'vmconfig' tool:
./vmconfig -c info -k "vm-MOREF" -l moref -p Password-for-vCenter -u username-for-vCenter -v vcentername
NOTE: to get the 'vm-MOREF' go to NMC, click 'show message' and 'Get Full log', here you will see the MOREf similar to 'vm-1311'. For more help- type './vmconfig'
  • if Migration Enabled is 'false'- proceed with workaround , Here you can see it is false

Note : If Migration enabled is 'true'- further diagnosis may be needed. In this case, there is no need to apply workaround

  • Run vmconfig tool with "enable-migration" to manually enable migration

  • Now Run the Backup and Check the status .

EMC Networker vProxy VM Backup Issue - Error disabling storage migration

This is one of the major issue with emc networker vproxy backup .The workaround for solving the issue , we have to enable the storage migration from proxy console 


Steps to verify issue:

  1. Log in to vProxy as root
  2. Change to bin directory
    cd  /opt/emc/vproxy/bin
  3. Run below command to load library path:
    source ../unit/vproxy.env
  4. Check if the migration is disabled with 'vmconfig' tool:
    ./vmconfig -c info -k "vm-MOREF" -l moref -p Password-for-vCenter -u username-for-vCenter -v vcentername
    NOTE: to get the 'vm-MOREF' go to NMC, click 'show message' and 'Get Full log', here you will see the MOREf similar to 'vm-1311'. For more help- type './vmconfig'
    Example output:
    Connected to session on vCenter 'vc.ad.box' Version '5.5.0 build-2646482, API:5.5', Uuid:F8C1509A-EEFA-47E7-AF4A-87C5804B27E6.
    Searching for virtual machine with MORef "vm-94" ...
    Querying vCenter for definition of virtual machine with MORef of "vm-94" ...
    Found virtual machine "VM1" with MORef "vm-94".
    Name: "VM-dir-101".
    CBT Enabled: true.
    Migration Enabled: false.
  5. If Migration Enabled is 'false'- proceed with workaround
  6. If Migration enabled is 'true'- further diagnosis may be needed. In this case, there is no need to apply workaround.

Note - "vm-MOREF"  is the  VM-ID of the virtualmachine shows the error , you will get this from full log option .

 Workaround:

Run vmconfig tool with "enable-migration" to manually enable migration

./vmconfig -c info -k "vm-MOREf" -l moref -p **** -u username -v vcentername -c "enable-migration"

example:
./vmconfig -c "enable-migration" -k vm-94 -l moref -p *** -u administrator@vsphere.local -v 10.1.3.4