Blog from June, 2022

VSS Command Line Interface (vss-cli) v2022.6.1
JM Lopez posted on Jun 23, 2022

VSS-CLI v2022.6.1 is available for download via PyPI or GitLab. Latest version includes the following improvements and bug fixes:

Bug Fixes

compute vm mk from-clib: --additional-params error even if not provided. (#530)

compute vm mk from-file: ignores admin in vss-cli configuration spec. (#531)

Improvements

core: update pyjwt to 2.4.0. (#527)

core: update pyvss to 2022.6.0. (#528)

compute vm mk from-clib: option --day-zero config and --id-token for Day0 configuration (#525)

compute vm mk from-file: support day-zero configuration via config and id-token in vss-cli config spec (#526)

compute vm get tpm: list virtual trusted platform module in virtual machine (#524)

compute vm set tpm mk: create virtual trusted platform module (#523)

compute vm set tpm rm: remove virtual trusted platform module (#523)

Upgrade

Remember, VSS-CLI documentation is now available in VSS-CLI and the full Change Log is available here. Upgrade or install VSS-CLI as follows:

# using vss-cli
vss-cli upgrade 

# upgrade with pip
pip install vss-cli --upgrade

# install
pip install vss-cli

For more information, please refer to the official documentation site.

Windows Guest OS encounters BSOD after Virtual Machine is vMotioned to ESXi 7.0+
JM Lopez posted on Jun 20, 2022

This issue has been fixed in vSphere 7.0 Update 3i (December 2022)

Incident

After completing the https://eis-vss.atlassian.net/wiki/spaces/VSSPublic/blog/2022/04/22/1073512459/VMware+vCenter+Server+Maintenance+May+14+2022+Sat+08+00AM+-+08+00PM, the VSS Team have been upgrading ESXi hosts on the ITS Private Cloud to vSphere 7.0U2 for the last month. This task has been performed without impacting end users due to VMware Live vMotion Technologies.

Today, at ~90% of hosts upgraded, we have had an increasing number of reports about Windows Guest OS (from Windows Server 2012 to 2019) running on cluster FD4 encountering Blue Screen of Death (BSOD) errors after being vMotioned to an upgrade host.

Symptoms

  • Windows crash with BSOD and/OR.

  • Windows cannot boot showing the message "OS failed to boot with no operating system found".

Cause

Apparently, we are experiencing a bug in vSphere 7.0 documented and recently published (June 1, 2022) in Windows Guest OS encounters BSOD after Virtual Machine is vMotioned to ESXi version 7.0U2 or higher (88516).

Virtual Machine File System (VMFS) under certain use case uses address optimization logic to avoid disk reads while resolving logical address to physical addresses. This address optimization helps in read performance and is used only in certain cases depending on how the virtual disk is allocated, alignment of address allocation and usage of large File Blocks (LFB)s for allocation. Due to a bug in this address optimization logic VMFS can return zeros during read causing the above issue. (VMware 88516)

Workaround

Virtual Machine

There is evidence that a power cycle (power off/power on) fixes the BSOD errors and potentially, a power cycle (power off/power on) might be helpful to avoid memory errors by bringing a fresh running version of the operating system with new vSphere settings. 

If one of your VMs is currently in crashed state:

  1. Power off the virtual machine. 

    vss-cli compute vm set <id> off

  2. Power on the virtual machine. 

    vss-cli compute vm set <id> on

Virtual Machine

We strongly suggest all VM admins to upgrade VMware tools to the latest version to ensure Guest Operating System stability.

ESXi Host

2022-06-20 01:20 PM: We are assessing the impact of the workaround suggested in the KB article with VMware Support and as soon we have the full impact of the change, we will implement it to fix the issues.

2022-06-20 01:54PM: Staff started the deployment of workaround suggested by KB 88516 in Cluster FD4.

2022-06-20 05:40PM: Staff applied the workaround suggested by KB 88516 in Cluster FD4.

2022-06-21 09:00AM: Staff started the deployment of workaround suggested by KB 88516 in Cluster FD1, FD2, FD3.

2022-06-21 05:00PM: Staff applied the workaround suggested by KB 88516 in Cluster FD1, FD2, FD3.

Solution

Currently there is no resolution to the issue (VMware).

Please, reach out to vss (at) eis.utoronto (dot) ca if you have any questions or you experience this issue on a different domain.

Announcing VSKEY-GW sunset and introducing VSKEY-VN
JM Lopez posted on Jun 03, 2022

We are pleased to announce that after around 10 years of operation, the ITS Private Cloud OpenVPN server at VSKEY-GW (Gateway) will be decommissioned on November 1, 2022 (Tue) 09:00AM. A replacement OpenVPN server VSKEY-VN (Virtual Network) is currently operational. VSKEY-VN is configured to work just like VSKEY-GW OpenVPN server, but with a fresh server certificate.

To ensure a smooth transition for VSS users, both servers VSKEY-GW and VSKEY-VN will run in parallel for 5 months. To begin using VSKEY-VN, please refer to the recently updated Install and configure VSS-VPN, or follow the next simple steps to get the new configuration file(s):

  1. Go to the new VSKEY-VN OpenVPN unified configuration retrieval page.

  2. Check your email, download the vskey-vn.ovpn file.

  3. Install the ovpn profile either with OpenVPN Client or Tunnelblick.

  4. Once installed, connect to vskey-vn and type your VSS credentials.

If you don’t see an email from “vss-webadmin (at) void.utoronto.ca”, please check spam/junk folders.

Just like that, you have been migrated to the new VSKEY-VN 🎉

WireGuard VPNs will not be affected by this transition.