Fix available
VMware has released a fix for this problem in the form of ESXi 7.0 Update 3k:
If you already face the issue, after patching the host to ESXi 7.0 Update 3k, just power on the affected Windows Server 2022 VMs. After you patch a host to ESXi 7.0 Update 3k, you can migrate a running Windows Server 2022 VM from a host of version earlier than ESXi 7.
You might remember that I'm a pretty big fan of Tailscale, which makes it easy to connect your various devices together in a secure tailnet, or private network. Tailscale is super simple to set up on most platforms, but you'll need to install it manually if there isn't a prebuilt package for your system.
Here's a condensed list of the steps that I took to manually install Tailscale on VMware's Photon OS, though the same (or similar) steps should also work on just about any other systemd-based system.
I've been leveraging the open-source Tanzu Community Edition Kubernetes distribution for a little while now, both in my home lab and at work, so I was disappointed to learn that VMware was abandoning the project. TCE had been a pretty good fit for my needs, and now I needed to search for a replacement. VMware is offering a free version of Tanzu Kubernetes Grid as a replacement, but it comes with a license solely for non-commercial use so I wouldn't be able to use it at work.
You may have heard that there's a new vSphere release out in the wild - vSphere 8, which just reached Initial Availability this week. Upgrading the vCenter in my single-host homelab is a very straightforward task, and using the included Lifecycle Manager would make quick work of patching a cluster of hosts... but things get a little trickier with a single host. I could write the installer ISO to a USB drive, boot the host off of that, and go through the install interactively, but what if physical access to the host is kind of inconvenient?
VMware vCenter does wonders for abstracting away the layers of complexity involved in managing a large virtual infrastructure, but when something goes wrong it can be challenging to find exactly where the problem lies. And it can be even harder to proactively address potential issues before they occur.
Fortunately there's a super-handy utility which can making diagnosing vCenter significantly easier, and it comes in the form of the vSphere Diagnostic Tool Fling.
Way back in 2020, VMware released vSphere 7 Update 1 and introduced the new vSphere Clustering Services (vCLS) to improve how cluster services like the Distributed Resource Scheduler (DRS) operate. vCLS deploys lightweight agent VMs directly on the cluster being managed, and those VMs provide a decoupled and distributed control plane to offload some of the management responsibilities from the vCenter server.
That's very cool, particularly in large continent-spanning environments or those which reach into multiple clouds, but it may not make sense to add those additional workloads in resource-constrained homelabs1.
I've been doing a bit of work lately to make my vRealize Automation setup more flexible and dynamic and less dependent upon hardcoded values. To that end, I thought it was probably about time to learn how to interact with the vRA REST API. I wrote this post to share what I've learned and give a quick crash course on how to start doing things with the API.
Exploration Toolkit Swagger It can be difficult to figure out where to start when learning a new API.
ESXi-ARM Fling v1.10 Update
On July 20, 2022, VMware released a major update for the ESXi-ARM Fling. Among other fixes and improvements, this version enables in-place ESXi upgrades and adds support for the Quartz64's on-board NIC. To update, I:
Wrote the new ISO installer to another USB drive. Attached the installer drive to the USB hub, next to the existing ESXi drive. Booted the installer and selected to upgrade ESXi on the existing device.
Not long ago, I deployed a Tanzu Community Edition Kubernetes cluster in my homelab, and then I fumbled through figuring out how to log into it from a different device than the one I'd used for deploying the cluster from the tanzu cli. That setup works great for playing with Kubernetes in my homelab but I'd love to do some Kubernetes with my team at work and I really need the ability to authenticate multiple users with domain credentials for that.
Now that VMware has released vCenter 7.0U3c to resolve the Log4Shell vulnerabilities I thought it might be fun to run a security scan against the upgraded VCSA in my homelab to see how it looks. Of course, I don't actually have a security scanner in that environment so I'll need to deploy one.
I start off by heading to tenable.com/products/nessus/nessus-essentials to register for a (free!) license key which will let me scan up to 16 hosts.
I recently wrote about getting started with VMware's Tanzu Community Edition and deploying phpIPAM as my first real-world Kubernetes workload. Well I've spent much of my time since then working on a script which would help to populate my phpIPAM instance with a list of networks to monitor.
Planning and Exporting The first step in making this work was to figure out which networks I wanted to import. We've got hundreds of different networks in use across our production vSphere environments.
When I set up my Tanzu Community Edition environment, I did so from a Linux VM since the containerized Linux environment on my Chromebook doesn't support the kind bootstrap cluster used for the deployment. But now that the Kubernetes cluster is up and running, I'd like to be able to connect to it directly without the aid of a jumpbox. How do I get the appropriate cluster configuration over to my Chromebook?
Lately I've been spending some time getting more familiar with VMware's Tanzu Community Edition Kubernetes distribution, but I'm still not quite familiar enough with the tanzu command line. If only there were a better way for me to discover the available commands for a given context and help me type them correctly...
Oh, but there is! You see, one of the available Tanzu commands is tanzu completion [shell], which will spit out the necessary code to generate handy context-based auto-completions appropriate for the shell of your choosing (provided that you choose either bash or zsh, that is).
I recently needed to export a list of all the Linux VMs in a rather large vSphere environment spanning multiple vCenters (and the entire globe), and I wanted to include information about which virtual datacenter each VM lived in to make it easier to map VMs to their physical location.
I've got a Connect-vCenters function that I use to quickly log into multiple vCenters at once. That then enables me to run a single query across the entire landscape - but what query?
Back in October, VMware announced Tanzu Community Edition as way to provide "a full-featured, easy-to-manage Kubernetes platform that’s perfect for users and learners alike." TCE bundles a bunch of open-source components together in a modular, "batteries included but swappable" way: I've been meaning to brush up on my Kubernetes skills so I thought deploying and using TCE in my self-contained homelab would be a fun and rewarding learning exercise - and it was!
I've been wanting to learn a bit more about SaltStack Config so I recently deployed SSC 8.6 to my environment (using vRealize Suite Lifecycle Manager to do so as described here). I selected the option to integrate with my pre-existing vRA and vIDM instances so that I wouldn't have to manage authentication directly since I recall that the LDAP authentication piece was a little clumsy the last time I tried it.
Thus far in my vRealize Automation project, I've primarily been handing the payload over to vRealize Orchestrator to do the heavy lifting on the back end. This approach works really well for complex multi-part workflows (like when generating unique hostnames), but it may be overkill for more linear tasks (such as just running some simple commands inside of a deployed guest OS). In this post, I'll explore how I use vRA Action Based eXtensibility (ABX) to do just that.
One of the requirements for my vRA deployments is the ability to automatically create a static A records for non-domain-joined systems so that users can connect without needing to know the IP address. The organization uses Microsoft DNS servers to provide resolution on the internal domain. At first glance, this shouldn't be too much of a problem: vRealize Orchestrator 8.x can run PowerShell scripts, and PowerShell can use the Add-DnsServerResourceRecord cmdlet to create the needed records.
Connecting a deployed Windows VM to an Active Directory domain is pretty easy; just apply an appropriately-configured customization spec and vCenter will take care of it for you. Of course, you'll likely then need to move the newly-created computer object to the correct Organizational Unit so that it gets all the right policies and such.
Fortunately, vRA 8 supports adding an Active Directory integration to handle staging computer objects in a designated OU.
In past posts, I started by creating a basic deployment infrastructure in Cloud Assembly and using tags to group those resources. I then wrote an integration to let vRA8 use phpIPAM for static address assignments. I implemented a vRO workflow for generating unique VM names which fit an organization's established naming standard, and then extended the workflow to avoid any naming conflicts in Active Directory and DNS. And, finally, I created an intelligent provisioning request form in Service Broker to make it easy for users to get the servers they need.
I was recently introduced to AdGuard Home by way of its very slick Home Assistant Add-On. Compared to the relatively-complicated Pi-hole setup that I had implemented several months back, AdGuard Home was much simpler to deploy (particularly since I basically just had to click the "Install" button from the Home Assistant add-ons manage). It also has a more modern UI with options arranged more logically (to me, at least), and it just feels easier to use overall.
A few days ago, I shared how I combined a Service Broker Custom Form with a vRO action to automatically generate a unique and descriptive deployment name based on user inputs. That approach works fine but while testing some other components I realized that calling that action each time a user makes a selection isn't necessarily ideal. After a bit of experimentation, I settled on what I believe to be a better solution.
My last post in this series marked the completion of the vRealize Orchestrator workflow that I use for pre-provisioning tasks, namely generating a unique sequential hostname which complies with a defined naming standard and doesn't conflict with any existing records in vSphere, Active Directory, or DNS. That takes care of many of the "back-end" tasks for a simple deployment.
This post will add in some "front-end" operations, like creating a customized VM request form in Service Broker and dynamically populating a drop-down with a list of networks available at the user-selected deployment site.
Picking up after Part Two, I now have a pretty handy vRealize Orchestrator workflow to generate unique hostnames according to a defined naming standard. It even checks against the vSphere inventory to validate the uniqueness. Now I'm going to take it a step (or two, rather) further and extend those checks against Active Directory and DNS.
Active Directory Adding an AD endpoint Remember how I used the built-in vSphere plugin to let vRO query my vCenter(s) for VMs with a specific name?
We last left off this series after I'd set up vRA, performed a test deployment off of a minimal cloud template, and then enhanced the simple template to use vRA tags to let the user specify where a VM should be provisioned. But these VMs have kind of dumb names; right now, they're just getting named after the user who requests it + a random couple of digits, courtesy of a simple naming template defined on the project's Provisioning page: I could use this naming template to almost accomplish what I need from a naming solution, but I don't like that the numbers are random rather than an sequence (I want to deploy server001 followed by server002 rather than server343 followed by server718).
I recently shared some details about my little self-contained VMware homelab as well as how I integrated {php}IPAM into vRealize Automation 8 for assigning IPs to deployed VMs. For my next trick, I'll be crafting a flexible Cloud Template and accompanying vRealize Orchestrator workflow that will help to deploy and configure virtual machines based on a vRA user's input. Buckle up, this is going to be A Ride.
Objectives Before getting into the how it would be good to start with the what - what exactly are we hoping to accomplish here?
In a previous post, I described some of the steps I took to stand up a homelab including vRealize Automation (vRA) on an Intel NUC 9. One of my initial goals for that lab was to use it for developing and testing a way for vRA to leverage phpIPAM for static IP assignments. The homelab worked brilliantly for that purpose, and those extra internal networks were a big help when it came to testing.
I recently ran into a peculiar issue after upgrading my vRealize Automation homelab to the new 8.3 release, and the error message displayed in the UI didn't give me a whole lot of information to work with: I connected to the vRA appliance to try to find the relevant log excerpt, but doing so isn't all that straightforward given the containerized nature of the services. So instead I used the vracli log-bundle command to generate a bundle of all relevant logs, and I then transferred the resulting (2.
I picked up an Intel NUC 9 Extreme kit a few months back (thanks, VMware!) and have been slowly tinkering with turning it into an extremely capable self-contained home lab environment. I'm pretty happy with where things sit right now so figured it was about time to start documenting and sharing what I've done.
Hardware (Caution: here be affiliate links)
Intel NUC 9 Extreme (NUC9i9QNX) Crucial 64GB DDR4 SO-DIMM kit (CT2K32G4SFD8266) Intel 665p 1TB NVMe SSD (SSDPEKNW010T9X1) Random 8GB USB thumbdrive I found in a drawer somewhere The NUC runs ESXi 7.
It's a good idea to take a snapshot of your virtual appliances before applying any updates, just in case. When you have multiple vCenter appliances operating in Enhanced Link Mode, though, it's important to make sure that the snapshots are in a consistent state. The vCenter vmdird service is responsible for continuously syncing data between the vCenters within a vSphere Single Sign-On (SSO) domain. Reverting to a snapshot where vmdird's knowledge of the environment dramatically differed from that of the other vCenters could cause significant problems down the road or even result in having to rebuild a vCenter from scratch.
I manage a large VMware environment spanning several individual vCenters, and I often need to run PowerCLI queries across the entire environment. I waste valuable seconds running Connect-ViServer and logging in for each and every vCenter I need to talk to. Wouldn't it be great if I could just log into all of them at once?
I can, and here's how I do it.
The Script The following Powershell script will let you define a list of vCenters to be accessed, securely store your credentials for each vCenter, log in to every vCenter with a single command, and also close the connections when they're no longer needed.
