Posts

Change DNS server on vLCM Appliance

Changing the DNS server configuration on your servers is not a task you do on a daily basis. But when you have to, it is not always a straight forward action, as I experienced when trying to change the DNS server on the VMware Life Cycle Manager Appliance.

Changing the DNS configuration under Windows is no big deal and is as easy as next, next, finish. Under linux however there are multiple ways to accomplish this task depending on the linux distribution. In most cases the easiest and common way is to edit /etc/resolv.conf and restart the service.

In this case, I am working with a virtual appliance, so I want to keep the editing on the command line to a minimum.  Virtual appliances are normally managed through a management webpage, and for the vLCM appliance this is equally so. I login to the vLCM management page and under settings I see the configured IP and DNS configuration.

Unfortunately, I can only view the configuration but can’t make any changes. So I searched the VMware knowledgebase to see what the supported method for changing DNS on Photon OS was, but got nothing. Even a Google search didn’t give any good results. This left me with no other option than to dive into the command line and edit the configuration by hand.

As mentioned earlier, most linux based distributions use resolv.conf as the configuration file for DNS. However upon opening resolv.conf I saw that Photon uses “systemd-resolved” to manage name resolving and that I shouldn’t edit this file by hand.

When working with system and DNS there a few places where you can configure settings. Let’s look at them for a second:

  • /etc/resolv.conf is a symbolic link to /run/systemd/resolve/resolv.conf and is the running configuration which must not be edited by hand.
  • /etc/systemd/network/10-eth0.network is the configuration file for the vm’s ethernet adapter. You set the IP address, subnet mask and gateway here, but can also specify search domains, NTP and DNS.
  • /usr/lib/systemd/network this folder contains the configurations of other (virtual) adapters on the system
  • /etc/systemd/resolved.conf this file contains the configuration of DNS that would normally be located in resolv.conf

I changed both 10-eth0.network and resolved.conf with the desired DNS servers and search domains, restarted the services, and for a short time everything seemed to work. But after a reboot all settings where back to the old configuration. I was rattled and searched for some other configuration files, looked through a lot of log files, but couldn’t find the culprit.

When all seemed lost, I had a bright moment. I did a search for the old DNS IP in all files (grep -rnw '/' -e '172.25.168.3'), and there between all the rows of text I saw the mentioning of “/opt/vmware/etc/vami/ovfEnv.xml”.

ovfEnv.xml is a file that is created at the deployment of an OVF and apparently is loaded every time the VM boots. I changed the configuration in the XML file and after a reboot the new configuration was still there.

VRA remove waiting on retry status

From time to time you’ll see machines in vRA that have the status “On (Reconfigure failed, waiting to retry)”. In some cases this status will never go away, because in reality the retry is already done and everything went well but the status in vRA still says “Waiting for retry”.

The problem with a “Waiting on retry” status is that you cannot edit the machine properties, as vRA thinks that the machine is still waiting for other changes to be applied.

In vRealize Automation there is no option to reset the status of a machine. The only solution to reset the machines status to “On” again, is to edit the database of vRA. You would think that editing the database is the worst option to get a status back to normal, but it is actually recommended by VMware in some cases (see https://kb.vmware.com/s/article/2114385 and https://kb.vmware.com/s/article/2141917?lang=en_US).

So we login to the vRA database server (in this case a MSSQL server) and select the vRA database. As you can see in the table view, there are many different tables containing everything that vRA is made of.

The table that contains the machines and its status is “dbo.Virtualmachine”. You can check the contents of this table by starting a new query like select * from [dbo].[virtualmachine]

To check what the current state of a machine is, you can query select CurrentTask from [dbo].[virtualmachine] where VirtualMachineName in ('VM_name').

Now as we want to reset this status to the default, we have to reset the CurrentTask field to nothing. To do this we execute the following SQL statement: update [dbo].[virtualmachine] set CurrentTask = NULL where VirtualMachineName in ('VM_name ').

When we run the previous query (select from) again, we can check the new status of the current task. And when we check in vRA we will see that the status is back to “On” and we can make modifications to the machine again.

Horizon View BLAST error in Chrome browser

We did an update of our Horizon View environment from version 7.4 to version 7.5.1. After the update we noticed something strange. Everything was working except for the BLAST client on the Chrome browser. Other browsers didn’t give errors and worked, but Chrome threw the error: “Failed to connect to the Connection Server”.

After some searching in the VMware knowledge base, I found that the error has something to do with security. The View Security document talks about Cross-Origin Resource Sharing (CORS) as the feature that handles the policies in regard to HTTP request. (https://docs.vmware.com/en/VMware-Horizon-7/7.5/horizon-security.pdf). This means that when an URL is used that is not the same as the listening domain, or when multiple domains are used, the policies can block access because the actions are considered not secure (like there could be a man in the middle attack).

In our case we have two URL’s to the Connection Servers. The first is a loadbalanced URL (http://ViewDesktop.LocalDomain) and the second is a direct URL to the Connection Server (http://HostName.LocalDomain). We noticed that the direct URL didn’t gave problems, but de loadbalanced URL did. So it seems clear that the problem must have something to do with CORS and in specific with the Chrome browser.

When we read a little bit further in the security documentation we’ll see an explanation for our Chrome problem: “Chrome extension clients set their initial Origin to their own identity. To allow connections to succeed, register the extension by adding a chromeExtension entry to the locked.properties file”.

Now, all CORS related settings are set in the file called locked.properties. You can find the file on your View Connection and Security Servers in the folder C:\Program Files\VMware\VMware View\Server\sslgateway\conf\ and if it doesn’t yet exist, you can simply create it.

So now that we know the problem in the Chrome browser seems to be coming from a security feature, how do we fix the problem? There are multiple solutions to solve this problem, which all include the locked.properties file.

  1. Disable CORS altogether. Not the most elegant solution.
  2. Set the checkOrigin property to “false”. This is probably not the option that you want. Though it works, it disables the security check. (https://kb.vmware.com/s/article/2144768)
  3. Set the balancedHost property to the URL on which you connect. This is a good option as you specify the loadbalanced address that is used by View. (https://docs.vmware.com/en/VMware-Horizon-7/7.4/horizon-installation/GUID-BFF2E726-A5EB-4105-A0EA-F3D718C5880E.html#GUID-BFF2E726-A5EB-4105-A0EA-F3D718C5880E)
  4. Set the property “chromeExtension.1=bpifadopbphhpkkcfohecfadckmpjmjd” in the locked.properties file. This is the best option for us as it is tailored to the issue that we are facing. (https://docs.vmware.com/en/VMware-Horizon-7/7.5/horizon-security/GUID-94DAC7B8-70A3-4A91-8E70-2B2591B82866.html)

After you’ve set the locked.properties file, you’ll need to reboot the server for the settings to take effect. And after a reboot you’re Chrome errors are gone.

Daisy Chaining VMware UMDS

I was wondering if one could daisy chain multiple VMware Update Manager Download Service (UMDS) appliances. The documentation doesn’t say a word about it. The only thing I found Googling this was one blog that say’s it can’t be done. But that blog was from 2014, now, 2018, let’s see…

I started with a Ubuntu 16.04 LTS server and used William Lam’s script to install the UMDS on top. It needed some more config:

(I tested this in Fusion virtual machines using vSphere 6.5 Update 1 (5969303))

Open the console of the first UMDS

sudo -i
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom
apt-get install openssh-server -y
ifconfig #This gives the IP address (Ubuntu in Fusion creates an ens33 interface)

Using a SSH (Windows: PuTTY / Mac: Termius) client, connect to the UMDS:

ssh vmninja@172.16.250.129

In the SSH session:

sudo -i
wget https://github.com/lamw/vghetto-scripts/raw/master/shell/install_umds65.sh
chmod +x install_umds65.sh
./install_umds65.sh /mnt/cdrom/umds/VMware-UMDS-6.5.0-5939545.tar.gz UMDSDB UMDS_DSN umdsuser VMware1!
/usr/local/vmware-umds/bin/vmware-umds -v
/usr/local/vmware-umds/bin/vmware-umds -G
/usr/local/vmware-umds/bin/vmware-umds -S --add-url https://vibsdepot.hpe.com/index.xml --enable-host --url-type HOST
/usr/local/vmware-umds/bin/vmware-umds -D

(In the example above I’ve added the HPE VibsDepot, to see if non-firstparty updates will get downloaded.)
It will now start downloading… It may take some time to complete, at the time of writing it was about 65 GB

Per William Lam’s suggestion for this test I used Python’s builtin webserver:

apt-get install python-minimal -y
cd /var/lib/vmware-umds
python -m SimpleHTTPServer 80

Using this as a foreground task, it shows all HTTP requests being received:

172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/HPQ/metadata-hpnmi-vmware55-bundle-2.3-6.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/csco/csco-VEM-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/__hostupdate20-consolidated-metadata-index__.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-5.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:19] "GET /hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /hostupdate/vmw/vmw-ESXi-6.5.0-metadata.zip HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/bootstrap_index.xml HTTP/1.1" 200 -
172.16.250.150 - - [18/Feb/2018 05:28:20] "GET /vaupgrade/__valm-consolidated-index__.xml HTTP/1.1" 200 -

But first I needed to build a second UMDS, mostly the same as above, except:

  1. I didn’t add the HPE VibsDepot
  2. I pointed hostupdate.vmware.com and vapp-updates.vmware.com in the hosts file to localhost, by editing the hosts file to prevent it to try to download anything directly from VMware.com’s website.
    • For some reason it is not possible to remove these entries from UMDS’s config.
    • Nor to remove the updates for older versions. I didn’t need updates for any ESXi prior to 6.5.
  3. vmninja@ubuntu:~$ cat / etc/hosts
    127.0.0.1       localhost
    127.0.1.1       ubuntu
    127.0.0.1       hostupdate.vmware.com
    127.0.0.1       vapp-updates.vmware.com
    
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    
  4. Instead of the HPE VibsDepot, I added the first UMDS as source:
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/hostupdate/__hostupdate20-consolidated-index__.xml --enable-host --url-type HOST
    /usr/local/vmware-umds/bin/vmware-umds -S --add-url http://172.16.250.129/vaupgrade/__valm-consolidated-index__.xml --enable-va --url-type VA
    
  5. Start the download task:

    /usr/local/vmware-umds/bin/vmware-umds -D
    

After downloading was complete, I added the second UMDS to vCenter Update Manager:
UpdateManagerConfig

After clicking download now, the Non VMware Patches are visible in vCenter:
PatchRepository

So it seams to be possible to daisy chain UMDS. I have no idea about the supportability of this configuration, and if using something in production, use a real webserver!

This post was published originally on my personal blog https://vmninja.wordpress.com

How to reinstall VMware Tools on a ParaVirtual VM

At a customer location I was asked to reinstall VMware Tools on machines, normally the customer would do this their selves, but after uninstall of the VMware Tools, the VM wouldn’t boot. They needed to reinstall the VMware Tools because of the issue encountered described in  VMware KB2063887
Read more

vCenter Server 5.1 with SSO 5.5

This blog describes how to install vCenter Server 5.1 with SSO (Single-Sign-On) 5.5

In my case, it’s not possible to upgrade to vCenter Server 5.5 because of the fact that the vendor doesn’t support a higher version (yet).

SSO 5.1 (and i will say it very nice) is not the best product VMware has ever made.
It has several bugs and is not very stable.
But VMware made a more stable version for 5.5, and i recommend everyone to use that version.

Ok, this is the procedure to install all needed vCenter components: Read more

/dev/sda1 has gone 255 days without being checked, check forced

/dev/sda1 has gone 255 days without being checked, check forcedOn initial boot of a freshly deployed (Ubuntu) Linux appliance on VMware vSphere 5 the machine stopped working because of an inconsistent disk. According to the machine 255 days where passed without the disk being checked.  After a file system check is forced the machine stops with an error UNEXPECTED INCONSISTENCY followed by fsck / [5001] terminated with status 4.

Read more

VMware: Recover vCenter Single Sign On (SSO) master password

VMware vSphere Web Client - vCenter Single Sign On InformationDuring the installation of the VMware vSphere Web Client I had to provide vCenter Single Sign On Information. Since no additional accounts / groups where granted SSO admin privileges (see VMware vSphere 5.1 Documentation Center)  the only account that had sufficient privileges was the default SSO admin user admin@System-Domain. The credentials of this account are provided during installation of the vCenter Single Sign On Service.

Unfortunately the password of the default SSO admin account was unknown. In this article I’ll explain how to change the password of the default SSO admin account.

Read more

Collect information about an ESX environment

This week I am creating an inventory of a customers network environment. Within the network of the customer is an VMware ESXi environment. While I am looking for an efficient way to collect information out of VMware vCenter I stumble upon a .NET tool called RV Tools from VMware vExpert Rob de Veij.

This tool gives you the possibility to collect information from your ESX  environment.

image

Read more

Snapshots present and growing, but not visible in Snapshot Manager

When using  a backup solution for your virtual environment and the database-server for this backup solution is also situated in the virtual environment, you have to face some challanges. The backup solution cannot back up its own database, while using it. Luckily there are work-a-rounds covering this particular problem. But what happens if you misconfigured this work-a-round? In this case snapshots will be taken of the database-server and will not get removed properly, resulting in snapshots on your datastore that keep growing and growing and you will not be aware of it, because the snapshot manager says there are no snapshots present.

Read more