We had an issue where we were getting hundreds of these messages in the event log every hour, and with some help we found old VIBs installed from the manufactuers custom images.  After this cleanup we only get 8-10 of these messages an hour now.  Searching through the hostd log they correspond with these entries:

2019-10-17T15:03:37.652Z info hostd[2100774] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=ee2461f0 user=:vsanmgmtd] Event 399 : User dcui@127.0.0.1 logged in as pyvmomi

It makes it seem like HA is causing this and someone in this thread seemed to say the same thing.  Is this normal behavior?  Does everyone with HA enabled get these in their logs?  This KB article says "In a default configuration of ESX Server host there is no process that repeatedly logs in as noted above in the Symptoms".  Should I just assume that by default configuration, they mean no HA?

I've spent a lot of time chasing down the sources of these messages in the last few months and I'd like to know the source of the last few I'm still seeing.  We're on ESXi 6.7 U2 and VCSA with all VMware and hardware vendor patches installed.  I don't recall seeing this in 6.5 before our upgrade, but I don't have the logs anymore to be certain.

Hi TheBobkin,

From the witness appliance, I run the following and get that result when I traceroute to the management IP of one of the nodes.

------------------------------

[root@esxi-witness01:~] traceroute 10.137.20.52

traceroute: Warning: Multiple interfaces found; using 10.137.20.40 @ vmk0

traceroute to 10.137.20.52 (10.137.20.52), 30 hops max, 40 byte packets

1  10.137.20.52 (10.137.20.52)  0.692 ms  0.195 ms  0.140 ms

------------------------------

I left the port group on the witness appliance the way it came out of the box. That is, vmk1 with vSAN enabled on it, is linked to witnessSwitch, which is connected to vmnic1. This vmnic1 is network adapter 2 on the appliance VM. Network adapter 2 here is connected to vlan60 port group on Dswitch. The vlan60 port group is tagged to vlan 60 (vsan vlan). Dswitch from this physical host has 2 uplinks to the Dswitch from the host. Those two uplinks are trunk ports to a network switch. The network switch has vlan60 on it which trunks to a 10G switch upstream. This 10G switch has the two vsan nodes connected to it with 2 uplinks each. From the 10G switch to those uplinks is a trunk port, entering into the 2 nodes into the same Dswitch as before, where the corresponding vsan vmks are located on the same vlan60 port Dswitch port group as before.

This is the same exact setup as vlan20 port group which is carrying management traffic between the 2 nodes and the witness.

Based on this, am I missing a step after deploying the witness appliance VM? Even though vlan60 gets tagged on the VM settings of the witness so I still need to set vlan ID to 60 on the internal witness Pg it makes? This seems redundant but is it needed for some reason? I have seen articles on static routes, but I don't think I need those right because this is all on the same site?

-------------------------------

If I go the route of moving witness traffic to vmk0 on witness appliance with management so what do I need to change or do on the witness appliance? Anything on the hosts then?

Thanks for all your input here,

Chris

Yesterday I reported 2 further  "Confirmed Kills by unresolvable VMFS 6 locks"  in Twitter and some folks asked for details.

The first cases I noticed must be about 2 years ago if I remember correctly.

Once I saw the first ones I started to see them more regularly.

Since about a year I have a defintion for this cases and started to count them for my own use.

Some weeks ago I noticed that VMFS 6 used by ESXi 6.7 has at least one new locktype that I never seen before.

VMFS 5 and earlier versions have 2 magic values for locks in the heartbeat section ( part of the hidden system file .vh.sf

When I noticed that new type I tried to get in contact with VMware Inc. via Twitter to ask wether this is a new feature, a bug or just a form of corruption.

One suggestion was to ask in VMTN - funny idea ...

Looks like VMware Inc. is satisfied with the quality-level of VMFS related support in VMTN - which is a bit strange as I have seen no VMware employee with VMFS knowhow visiting VMTN in the last years.

Anyway - you all know how a locked file behaves. You cant copy them, you cant launch a VM that uses them and most important: you cant query the mapping of the file with vmkfstools -p 0.

To make it into the "Confirmed Kills" list the locked file must fail some more tests:

- hexdump -C file fails

- mount vmfs datastore readonly via sshfs and try to use ddrescue against the locked file must fail

- vmfs-fuse must fail

- then I run some more advanced manual tests that try to get the mapping of the file - no use to list them here - they must fail as well.

For a layman there is one test that is easy to use that I also consider to be conclusive:

mount the datastore readonly via sshfs from Linux and then use the best VMFS reading tool on the market and try to open the file or just list the details - if this fails as well I consider the vmdk as a "Confirmed Kill"

FAQ:

What is the best VMFS reading tool available on the market (state oktober 2019) :

UFSexplorer - Linux LiveCD - get the latest version.

Is it possible to extract data from a "Confirmed Kill" ?

- as always with vmdks without usable VMFS-metadata:

thin provisioned = no way,

lazy zeroed = why would some one use lazy zeroed ???,

eager-zeroed = if allocated in one piece data extraction is possible - if the vmdk uses several pieces extraction is still possible but you need highly skilled personnel to even have a chance.

vmfs-sparse = advanced but possible (I managed it in a few rare cases

sesparse = oh dear - from a recovery point of view sesparse is a nightmare - I have no success so far, as far as I know UFSexplorer cant read latest sesparse version either

Do you have a "Confirmed Kills" list for VMFS 5 or 3 ?

Nope - with those version I  always find  workarounds - so no problem here.

Do you have a therapy yet ?

Yes - I had success in a few cases. I did a complete reset of the heartbeat section and could fix a handful of cases. BUT those were standalone hosts only. I have not seen enough cases with clustered VMFS datastores to suggest this "brute force" approach at the moment.

Do you work on a minimal invasive therapy ?

Of course I do. Progress is slow as the research requires a Lab-environment with at least 2 ESXi, a shared storage and at least one single working VM. Currently I only can run one or two nested ESXis on top of Workstation 12. I cant use running Test VMs at all and so my lab helps a little bit with basic research - but it is not good enough to develope a safe therapy.

Can VMware-support deal with this problem ?

I dont know.

Can Ontrack help ?

I dont know but I assume they can help.

Why is this an anoying problem ?

I expect that I will find a therapy sooner or later that just requires one or two very small hexedits.

It is possible that VMware Inc. already is aware of the problem and maybe they already have a working and safe therapy.

It is possible that a VMFS-developer already has  a working therapy and that it is flagged as "unsupported" and so does not need to be documented at all.

Did you open a support ticket with VMware-support ?

Good joke - I am not a paying cuastomer

Did you report the issue in the moderator-forum ?

That forum appears to be abandoned at the moment.

What else did you try ?

I asked for detailed documentation for the .vh.sf with VMTN posts and Twitter.

Did you try emails to a VMFS-engineer ?

No - I requested an email list with engineers, product manager and so on in the moderator-forum but I dont think that the moderators office has been used last couple of months.

Where can I report future "Confirmed Kills" ?

Just add a note here or call me via skype.

Nothing about the voma-tool ?

Correct - I also did not mention snake-oil ...

What about reward points ?

As always: 10 points for 4 random links - make it double if they rhyme

Ulli

Hi,

The problem with SID is not windows updates, but duplication within the Active Directory Domain. I usually create a virtual machine, run all the updates and then transform it into a template. When I have to create a new server I "create a new virtual machine for this template" and then I apply a "customization" which also performs the SID change, so in 10/15 minutes I have a Servers ready and updated. When the Microsoft updates (the second Tuesday of each month) come out, I reconvert the template into vm, apply the updates and reconvert it into a template. Ready to create new updated Servers.

Having said that I suggest you to change the SID immediately and not wait to update the vm leaving the duplicate SID.

ARomeo

Just received this today may be of interest. The link to the KB is VMware Knowledge Base 

Dear Customer,

Customers that have installed vRealize Operations v6.x and above may be impacted by the internal SSL Gemfire certificate expiration on November 19, 2019. The internal certificates for vRealize Operations Manager are set to expire five years after its initial installation. For example, if vRealize Operations 6.0.0 was installed on November 19, 2014, and incrementally upgraded to later versions, the internal SSL certificate will expire on November 19, 2019.

There is a required PAK file update that has been released for customers running vRealize Operations v6.x and above that renews the certificate to prevent it from going into system inoperability.

Please refer to this KB article for more information on how to check the expiration date and instructions to upgrade your vRealize Operations internal certificate.

  - The Product Team

Had the same issue on a windows 10 guest OS. Reinstalling VMTools then went to device manager > mouse and pointer devices > right click on hid-compliant mouse and update driver then search automatically for updated driver resolved the issue.

The ResourceElement is the simplest one and it has no external dependencies. The only thing to be careful of would be how many entries. I've stored ~100,000 generated MAC addresses in a JSON formatted text element with no issues previously. We used this approach to build a MAC address assignment pool for server provisioning with an integration to DHCP during server provisioning for a client. Depending on the amount of data each retry record requires this could work nicely (just make sure you use workflow Locking for all reads & writes!)

Database would obviously be fine and would permit large amounts of data (say Request & Response attempt logs for each retry) as well as the retry table your primary requirement indicates. The external system requirement is the only piece to consider but it might be warranted if you have the need.

I've never used the MessageQueue plugin but my understanding is that it is trigger/event based which might make the delay period part of this difficult. Even if you went this way and just put a message back on the queue if it's retry date was not met your system would just spend all it's cycles firing messages to itself (if I understand how this works correctly at least)

You can submit an idea on the PowerCLI Ideas website.

Powershell Core on Photon OS does not built-in register PSGallery. Hence cmdlets install-module, find-module, etc. are not working.

On Windows there were (and still are) many security related issues with Packagemanagement and Powershellget. Imho the best move became to use the x86 cmdline nuget.exe to register source provider.

I put some findings together on GitHub - dcasota/photonos-scripts: This repo contains several scripts to automate the installation of Photon OS on Azure… You may use the Dockerfile. Actually It contains Pwsh7 (Preview), though.

Update:

Simply pull and run:

• docker pull vmware/powerclicore:ubuntu16.04
• docker run -it vmware/powerclicore:ubuntu16.04

PS C:\scripts> Register-PSRepository -Name "PSGallery" –SourceLocation "https://www.powershellgallery.com/api/v2/" -InstallationPolicy Trusted

WARNING: Unable to download from URI 'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409' to ''.

WARNING: Unable to download the list of available providers. Check your internet connection.

PackageManagement\Install-PackageProvider : No match was found for the specified search criteria for the provider 'NuGet'.

The package provider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7405 char:21

+ ...     $null = PackageManagement\Install-PackageProvider -Name $script:N ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvider:InstallPackageProvider) [Install-PackageP

   rovider], Exception

    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackageProvider

PackageManagement\Import-PackageProvider : No match was found for the specified search criteria and provider name 'NuGet'.

Try 'Get-PackageProvider -ListAvailable' to see if the provider exists on the system.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7411 char:21

+ ...     $null = PackageManagement\Import-PackageProvider -Name $script:Nu ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidData: (NuGet:String) [Import-PackageProvider], Exception

    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.ImportPackageProvider

WARNING: Unable to download from URI 'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409' to ''.

WARNING: Unable to download the list of available providers. Check your internet connection.

PackageManagement\Get-PackageProvider : Unable to find package provider 'NuGet'. It may not be imported yet. Try

'Get-PackageProvider -ListAvailable'.

At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:7415 char:30

+ ... tProvider = PackageManagement\Get-PackageProvider -Name $script:NuGet ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ObjectNotFound: (Microsoft.Power...PackageProvider:GetPackageProvider) [Get-PackageProvider],

    Exception

    + FullyQualifiedErrorId : UnknownProviderFromActivatedList,Microsoft.PowerShell.PackageManagement.Cmdlets.GetPackagePro

   vider

Register-PSRepository : NuGet provider is required to interact with NuGet-based repositories. Please ensure that

'2.8.5.201' or newer version of NuGet provider is installed.

At line:1 char:1

+ Register-PSRepository -Name "PSGallery" –SourceLocation "https://www. ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (:) [Register-PSRepository], InvalidOperationException

    + FullyQualifiedErrorId : CouldNotInstallNuGetProvider,Register-PSRepository

PS C:\scripts>

I give up, you seem to be in a rather special environment, which I can't seem to replicate.
In any case, your original question about HW version 15 is resolved when you upgrade your PowerCLI installation.

We've just updated VRLI in our lab environment from v4.8 to v8.0 but encountered a couple of issues along the way. Am documenting them here for the benefit of others and any input from VMware.

Failure to migrate network adapter gateway configuration

The gateway for the network adapter wasn't correctly migrated during the conversion to Photon OS. This resulted in the upgrade appearing to hang as the appliance was unable to communicate with any system outside of its IP subnet. The issue is alluded to in the Release Notes but they're somewhat vague and lack instructions on how to check if you're affected and any workaround steps:

• When using static IPs (as opposed to IPs obtained via DHCP), ensure that the corresponding SLES network configuration file contains an entry for gateway on all the cluster nodes, before upgrading to vRealize Log Insight 8.0.

The issue is caused by the /etc/sysconfig/network/ifcfg-eth0 file "missing" a GATEWAY='w.x.y.z.' statement. I say "missing" as this doesn't appear to be where SLES typically saves static routes. Those are contained in the /etc/sysconfig/network/routes file, which is what the upgrade process should presumably be looking at. In our case it contained a single line specifying the default route.

As a workaround, you can SSH to the box and add the line the upgrade scripts are expecting before performing the upgrade.

Regression in handling of STARTTLS for SMTP

After upgrading we had to modify the SMTP configuration to not use STARTTLS. This was working fine previously but now sending a test email with STARTTLS enabled fails with the error:

No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

The issue appears to be something in the Java configuration of VRLI. In our configuration we're sending to Office 365 via "Direct Send". Keen to get any input from VMware on this one.

As a workaround you can disable STARTTLS, however, you may wish to do so before starting the upgrade so you don't potentially lose any emails before you can apply the workaround post-upgrade.

In reference to the interoperability matrix

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&455=&237= Workstation Pro is not supported on Photon OS (as host).

No need to switch between different distros and as shellbased administering is good enough, the combo may become a niche option in dev/test and IoT pipelines.

Hi,

HPE support says it's because my SSD is not HPE branded and they don't want to investigate further.

Pull the drives and install vSphere on a USB stick...

The same problem will occur and they cannot use that line anymore.

--

Wil

Hi,

If you have enough RAM and a modern processor then yes it works remarkably well.

edit: having more cores on your physical CPU helps too.. at least 4 physical cores preferably more.

--

Wil

Fairly new install of Linux Mint, 19.1

Brand new install of VMware Workstation 15 Player

Install went off without any hitch, player loads fine, but I'm unable to load any VMs.  It appears to create the VM but the program closes out/crashes whenever I try to load a VM.  If I load from CLI I get initial error of:

I/O warning : failed to load external entity "/etc/vmware/hostd/proxy.xml"

When I try to open a VM I get the following string of warnings before it closes

(vmplayer:5950): Gtk-WARNING **: 21:10:37.995: Inserting action group 'Base' into UI manager which already has a group with this name

 

(vmplayer:5950): Gtk-WARNING **: 21:10:37.996: Inserting action group 'WindowActions' into UI manager which already has a group with this name

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0: missing action removable-device-0

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1: missing action removable-device-1

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0-choose-image: missing action removable-device-0-choose-image

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0-edit: missing action removable-device-0-edit

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-connect: missing action removable-device-1-connect

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-bridged: missing action removable-device-1-nic-bridged

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-nat: missing action removable-device-1-nic-nat

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-host-only: missing action removable-device-1-nic-host-only

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-edit: missing action removable-device-1-edit

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0: missing action removable-device-0

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1: missing action removable-device-1

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0-choose-image: missing action removable-device-0-choose-image

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-0-edit: missing action removable-device-0-edit

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-connect: missing action removable-device-1-connect

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-bridged: missing action removable-device-1-nic-bridged

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-nat: missing action removable-device-1-nic-nat

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-nic-host-only: missing action removable-device-1-nic-host-only

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.326: removable-device-1-edit: missing action removable-device-1-edit

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-0: missing action removable-device-0

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1: missing action removable-device-1

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-0-choose-image: missing action removable-device-0-choose-image

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-0-edit: missing action removable-device-0-edit

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1-connect: missing action removable-device-1-connect

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1-nic-bridged: missing action removable-device-1-nic-bridged

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1-nic-nat: missing action removable-device-1-nic-nat

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1-nic-host-only: missing action removable-device-1-nic-host-only

 

(vmplayer:5950): Gtk-WARNING **: 21:10:38.327: removable-device-1-edit: missing action removable-device-1-edit

Googling has really not brought up anything other than the issue where it doesn't save the previously open VMs.  At a loss as to where to go from here.  Any suggestions/assistance would be helpful.

Ehh never mind, I'm an idiot.  I didn't realize I had NEVER virtualized on this computer so my BIOS wasn't set.  Got it working.

Wow.

I did not know that!

May I assume the same if we use that internal SD Card slot inside some of the G6/G7s?

Didn't many ProLiants also have internal USB female sockets?

I know there are a lot of freenas folks out there that would boot ProLiants by ESXi on flash only, and then 10Gbe the rest.

The swap "correct solution" did not help in my case (also swap was alrerady assigned to a datastore).  Downloading the tools .vib and manually installing, as originally suggested by FirstS0u did the trick.  Thank you so much!

That's correct, Daniel.