Quantcast
Channel: Ken Umemoto's vReality
Viewing all 77 articles
Browse latest View live
↧

vSphere 6 - SSL Certificates - Overview and Best Practices

April 25, 2018, 5:14 pm
$
0
0
As corporate security becomes a higher priority, I just wanted to give a quick rundown on SSL Certificates in vSphere 6 since it has changed drastically from previous versions.

The VMware Certificate Service is part of the Platform Services Controller (PSC)

Key Terms:
VMWare Certificate Authority (VMCA) – Certificate authority for vSphere components only.  A single point of contact for vSphere Certificate needs.  Issues certificates for VMware solution users, machine certificates for machines on which services are running and ESXi host certificates.  Operates in the PSC.  Certificates are managed by the certificate-manager utility. 

VMware Endpoint Certificate Store (VECS) – Serves as a local repository for certificates, private keys and other certificate information.  Runs in vCenter Server Node. 

Types Of Certificates Used by vSphere
  •  Machine Certificates- For Secure connections.  This is what causes the web browser certificate warning if the certificate used is self signed. (ex. vSphere Web Client - vCenter server and external PSC have them)
  • Solution user certificates - authentication of services to vCenter SSO. (ex vcenter service (vpxd))
  •  ESXi certificates – provisioned when the host is added to vCenter. Stored locally on ESXi host.
Certificate Deployment Types:
  • VMCA Default – By default, the VMCA uses a self-signed root certificate.  The VMCA is then the CA for all VMware components.
  • VMCA Enterprise -  The VMCA is used as a subordinate/Intermediate CA and is issued a subordinate CA signing certificate.  It can now issue certificates that trust up to the enterprise CA’s root certificate.  Notaccepted by most security groups, since this poses a security risk. 
  • Custom Certificates – The VMCA is bypassed. Need to issue a enterprise/3rd party cert for every component. Must replace each certificate explicitly. Administrative nightmare! 
  • Hybrid - The VMCA supplies some of the certificates, but also uses custom certificates for other parts of the VMware infrastructure.  As of the time of this writing, this is the RECOMMENDED approach.
Hybrid Deployment Details:  
In a vast majority of environments, the following hybrid deployment is the best fit.  

Trusted Certificates are used for the Machine Certificates of the vCenter server and external PSC.  The management interfaces are using a 3rd party/Corporate trusted CA.  These are the most important certificates and is the only user-exposed certificates.

VMCA certificates are the used for the Solution user and ESXi certificates.

Added bonus, no more "Not Secure" warnings in your browser!

VMware KB regarding replacing machine certificates:

Fantastic VMWare Walkthrough for SSL cert replacement on your vCenter server and External PSC:

↧
Search

ESXi 6 - How to Unlock Your SSH Account.

May 15, 2018, 10:17 pm
$
0
0
ESXi Account lockout info:
1.  Accounts are locked after 10 failed attempts through SSH and the vSphere Web Services SDK.
2. The Direct Console Interface (DCUI) and ESXi shell do not support the account lockout feature.
3. The account automatically unlocks after 120 seconds by default.
4. ESXi leverages the Linux Pluggable Authentication Modules (PAM)

If you are unable to wait for the account to unlock, you can reset the account by doing the following:
1. Console into your server by using your DRAC/iLO/UCS Manager etc.
2. log in as root, and run the following command to unlock the account.  In the example below, you can see there was 11 failed attempts:

pam_tally2 --user root --reset

↧
↧

Deploy OVF Template - The following manifest file entry (line 1) is invalid: SHA256

May 15, 2018, 10:44 pm
$
0
0
AKA - Another reason to stop using the vSphere C# Client....

I created an OVF template from my vSphere 6.5 environment.  When trying to import the template using the vSphere C# Client, I received the following error:

The following manifest file entry (line 1) is invalid: SHA256


vSphere 6.5 started using SHA256 as the default hashing algorithm when exporting OVF templates.  Unfortunately, the vSphere Fat/C# Client only supports SHA1.

There a several ways to resolve this issue:
Option 1- Use the Web or HTML 5 client to import the OVF.  Both support SHA256.

Option 2. Use the OVFTool to convert the Cryptographic Hash Algorithm from SHA256 to SHA1.  This free tool can be downloaded here:
https://www.vmware.com/support/developer/ovf/

Option 3. (Not recommended) If you trust the source of the OVA, you can delete the optional .mf file (manifest file) and just use the .ovf and .vmdk files to import the VM..  The .mf file contains the SHA256 info





↧

VMWare vCSA 6 - Failed to start File System Check on /dev/dis...

May 16, 2018, 10:25 am
$
0
0
AKA - The vCSA is FRAGILE...

Errors encountered during this process:
Failed to start File System Check on /dev/dis...
Failed to start update UTMP about System RunLevel Changes.
Failed to start Network Service.

We recently had a quick "blip" on one of our storage arrays.   All the Windows Servers had no disruption of service or came back up without incident.

This was not the case with our vCSA and external PSC.  BOTH servers were not functional.   So, it doesn't appear to be a one-off or fluke.  The appliances were running version 6.5.0.15000, the March 2018 release.

Both servers showed "Detected aborted journal"  and "journal has aborted" errors on the console.  I started trouble shooting with the external PSC.

Upon restart, I received the following error, and the server entered Emergency Mode:

Failed to start File System Check on /dev/dis...


Log in and run the following commands to determine the device which is causing the error (Both were /dev/sda3 in my case):

/bin/sh
/bin/mount
blkid

Match the UUID in the error message with the PARTUUID in the output.  In the example below, we see it matches up with /dev/sda3.


Run the following command which runs a check on ext2, 3 and 4 File Systems. "-y" answers "yes" to all the questions. (Super handy)

e2fsck -y /dev/sda3



After the file system check has completed, restart the appliance.

This resolved the issue with the the external PSC.  Cool, just repeat the process on the vCSA right?  Not so fast....

I had the following additional errors with the vCSA after running the file check on /dev/sda3.

Failed to start update UTMP about System RunLevel Changes.
Failed to start Network Service.



Running the following command to view the contents of the systemd journal.  This pointed me to log_vg-log

journalctl -xb



Run a file system check against log_vg-log by running the following:

fsck -y /dev/mapper/log_vg-log



Reboot the server after the fsck has completed.  After coming back up, the vCenter services started successfully and I was able to log into the vCSA.
↧

Ecoboost Mustang - How to Prevent an Ecoboom - Lowside Fuel Pressure Sender Replacement

June 15, 2018, 11:10 am
$
0
0
Adam Brunson had posted on social media that he had several customers with Low Side Fuel Pressure Sensor Failures.  This caused a lean condition leading to a catastrophic failure of the motor (EcoBoom).  Apparently, there is no warning or fault code thrown when the sensor fails...

As a preventative measure, I replaced the sensor.   The sensor was $23.58 from Tasca Parts and only takes 10-15 minutes to replace.  Fortunately, the sensor is easily accessible at the top/rear of the engine compartment.

The item ordered was BU5Z-9F972-B.  However, the actual part number on the sensor is BU5A-9F972-CA. (New sensor on bottom).   According to some posts, the 2018 Mustangs still have the old sensor.

It appears the part has been revised by Ford.  The revised sensor has an additional hole for atmospheric pressure.

***Perform the following fix at your own risk.  You are working with fuel.  I am not responsible for any damage or injury while performing the steps below***

I let the car sit overnight prior to replacing the sensor.  You will need a 12mm and 24mm wrench to perform this job.  Press the tab marked in red and removed the sensor. 

Place a rag under the sensor and use the 12mm and 24mm wrench to remove the sensor.  Use extreme care when handling the metal on the fuel line.  It appears to be soft.  Since the fuel line is under pressure, remove the sensor SLOWLY.  I had a slight "mist" of gas emitted when removing the sensor.

Replace the sensor and reattached the connector.  Wipe up any gas that may have spilled.  Start up the car to confirm there are no leaks.  I used a mirror to view the back side.

It may be a placebo effect, but I seem to have a smoother idle and acceleration after the swap.  Hope this helps!

Side Notes - Additional things I do:
UPR catch can installed.
Only top tier gas of the highest octane used.
I change my oil when the maintenance minder is at 55% and 10% with synthetic motor oil. 
I do not accelerate aggressively in 6th gear at low RPMs.   Low Speed Pre-Ignition (LSPI)


↧
↧

VMware vCSA VAMI :5480 - Certificate Error - Not Secure - You cannot visit right now because the website uses HSTS.

July 10, 2018, 12:23 pm
$
0
0
I recently replaced the self signed cert on our vCSA with one generated from a proper CA server.  The Web Client and vSphere (HTML5) client showed the nice green Secure padlock.

However, when I tried to access the VMware Appliance Management Interface (VAMI), I received a Not Secure prompt.  I was unable to proceed to the site. 



Advanced details showed the following;


MyServer.MyDomain.com normally uses encryption to protect your information. When Google Chrome tried to connect to MyServer.MyDomain.com this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be MyServer.MyDomain.com, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.
You cannot visit MyServer.MyDomain.com right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

Well, there appears to be a bug where with the VMware vCSA 6.X.  After applying a new vCSA certificate, the VMWare Appliance Management Interface (VAMI) does not display the new certificate.

To resolve this issue for a vCSA running 6.5,  perform the following:

1. Copy the CA cert to the following directory : /etc/applmgmt/appliance/ca.crt

By default, the ca.crt file does not exist in this directory.  FYI, Applmgmt is the VMware Appliance Management Service.



2. Using VI, open the following file: /opt/vmware/etc/lighttpd/lighttpd.conf

3. Add the following line to the file: ssl.ca-file="/etc/applmgmt/appliance/ca.crt"


3. Restart the VAMI Service by running: /etc/init.d/vami-lighttp restart



Enjoy the nice Green Secure lock! 

↧

Unable to access file since it is locked - An error occurred while consolidating disks: msg.fileio.lock.

July 18, 2018, 5:08 pm
$
0
0
This is an oldie but a goodie.  

I was recently asked by the backup team to look into an error they were seeing in NetBackup.  The error was that NetBackup was unable to consolidate a virtual machine's disks.



Just right click on the VM --> Snapshot --> Consolidate.  Done!  Not so fast, this time I received the following error:

An error occurred while consolidating disks: msg.fileio.lock.


Unfortunately,  the usual create/delete snapshot and vMotion of VM did not work.

To resolve this issue perform the following:

1. SSH or Console into the ESXi host.
2. View the vmware.log file of the offending VM and look for the locked file:

ex.  /vmfs/volumes/offendingVM/offendingVM-dir/vmware.log 



3. Run the vmkfstools -D command against the locked .vmdk to determine the MAC address of the ESXi host which has the lock.  The MAC address of the ESXi host which has locked the file is circled in RED.


4. Log into the vCenter server using you're favorite client, then look for the ESXi host which has the NIC matching the MAC above.

5. Place the host locking the vmdk in Maintenance Mode.  Then restart the hostd service:

/etc/init.d/hostd restart


6.  Exit Maintenance Mode.  I was then able to successfully perform the Consolidate function.  As an additional test, I created and deleted a test snapshot.

↧

vSphere 7 Lab on a Dell Poweredge R820

April 5, 2020, 3:16 pm
$
0
0
With the Shelter in Place Order, I took the opportunity to fire up the ol' Home Lab.   Nothing fancy,  I affectionately call it my "Ghetto Lab"



The timing is perfect,  I'm stuck at home and vSphere 7 just went GA this week.  I'm taking this opportunity to do some testing and play with the latest release of vSphere from VMware. 

The core of my lab consists of a pair of Dell PowerEdge R820s.   Sadly, this model hasn't been officially supported by VMware since ESXi 6.5.  Fortunately for me, it runs fine in my lab for testing!  Below are the server specs for those who would also like to test on older hardware.

Server Specs:
Dell PowerEdge R820
Intel Xeon E5 - 4620
Broadcom PERC H710
Qlogic BCM57800 1GB Ethernet Adapter

Stay positive, use this time to learn and better yourself.  Happy Testing!



↧
Search

Repairing Mego Buck Rogers Action Figures

May 17, 2020, 4:46 pm
$
0
0

During the Shelter In Place, I've been cleaning up the house and going through my childhood "Treasures".  

A bit of background - After Star Wars, there were a bunch of sci fi TV shows and movies like Alien, Battlestar Galactica, The Black Hole and off course Buck Rogers.

The "Buck Rogers in the 25th Century" TV Series ran for 2 seasons starting in 1979.  As a 7 year old I absolutely loved this show.  I totally had a crush on Erin Gray who played colonel Wilma Deering on the show.

All my Star Wars action figures were still intact, but my Buck Rogers action figures didn't fare as well.  It appears that over time, the rubber band holding the figure together begins to dry rot and snap. 

It's a quick fix,  just an inexpensive O-Ring and a single Philips head screw. 

Twiki used a #7 O-Ring
O.D    1/2"
I.D.     3/8"
Wall    1/16"

Buck Rogers used a #11 O-Ring
O.D    3.4"
I.D.     9/16"
Wall    3/32"



↧
↧

vCSA - 503 Service Unavailable - Failed to connect to endpoint

June 26, 2020, 6:52 pm
$
0
0
I was recently asked to look into a vCSA issue. The user was unable to log in using the web client. The vCSA was subsequently rebooted and it then produced the following error: “503 Service Unavailable”


503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb7d00200a0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe

Upon running “service-control –status” in the vCSA, it was determined that  the vapi-endpoint service was in a stopped state.


It tuns out The Security Token Service (STS) Certificate had expired. If your vCSA was deployed using version 6.5 U2 or later, the STS cert may only be good for 2 years.


Sadly, there is no warning or easy way to determine when the STS Cert will expire using the HTML 5 client.


Resolution:
Part 1:
VMware has created a script to resolve this issue (fixsts.sh).  It can be found here:

https://kb.vmware.com/s/article/76719


1. Create a snapshots of all vCSAs (and external PSCs, if applicable).
2. Copy the .sh script to the /tmp dir of the appliance running the PSC role.
3. Make the script executable by running “chmod +x fixsts.sh”
4. Execute the file “./fixsts.sh”
5. Restart the appliance and confirm functionality.
6. The script only needs to be run once per SSO Domain.

Part 2:
Upon restart, the Web client was then throwing an Error 400. The user certificates needed to be replaced as well.

1. Launch the Certificate Manager utility: /usr/lib/vmware-vmca/bin/certificate-manager

2. Select Option 6.

3. The default options were taken except for the following:

Enter proper value for 'IPAddress' [optional] : IPADDRESS
Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : VCFQDN
Enter proper value for VMCA 'Name' : VCSHORTNAME

4. The Certificate Replacement process was stopped at 85% by hitting CTRL+C, and the services were manually started by running “Service-control –start –all”.

The process was stopped at 85% since the certificate replacement was completed. At 85%,, the vCSA was just waiting for all the processes to restart. If any of the services fail to start, the certificate replacement will be rolled back.

5. This process can be done at any time. It only affects the vCSA the process is performed on.


Checking the Expiration Date of the STS Certificate: 

Method 1 - Flash Client - login using administrator@vsphere.local using the:


Method 2
1. Download the “checkSTS.py” script from the following location: https://kb.vmware.com/s/article/79248
2. Copy the script the /tmp dir.
3. Run the Python Script “python checksts.ph”
↧

Deploy OVF Template - A Required Disk Image Was Missing

July 1, 2020, 7:10 pm
$
0
0
I was asked to look into an OVF import issue.  He was receiving the following error:  "A required disk image was missing."


I looked in the .ovf file and confirmed that the names of the .vmdk and .ISO files referenced matched the files selected during the deployment process.  (I've had instances where the files were renamed.  Why, I don't know).  Everything checked out.  

I was using the HTML 5 client as recommended.  On a whim, I tested the process using the old school Flash client.  The same files and process was used.  Using the Flash Client, the job completed successfully.

Unexpected, but it worked. 

↧

vCSA 7 Update Failed - "Error in method invocation [Errno 2] No such file or directory" and "Update installation is in progress"

August 9, 2020, 10:59 am
$
0
0

Well, that didn't go as planned...

I had a VMWare vCSA that was running version 7.0 GA that was in need of an update.  The pre-update check ran successfully, the update was staged (5GB) and I began the installation.  

After an hour, the progress was still 0%, I clicked "Cancel Installation" and received the following error:

Error in method invocation [Errno 2] No such file or directory


I then performed a graceful restart of the appliance and received the following from the VAMI when logging in using root creds:

"Update installation is in progress"

To resolve this issue I SSH'd into the appliance and performed the following:

1. Create a copy of the software_update_state.conf file.  The file is located in the /etc/applmgmt/appliance/ directory

cp software_update_state.conf software_update_state_backup.conf

2. Stop the applmgmt service.

service-control --stop applmgmt

3. Delete the software_update_state.conf file.

rm software_update_state.conf

4. Confirm the applmgmt service starts successfully.

service-control --start applmgmt

I was then able to successfully log into the VAMI and complete the install. 

5. I un-staged the update. (Neat, reboot not required)

6. This time, I selected "Stage and Install"

7. The update installed successfully and the vCSA is now running the latest release. 

This appears to be an issue when a vCSA update is unsuccessful.  Deleting the original software update state config file and having the vCSA create a fresh one looks to resolve the issue.

↧

Azure Migrate - How to reset the Azure Migrate Appliance

August 12, 2020, 4:23 pm
$
0
0

One of the limitations of Azure Migrate is that each Azure Migrate Appliance can only be associated with one Migrate project.  

After successfully migrating a VM from on-prem to Azure, here’s a quick and easy way to reset the appliance.

 

1. Open the following JSON file:


C:\ProgramData\Microsoft Azure\Config\appliance.json


Update "IsApplianceRegistered" from "True" to “False” and save the JSON file.

 

 

2. Restart the server.  Upon restart, the appliance has now been reset.


3.  That's it!  The appliance is now ready to be used for your next migration project. 

↧
↧

Upgrading my Dell PowerEdge R820 with an NVMe SSD!

September 6, 2020, 4:07 pm
$
0
0

 This was the best $14 I've spent on my Homelab.   

The processor and memory specs on my Dell PowerEdge R820 running VMWare ESXi 7 are completely fine. Unfortunately, the storage performance was lacking... So, I used a generic $14 PCIe card that allowed me to use an existing M.2 SSD drive as a VMWare datastore. IOPs went from 1K to 54K and power consumption went down by 40W.

Quick video on the process:

https://www.youtube.com/watch?v=fG4WQbLUhuE

↧

vCSA Update Abruptly Stops at "Converting Data 80%"

March 17, 2021, 6:02 pm
$
0
0

While updating one of my vCSAs from Update 3j to 3l, the update process would abruptly stop at “Converting Data 80%”.  


This status window would just disappear, no warning or failure messages.  This occurred twice...  Surprisingly, the partner vCSA in enhanced linked mode was built at the same time with the same version upgraded without issue.

The following 2 logs should be used to determine if the update was successfully completed:

1. /var/log/vmware/applmgmt/update_microservice.log

2021-03-17 18:13:09,367 - update_script_day0_patching - DEBUG - Patching completed successfully

2021-03-17 18:13:09,368 - vmware.update.extensions - DEBUG - The component script returned 'None'

2021-03-17 18:13:09,374 - vmware.appliance.update.update_b2b - DEBUG - update script result file '{\n    "error": null,\n    "info": [],\n    "progress": 100,\n    "progress_message": null,\n    "status": "success",\n    "warning": []\n}

'

2021-03-17 18:13:09,374 - vmware.appliance.update.update_functions - DEBUG - /var/vmware/applmgmt/patch-history directory created successfully

2021-03-17 18:13:09,388 - vmware.appliance.update.update_b2b - INFO - Setting appliance version 6.7.0.46000 build 17138064

2021-03-17 18:13:09,391 - vmware.appliance.update.update_b2b - DEBUG - Appliance version is updated for login banner

2021-03-17 18:13:09,394 - vmware.appliance.update.update_functions - DEBUG - Running /usr/bin/banner_ctrl reset

2021-03-17 18:13:16,146 - /usr/lib/applmgmt/update/py/vmware/appliance/update/task_manager.py - DEBUG - UpdateTask: status=SUCCEEDED, progress=100, message={'id': 'com.vmware.appliance.install_complete', 'default_message': 'Installation

complete', 'args': []}

2021-03-17 18:13:16,153 - vmware.appliance.update.update_b2b - DEBUG - Installation completed successfully!

2021-03-17 18:13:16,153 - vmware.appliance.update.update_functions - DEBUG - Running reboot

2021-03-17 18:13:21,928 - __main__ - DEBUG - Event callback succeeded

2. /var/log/vmware/applmgmt/upgrade_hook_PatchHook

{

    "info": [],

    "progress_message": null,

    "warning": [],

    "status":"success",

    "error": null,

    "progress": 100

}

This appears to be a "cosmetic" issue and the upgrade completed successfully.  

↧
Search

Microsoft Azure Storage Explorer - "Error: self signed certificate in certificate chain"

December 14, 2022, 3:07 pm
$
0
0

We've been moving resource around in Azure and one of the tools we've been using is Microsoft Azure Storage Explorer.  One question I get asked frequently is how to resolve the following Certificate error when logging in:

{

  "message": "\"{\\n  \\\"name\\\": \\\"Error\\\",\\n  \\\"message\\\": \\\"self signed certificate in certificate chain\\\",\\n  \\\"stack\\\": \\\"Error: self signed certificate in certificate chain\\\\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1530:34)\\\\n    at TLSSocket.emit (node:events:390:28)\\\\n    at TLSSocket._finishInit (node:_tls_wrap:944:8)\\\\n    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:725:12)\\\",\\n  \\\"code\\\": \\\"SELF_SIGNED_CERT_IN_CHAIN\\\"\\n}\""

}


Fortunately, the solution is extremely simple.  Although the option is in "Preview", its been working fantastic.   Go to Edit -->  Settings and change the Proxy Configuration to "Use system proxy (preview)".



Thats it.  No more login errors and I'm able to see all my resources.   Hope this helps!


↧

vCenter - You have reached the maximum number of connected consoles: X. Please contact your administrator

July 11, 2023, 9:17 am
$
0
0

 Just a quick post regarding the following message:

You have reached the maximum number of connected consoles: X. Please contact your administrator

I've been getting this question frequently after reducing the maximum number of connected consoles down to 1 a while back.  We've been having issues with Hung Sessions.

 To resolve this issue, just vMotion the affected VM to another ESXi host.  This will sever the hung console session and allow a user to connect. 

↧
↧
Viewing all 77 articles
Browse latest View live
Search