We are a bit confused about why this could happen. The server is hosted in a fairly large and reputable data centre and is a true dedicated instance. However, said facility wants to charge rather excessive rates to put this VM server behind a hardware firewall. So we have been relying on the built-in firewall of ESXi.
Despite this, it seems the attacker is still able to at least get through and somehow trigger a ‘wrong password’ error because the server’s ESXi event log shows numerous lines like this:
This is despite the fact that only our office IP is authorized, and we know that nobody here is initiating this.
What are we doing wrong?
After upgrading some hardware, I decided that it might be wise to use ESXI 7 instead of 6.5. Turns out the move is proving to be some horrible nightmare, but I will spare (most of) the details.
For the ESXI DCUI (or «local yellow console»), I can install ESXI and do the basics as root.
Now based on the 3 other questions I’ve looked at, they all seem to discuss lockouts, lockdown, or some combination of the two.
None of these seem to fit my issue, but an answer does seem to mention removing vendor scripts running via cron jobs. I am using Dell’s ISO of ESXI 7 Build 15843807.
I’ve tried reinstalling twice, with a full clean install and overwrite, but I have had no luck.
Also the «lockdown mode» option in the DCUI is greyed out, but not enabled.
I’d like to be able to go the route of the answer mentioned above, but it looks as if that magic was done with SSH.
This setup is a simple 1 ESXI host, without vcenter.
As an interesting note, I’ve tried changing the root password from the ESXI shell, with strong random 13 character passwords from Keepass and whatever recommended password given by the command. Both fail unless you meet additional criteria not listed in the console. Same thing happens on ESXI 6.7 update 3.
In short, there’s a root account lockout feature that was introduced since ESXi 6, that gradually increases lockout timer after certain amount of failed login attempts up to 900 seconds, and those timers seem to be additive. DCUI console is not affected and your password should work there. If your management interface faces public network then you’re in trouble, as brute-force ssh bots will make sure this timer never expires for root account. Look up your /var/log/hostd.log and /var/log/auth.log to see the gory details..
You can, in order of preference, make your management interface not to face public network, configure a built-in firewall to accept traffic from trusted networks only or disable SSH and wait for a few hours until the lockout expires. Then you should be able to login again.
At the same time he could no longer log in to the vSphere client although he was using the correct password. I immediately thought: Wow, this is new in 6.0, never heard of this before! What the heck was going on here?
Well, I just RTFM (read the fine manual) and quickly found out what was going on (yes, VMware’s public product documentation is pretty good, highly recommended!).
ESXi Account Lockout Behavior
Starting with vSphere 6.0, account locking is supported for access through SSH and through the vSphere Web Services SDK. The Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout. By default, a maximum of ten failed attempts is allowed before the account is locked. The account is unlocked after two minutes by default.
And indeed this is new in version 6.0!
In this specific case it turned out that the host was directly connected to the Internet and had shell access via ssh permanently enabled. Of course this is an invitation to hackers all over the world, and they soon and repeatedly tried to break into the ESXi host using brute force. The new account lockout feature helps to mitigate this kind of attack, but it will also prevents legitimate access to the system like in this case. It is important to understand that not only ssh access, but also access to the host via the vSphere client or API calls is prevented when the root account is locked out!
This case reminds us of some basic security rules and best practices for ESXi that are mandatory for production use, but should also be honored for home and lab use:
1. Connect the host management interface to a secured internal network, but not to the public Internet!
2. Enable ssh access only temporarily for troubleshooting, but do not keep it permanently enabled!
3. Create a timeout for ESXi shell availability and for idle ESXi shell sessions. This way the ssh access will automatically be disabled again after a configurable timeout, and forgotten idle sessions will automatically be terminated.
4. If you frequently enable ssh access to your hosts for whatever reason then consider using key based ssh authentication and disable password log in. For further information see KB1002866 and the instructions on Uploading an SSH Key Using a vifs Command.
5. Use the ESXi builtin firewall to limit access for at least ssh (port 22) and the vSphere client (port 443) to known trusted IP addresses. You can read my earlier blog post about Protecting your ESXi hosts against Heartbleed attacks for a practical scenario and examples.
I hope you will find this information useful — stay safe!
You might be
wondering why your ESXi Root Account get’s Locked. Well, if you see VMware is tightening
the security of ESXi for its every version of vSphere ESXi.
When you try to log in your ESXi root account with a wrong password you would be locked for sure, but you get locked even if you try with a correct password. Also, you might wonder what the heck going on! You would probably get the below error:
Vmware is supporting account locking for access through SSH and vSphere Web Services SDK, but it does not support lockout for Direct Console Interface (DCUI) and the ESXi Shell.
If you get
stuck on any issues, our friendly experts are ready to
fix it for you.
By default, you have 5 failed attempts and after 15 minutes you can unlock your account.
You have the option to configure the login behavior too, lets see that:
Note: Your access to the host via vSphere client or API calls is also prevented when the root account is locked out!
Now let’s fix ESXi root Account
Locked Out
Also, select “Disable ESXi Shell” now you can see that the status will change from Disabled to Enabled.
Then, select “Disable SSH” and ensure SSH is enabled, if not you can enable it.
To view the number
of login failure occurred you can use the below command:
In the sameway, if you want to unlock it later, you can enter the below command:
Now you can successfully log in as root using SSH. You can also get DUCI back as normal ESXi interface by using ALT+F2.
Take a look at: Troubleshooting ESXi hostsMonitor VMware ESXi Host Using LibreNMS
