Troubleshooting 2FA login failures

Administration
, , ,
1

Troubleshooting 2FA-enabled users who are unable to log in to the Morpheus UI

Outline of a recent support case in which all 2FA users could not log in to the Morpheus UI successfully.

Issue Summary:

Every user with 2FA enabled is unable to log in to morpheus. We are using the default login and have no external identity source configured. Users are seeing the following error in the UI:

User could not be authenticated. Invalid verification code.

In the logs I can see this:

2024-04-21_14:04:55.88015 ''[2024-04-21 14:04:55,897] [http-nio-127.0.0.1-8080-exec-5] INFO  c.m.AuditLogService - CEF:0|MorpheusData|Morpheus|7.0.1|security|Failed Login Attempt 1 of 0|cn1=1 cn1Label=User Id src=<ip-address> 
2024-04-21_14:04:55.89813 ''[2024-04-21 14:04:55,906] [http-nio-127.0.0.1-8080-exec-5] ERROR c.b.p.LoginController - Auth Exception: com.bertramlabs.plugins.AccountsException: com.morpheus.BadVerificationCode 
2024-04-21_14:04:55.90671 ''[2024-04-21 14:04:55,906] [http-nio-127.0.0.1-8080-exec-5] ERROR c.b.p.LoginController - Auth Exception: com.bertramlabs.plugins.AccountsException: com.morpheus.BadVerificationCode 
2024-04-21_14:04:55.90672 ''[2024-04-21 14:04:55,907] [http-nio-127.0.0.1-8080-exec-5] INFO  c.m.AuditLogService - CEF:0|MorpheusData|Morpheus|7.0.1|security|User failed to log in|cs1=ueqbal cs1Label=Username src=<ip-address>

We have already checked the 2FA codes and these are being entered correctly. Pls help

Since the issue is affecting ALL users with 2FA enabled, we can rule out bad/expired passwords or mistyped/incorrect 2FA codes. Although, it is important that these are adequately checked (step 1 outlines some basic checks).

Troubleshooting Steps:

  1. Basic checks:

    • The 2fa code is valid - for the morpheus application, 6 digits, and is not the last successful code or an expired one
    • The code is entered within the time window of its validity (30 seconds by default)
    • The authenticator application is generating a different code at each 30s interval

  2. Can users log in to another application successfully using the same or different authenticator app?

  3. [:exclamation:IMPORTANT​:exclamation:] Check the NTP service configured on the morpheus service node. This is a base requirement irrespective of the selected architecture (Single-Node vs HA). For HA environments in particular, it is crucial that the time is correctly synchronised between all nodes in the application stack

    Example below on how to check the NTP service status on Linux using timedatectl:

     user@app-node-1:~$ timedatectl status
                    Local time: Sun 2024-04-21 14:38:05 UTC
                Universal time: Sun 2024-04-21 14:38:05 UTC
                      RTC time: Sun 2024-04-21 14:38:05
                     Time zone: Etc/UTC (UTC, +0000)
     System clock synchronized: yes
                   NTP service: active
               RTC in local TZ: no
  • Is the NTP service inactive? Is System clock synchronized: no? If so, this could mean that the NTP service is misconfigured or hasn’t been set up at all. Please contact your internal infrastructure team (IT / Networking) to rectify this since Morpheus Support are not responsible for the management of the infrastructure components in your environment

Issue Resolution:

  • Corrected issues with the NTP service and clock synchronisation between nodes - node 1’s time had been manually set and ~1 minute ahead of nodes 2 and 3

Administrators should follow the instructions here when trying to recover an appliance for which all users are locked out of.

Feel free to contact support@morpheusdata.com if you are still having issues after running through the troubleshooting steps above.

1 Like