Error "Host key verification failed" when trying to SSH to the router from a network computer.

[funky28]

Hi, i have Merlin V 384.16 installed on my ASUS RT-AC3000.

I am able to SSH into the router from many computers on my network, but i think that one particular user from a computer has been locked out and when i try to ssh into it i get the: "Host key verification failed".

Its important to note that on this machine, which is a docker container with the SmartHome software called "OPENHANB" installed, i a can ssh into the router when logging as a root (connected to the machine container via CLI), but when trying another user it give me the error. On my machine command line i run the ssh via :

runuser -l openhab -c 'ssh admin@10.0.1.1"

And i get the above described error.

Searching only it seems that i can resolve this issue by editing the "known_host" file, that is supposed to be located on the .shh on the home location ('~/.ssh).. but for the life of me i cannot find that file. I have tried searching on the /etc folder but cannot find it.. i even tried populating all nvram parameters and filtering (grep) by "rsa" "host" or "ssh" but could not find something that remotely resembles the router blocking the computer IP for a given user.

I even generated rsa keys for the user openhab:

runuser -l openhab -c 'ssh-keygen'

And copied the public key into a .ssh folder that i manually created (see this link) .. no luck either (btw, this method works like a champ for my root user, no issues there!)

Where is the "know_host" file locate on Merlin? and, is this the correct method for trying to solve this or if there is anything else recommended?

Any help is greatly appreciated.

Hugo

 

[eibgrad]

I don't know if this is the problem, but one thing you need to be careful about when dealing w/ ssh on these routers (and other embedded devices) is that by default, most use dropbear for both the ssh client and server. And if you use dropbear to generate your public/private keypairs and access other systems/routers only using dropbear clients and servers, everything is fine.

However, what's more typical is that users generate a public/private keypair w/ either OpenSSH or PuttyGen, upload the public key to the router, then use the private key for authentication. That all works fine when the ssh client is OpenSSH or PuttyGen (e.g., Windows, Ubuntu). But if you attempt to access those same ssh servers from the *router* when it's using dropbear as its client (NOT OpenSSH), it won't work; dropbear requires its own special private key format! You have to convert the OpenSSH/PuttyGen private key to dropbear format using the dropbearconvert utility.

Again, I don't know if this is the problem, but when you're mixing all these OS platforms, some using OpenSSH, others dropbear (usually because it has a much smaller footprint for embedded devices), you can unwittingly create this sort of incompatibility.

 

[funky28]

Thanks @eibgrad for the consideration and suggestion... i can understand your points... so, i have to a few follow up questions to that:

1- So the main secure server connection for ASUS Merlin in this case is dropbear and not ssh?

2- Where is the dropbear known host list located? maybe i can try and read it and see if my machine IP is there marked as blocked or something? i did see the dropbear folder at /etc/dropbear, but since i dont know much about it i did not peaked into it.. i have read in many sites that people prefer dropbear.. i can try and see if i can log in in a different way.

3- Also, what i find weird is that I can ssh from many other locations using just the normal command and typing in my password (i rarely use the keyless log on), including this same machine but just a different user.. i only tried storing the rsa key into the router itself trying to see if it will fix the issue, but at this point i just want to ssh into the router from this specific user at my specific machine/location via the command line

I dont mind trying to learn dropbear, but i think based on the error code if I could manually delete my IP and user from that "blocked" state, then i can move forward with a simple ssh.

Again, i appreciate the comments and points.. if you can tell me where to find the dropbear list of accepted host or how to clean itn... and i can try and look into it and maybe see if this can lead me to a path where i can resolve this.

Hugo

 

[eibgrad]

1- So the main secure server connection for ASUS Merlin in this case is dropbear and not ssh?

It's ssh (the application protocol), but the *implementation* of ssh is called dropbear. Embedded systems typically prefer dropbear because of its small footprint (at least compared to OpenSSH). If you dump the process table (ps command), you'll likely see the following:

3619 admin     1128 S    dropbear -p 22 -s -j -k

That's the dropbear ssh server listening on port 22.

And if you list the ssh client, it will likely be aliased to dropbear as well (usually dropbearmulti, which supports all the functions of dropbear's implementation of ssh in a single module).

admin@merlin-lab1:/tmp/home/root# ll /usr/bin/ssh
lrwxrwxrwx    1 admin    root            13 Apr 25 20:59 /usr/bin/ssh -> dropbearmulti*

In the above case, dropbearmulti detects the name "ssh" (knowing you want the ssh client) and calls dbcient.

admin@merlin-lab1:/tmp/home/root# ll /usr/bin/dbclient
lrwxrwxrwx    1 admin    root            13 Apr 25 20:59 /usr/bin/dbclient -> dropbearmulti*

Yes, confusing. Bottomline, it's using dropbear, NOT OpenSSH. Not unless you specifically installed OpenSSH (e.g., via Entware) and are specifically using the OpenSSH client and server.

2- Where is the dropbear known host list located? maybe i can try and read it and see if my machine IP is there marked as blocked or something? i did see the dropbear folder at /etc/dropbear, but since i dont know much about it i did not peaked into it.. i have read in many sites that people prefer dropbear.. i can try and see if i can log in in a different way.

Remember, the known_hosts file is stored on the ssh *client*! The fact you keep asking about where dropbear stores it makes me concerned you believe its associated w/ the ssh *server*. It's only relevant to the router if the ssh client is running on the router. The known_hosts files should be where it's always supposed to be w/ Linux; under the .ssh folder of the user's home folder. If the ssh client is running on Windows, it's probably stored in the registry.

3- Also, what i find weird is that I can ssh from many other locations using just the normal command and typing in my password (i rarely use the keyless log on), including this same machine but just a different user.. i only tried storing the rsa key into the router itself trying to see if it will fix the issue, but at this point i just want to ssh into the router from this specific user at my specific machine/location via the command line

Again, this could all be a red herring. It's actually very difficult to fully understand what's happening there based solely on your description. I'm offering this information in the hope that *maybe* it has some relevance. But I just can't be sure.

The only reason I even brought it up is because it seems your ssh client is NOT on the router, but some other system. But it's not clear which OS platforms are specifically in play here, and therefore whether those clients are using dropbear or OpenSSH. And it's also not clear if the public/private keypairs were generated w/ dropbear or OpenSSH.

As I said, you can unwittingly create incompatibilities here depending on how you generated your keys, and whether you're using dropbear or OpenSSH for the client and/or server. Without such details, it's hard to be sure what's happening there.

 

[funky28]

Thanks again for all the detailed information.. i think i am beginning to understand why i was confused about this.

Yes, i was under the impression that the known host file should be located not the server, i thought i read it somewhere.. or maybe i just wanted to believe that... either way, i will start paying more attention on the client side and see what i can find.

I know that my current distribution is not well explained, but its also a bit complicated. I am running several docker containers an they have minimal applications install, just what is needed to run the specific program.. i did try searching for dropbear on my containers, but i did not see it.. so probably they have a flavor of openssh.. other than the dockers, i use my Mac which i dont think it has dropbear in it.

I have not manually installed openssh or any other application protocol via entware or apt-get (on the containers).

I think you have shed some light into some facts that i was not aware of, i will do some research and try to look at my client docker too and see what i can find... i thought of two other possible solutions last night, 1- run the ssh call as a "root" when logged with the other user (i dont really want to do this) or 2-create a new container with a different IP and start again fresh... i will use these as a last resort, but i am a problem solver and i really want to learn and get to the bottom of this

I will let you know what i find.

Thanks

 

[MaziahBebop]

Thanks again for all the detailed information.. i think i am beginning to understand why i was confused about this.

Yes, i was under the impression that the known host file should be located not the server, i thought i read it somewhere.. or maybe i just wanted to believe that... either way, i will start paying more attention on the client side and see what i can find.

I know that my current distribution is not well explained, but its also a bit complicated. I am running several docker containers an they have minimal applications install, just what is needed to run the specific program.. i did try searching for dropbear on my containers, but i did not see it.. so probably they have a flavor of openssh.. other than the dockers, i use my Mac which i dont think it has dropbear in it.

I have not manually installed openssh or any other application protocol via entware or apt-get (on the containers).

I think you have shed some light into some facts that i was not aware of, i will do some research and try to look at my client docker too and see what i can find... i thought of two other possible solutions last night, 1- run the ssh call as a "root" when logged with the other user (i dont really want to do this) or 2-create a new container with a different IP and start again fresh... i will use these as a last resort, but i am a problem solver and i really want to learn and get to the bottom of this

I will let you know what i find.

Thanks

When you get into the client docker (the one having the errors):

nano ~/.ssh/known_hosts

Remove the entire line of the offending key. More info here (look in the Method 1 section).

 

[funky28]

You guys are pointing me on the right path! here is what i have been able to find this morning:

The container/machine (client) i am trying to ssh from can only be logged as "root" but i want to execute the command under a different user (openhab), the root "~/" folder does have the ".ssh" folder, but the "openhab" user does not, and therefore there are no known_host files, or rsa keys... I ended up creating the ".ssh" folder under that user (/openhab/.ssh). And then tried ssh into the router. At first it kept giving me the same error, but thanks to the link provided by Maziah i used the "strickhostkeychecking" parameter as "no"... This time i got a different error:

root@cdb002404735:/openhab/.ssh# runuser -l openhab -c "ssh -o 'StrictHostKeyChecking no' admin@10.0.1.1"
Warning: Permanently added '10.0.1.1' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Connection closed by 10.0.1.1 port 22

It indeed created the known_host file, under /openhab/.ssh.. but still no luck in logging in...Then i tried to verbose the ssh command, and got the following:

/openhab/.ssh# runuser -l openhab -c "ssh -v -o 'StrictHostKeyChecking no' admin@10.0.1.1"
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.0.1.1 [10.0.1.1] port 22.
debug1: Connection established.
debug1: identity file /openhab/.ssh/id_rsa type -1
debug1: identity file /openhab/.ssh/id_rsa-cert type -1
debug1: identity file /openhab/.ssh/id_dsa type -1
debug1: identity file /openhab/.ssh/id_dsa-cert type -1
debug1: identity file /openhab/.ssh/id_ecdsa type -1
debug1: identity file /openhab/.ssh/id_ecdsa-cert type -1
debug1: identity file /openhab/.ssh/id_ed25519 type -1
debug1: identity file /openhab/.ssh/id_ed25519-cert type -1
debug1: identity file /openhab/.ssh/id_xmss type -1
debug1: identity file /openhab/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version dropbear
debug1: no match: dropbear
debug1: Authenticating to 10.0.1.1:22 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:1tJjbGPqzV-xxxxxxxxxxxxxxxx
debug1: Host '10.0.1.1' is known and matches the ECDSA host key.
debug1: Found key in /openhab/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: Will attempt key: /openhab/.ssh/id_rsa
debug1: Will attempt key: /openhab/.ssh/id_dsa
debug1: Will attempt key: /openhab/.ssh/id_ecdsa
debug1: Will attempt key: /openhab/.ssh/id_ed25519
debug1: Will attempt key: /openhab/.ssh/id_xmss
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /openhab/.ssh/id_rsa
debug1: Trying private key: /openhab/.ssh/id_dsa
debug1: Trying private key: /openhab/.ssh/id_ecdsa
debug1: Trying private key: /openhab/.ssh/id_ed25519
debug1: Trying private key: /openhab/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
Connection closed by 10.0.1.1 port 22

I think is looking for he rsa keys, and since there are none on the .ssh folder is throwing the error... but i would have expected that if no keys are provided it would have asked me for my admin router passoword, anyways, I have to go to work now, but will try later to create set of keys using sshkeygen for the openhab user and put them where they belong and see if that lets me connect. (see this link) .

I think i am closer to getting this resolved! will post back if i get more errors or if i can finally log in.

Thanks!

 

[MaziahBebop]

You guys are pointing me on the right path! here is what i have been able to find this morning:

The container/machine (client) i am trying to ssh from can only be logged as "root" but i want to execute the command under a different user (openhab), the root "~/" folder does have the ".ssh" folder, but the "openhab" user does not, and therefore there are no known_host files, or rsa keys... I ended up creating the ".ssh" folder under that user (/openhab/.ssh). And then tried ssh into the router. At first it kept giving me the same error, but thanks to the link provided by Maziah i used the "strickhostkeychecking" parameter as "no"... This time i got a different error:

root@cdb002404735:/openhab/.ssh# runuser -l openhab -c "ssh -o 'StrictHostKeyChecking no' admin@10.0.1.1"
Warning: Permanently added '10.0.1.1' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Connection closed by 10.0.1.1 port 22

It indeed created the known_host file, under /openhab/.ssh.. but still no luck in logging in...Then i tried to verbose the ssh command, and got the following:

/openhab/.ssh# runuser -l openhab -c "ssh -v -o 'StrictHostKeyChecking no' admin@10.0.1.1"
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.0.1.1 [10.0.1.1] port 22.
debug1: Connection established.
debug1: identity file /openhab/.ssh/id_rsa type -1
debug1: identity file /openhab/.ssh/id_rsa-cert type -1
debug1: identity file /openhab/.ssh/id_dsa type -1
debug1: identity file /openhab/.ssh/id_dsa-cert type -1
debug1: identity file /openhab/.ssh/id_ecdsa type -1
debug1: identity file /openhab/.ssh/id_ecdsa-cert type -1
debug1: identity file /openhab/.ssh/id_ed25519 type -1
debug1: identity file /openhab/.ssh/id_ed25519-cert type -1
debug1: identity file /openhab/.ssh/id_xmss type -1
debug1: identity file /openhab/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version dropbear
debug1: no match: dropbear
debug1: Authenticating to 10.0.1.1:22 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:1tJjbGPqzV-xxxxxxxxxxxxxxxx
debug1: Host '10.0.1.1' is known and matches the ECDSA host key.
debug1: Found key in /openhab/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: Will attempt key: /openhab/.ssh/id_rsa
debug1: Will attempt key: /openhab/.ssh/id_dsa
debug1: Will attempt key: /openhab/.ssh/id_ecdsa
debug1: Will attempt key: /openhab/.ssh/id_ed25519
debug1: Will attempt key: /openhab/.ssh/id_xmss
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /openhab/.ssh/id_rsa
debug1: Trying private key: /openhab/.ssh/id_dsa
debug1: Trying private key: /openhab/.ssh/id_ecdsa
debug1: Trying private key: /openhab/.ssh/id_ed25519
debug1: Trying private key: /openhab/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
Connection closed by 10.0.1.1 port 22

I think is looking for he rsa keys, and since there are none on the .ssh folder is throwing the error... but i would have expected that if no keys are provided it would have asked me for my admin router passoword, anyways, I have to go to work now, but will try later to create set of keys using sshkeygen for the openhab user and put them where they belong and see if that lets me connect. (see this link) .

I think i am closer to getting this resolved! will post back if i get more errors or if i can finally log in.

Thanks!

The problem as I see it is with the user, openhab, and how it was created. It looks like that user was not meant to be used this way because: openhab had no homedir and no tty. It also appears that ssh is authenticating with key first, then a password, and failing on the password input with a tty error.
Ideas: use the 'adduser' command to make a new user account (with a homedir). If that is really not possible, at least start by adding a home directory for openhab, not manually but with a helper function like mkhomedir_helper. Except for root, a user's home will usually be in /home/openhab for example, not /openhab

[edit]
The tty message is very revealing, it shows that your ssh server is looking for a password input from you, but the way you're using the container it can't give you the pw prompt and fails. See https://forums.cpanel.net/threads/r...no-such-device-or-address.621071/post-2524503

The permission denied message is also revealing. Did you create any files/folders as root? If yes, they need to be chown as openhab.

 

[funky28]

I was able to ssh into the router by creating private/public rsa keys using ssh-keygen and then invoking their custom location from the command line:

runuser -l openhab -c "ssh admin@10.0.1.1 -i /openhab/conf/exec/ASUS/id_rsa "

note: in the end i did not need the 'StrictHostKeyChecking no' option.

I think you had a great point about the tty message and trying to create the home folders, but you see with containers, i loose anything i create once i restart the container, and creating too many customer folders and permissions might bring me problems later...the location where i stored the key is in a volume that i have mapped from my network to the container (so that i survives restart). Hence my custom location.

I think the deal breaker in getting this done was the fact that i did not have the .shh folder on the "openhab" user for the ssh command to store/create the known_hosts. Probably the .shh won't survive if i restart the container, that i can easily include a custom script to check for this folder and create it if it is not present.

i will, however give it a try creating the home folders like noted, assigning the correct permissions and see if it prompts me for the pswrd as it should.. but that will be just for fun

Thank you for MaziahBebop and eibgrad for all the great pointers and help.

For now, mission accomplished