We have key-based authentication implemented in the NX protocol and we use it for server-to-server communication, for example in the cluster. Unfortunately still missing is key selection and GUI integration in the client. Once implemented, key-based authentication for the NX protocol will mirror the SSH implementation, so you will be able to insert the key as you do now for SSH.
There are multiple reasons for using our own protocol rather than SSH. The first is performance. Tunneling over SSH means that our packets have to traverse at least 1 additional process before coming to the destination (at least the SSHD process, if we don’t run a separate SSH client). This is an additional process for each machine traversed, so in a multi-node server there are at least two. Then with SSH we have processes communicating through pipes (like multiple separate commands piped in a shell), so we are adding a further encryption stage at each hop. With NX we can simply hand over the SSL context from one process to the next (as Apache does) and relay connections by only running encryption end-to-end. I can’t provide the details, but we were in a situation where a display packet, to come to the client, had to traverse 12 processes and be encrypted 3 times. Not so with the NX protocol. Additionally, when using the NX protocol, audio and video can use UDP. Not that we couldn’t use a UDP side-channel with SSH, but it would have been hard to explain to managers in a company that, yes, we use SSH for the connection but then most data is not going through SSH. There are additional tiny details, like the efficiency of the crypto key used, that adds to the speed, or the fact that SSHD is a single-threaded process while with NX everything is multi-threaded and can run on platforms, like iOS, where multi-process is not an option.
A second reason is that, using SSH, we can’t simply support a number of features we need in NX, like keeping a NX users separate from the system users, supporting guest connections and redirecting users to different machines without having to create system accounts. Unless recurring to workarounds. That is what we did with the use of the “NoMachine login”. These workarounds are perfectly in the spirit of SSH but were judged “questionable” by some, simply because they by-passed the PAM God and let NoMachine create users and check passwords by its own. With the NX protocol these “questionable” uses of SSHD are gone.
A third reason is supportability. It’s hard to support something you have no control on the way it is used or configured. SSH is an extremely powerful and configurable tool. When users install NoMachine and NoMachine doesn’t work because the SSH client or server are configured to do something special that we could not foresee, users tend to blame NoMachine. They have all rights to do so, but you will agree that hardly this way NoMachine could aspire to become mainstream. Now enterprise users can still use SSH, if they like, but at least we can know if a problem is due to SSH or NoMachine.
A fourth reason is Windows. We ported the OpenSSH client and server to native Windows and released it on the same licensing terms of OpenSSH. This was done to offer the same set of features on all platforms, but clearly we don’t want to keep developing SSH just to be able to run NoMachine on Windows.