We are using NoMachine Terminal Server to virtualize our callcenter and enable home office for our interviewers.
We tried different sip clients and OS for the server. With ubuntu 18.04 we also had a 2 to 3 sec delay on both legs. It was independant of the sip client (jitsi, linphone, zoiper), location (on remote site or lan) or the pbx (direct connection to the pbx was without delay). On debian 9 we had a 2 to 3 sec delay, but only in one direction. On debian 10 we now persistently have approx. 1 sec delay on both legs. We settled with this delay for now and have been running hundreds of interviews per day the last week. Of course we would prefer to have no delay at all because it can produce awkward situations sometimes during an interview.
The server is running as kvm on a high performing proxmox cluster. I don’t know if that might also play a role but I doubt it. We run other servers and the pbx on proxmox, never had any performance problems.