This is strange indeed. Is it possible that the problem is with screen refresh and not with mouse and keyboard? Do you have any other means for accessing the server’s physical screen and verify?
You may try to check the ‘Use acceleration for display processing’ option in Server preferences -> Performance tab. You can do this before assigning the problematic card, then restart the server and try to connect.