Well, just try
fish://username@computerort
I can't try this at the moment (I use the standard ports on all machines), but it just may work exactly as planned.