UPX really only compresses exe's, you shouldn't rely on it to protect your exe's alone. There are some programs that can encrypt your exe, but just like UPX has unpackers, most have decrypters.

When you run a UPX compressed exe, it automaticaly decompresses the program to it's original state, so there's no real loss of performance while running.

Yes, you should test everything, not only test that the protection hasn't caused bugs in your code, but also the protection itself. Grab a memory editor and/or a hex editor and just tinker around, see if the security actually does anything to hinder cracking.

Good Luck!