The .qt files in the other directories are fine and safe. Don't remove them!

If you're still feeling paranoid (and it's not such a bad thing!) you might like to install a firewall and look for Linux security articles on the internet (Google will chuck up plenty).

But it's worth mentioning that the best security strategy is to simply back up your data.