Just want to report my success to get these things to work together.

Steps:
1. Run alsaconf under root.
It will configure the sound card to load the corresponding drivers. Did a lot of crucial steps to make those drivers to work.
2. Run timidity -iA -Os.
I still don't understand why timidity cannot output to the sound card directly, but outputting to Alsa is working fine. Now I can use pmidi to play my midi file through timidity.
3. Run Rosegarden4
Now everything is working. Since the alsaconf will establish one more midi port corresponding to alsaconf which I never able to play correctly. However, those Timidity ports work fine as pmidi.

It looks like an ugly workaround, at least it is working so I can start to use Rosegarden for my project. I don't know it is a problem with timidity or the alsa stuff.

There are some good doc in TLDP http://www.tldp.org/HOWTO/HOWTO-INDEX/howtos.html. Even though it didn't help directly, but still a good read to understand how the sound works.

ys