I use TC and the the following script. My PBX sets the TOS for SIP and RTP packets and then I use a U32 filter to put them in the appropriate queues.
You could change the U32 filter to look for packets coming from a certain port (sport) or going to a specific port (dport). I think you would change " u32 match ip tos 0xb8 0xff" to something like " u32 match ip sport 5060" (check the man pages for tc to see if this is correct).
Additionally, you should change the bandwidth setting for the third queue to something that is appropriate for you (I have a 1Mbs outgoing connection).
If you search for "tc" and "voip" you will find similar scripts
#!/bin/bash
# this is intended to be low latency which is why prio, sfq and tbf are used
# set queues up for traffic on WAN (eth0)
# clear exitisting queue structure (this will cause a non-fatal error if no queues were previously set)
/sbin/tc qdisc del dev eth0 root
# put everything in the third queue by default
/sbin/tc qdisc add dev eth0 root handle 1: prio priomap 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
# set up the queues, rate limit the third queue so that there is always some bandwidth for voip traffic
# note: the use of perturb with sfq seems to increase jitter
/sbin/tc qdisc add dev eth0 parent 1:1 handle 10: sfq
/sbin/tc qdisc add dev eth0 parent 1:2 handle 20: sfq
/sbin/tc qdisc add dev eth0 parent 1:3 handle 30: tbf rate 750kbit latency 30ms burst 10000
# Audio packets go to first queue, PBX sets TOS for RTP audio packets=ef, ef=0xb8
/sbin/tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip tos 0xb8 0xff flowid 1:1
# SIP packets go to the second queue, PBX sets TOS for SIP pakets=cs3, cs3=0x60
/sbin/tc filter add dev eth0 protocol ip parent 1: prio 1 u32 match ip tos 0x60 0xff flowid 1:2