Depends on time, interest and comfort.
The quickest, easiest and most comfortable trip would be HCMC - PP and then get the PP - Kampot bus at 2.30 pm and you can be there in 1 day if you leave HCMC in early morning.
If you have more time and would like to see more things along the way you can travel all the way down through the delta to Ha Tien and then cross the border to Kampot. If you do this trip it would be best to do it in a 2-3 day trip so you can enjoy the area. I'd spend a night in Can Tho, visit the floating market then head to Ha Tien and spend a night there. The next day you'll be in Kampot in 2 hours.
If heading Kampot to Saigon and you want to get there within a day or so(don't want to spend extra time checking out the mekong delta) I would do the same thing I ended up doing.
First from Kampot or Kep, take the tourist minibus to Ha Tien . On the drive into Ha Tien make it known you want to be dropped off at the bus station(even then they might not stop so keep your eye out on right hand side for the bus station and tell the driver to stop). If it gets missed will just cost you an extra buck to get there from town. Another option is hire a moto guy for the journey cost a couple bucks more but probably worth it for comfort sake if you are travelling solo, I saw a couple guys get the reverse trip for $8(but I think they got a pretty good deal).
At the bus station on the right hand side you will see Kumho bus lines, book a night bus with them and then take a moto into town(you can leave bag in restaurant just inside of the bus station entrance) and spend the afternoon looking around the river/market etc. I found the Kumho bus to be good quality(it was recommended to me by the moto drivers out front) and it was only around $10.
Ha Tien was actually a fairly nice town and most of the people there were quite friendly. The lady that operates the travel agency that does the Vietnam side of the tourist minibus is a dirt bag though and both her and the driver don't listen very well... They drove right past the bus station but luckily there was like 5 of us that wanted to go there and we stood our ground until they took us back.
Edit: Oh and for a cheap way into central Ho Chi Minh hop on I believe bus 102 which will take you from Mien Tay bus station(which you arrive at) to the Ben Thanh station which is walking distance to Pham Ngu Lao.