Ha Tien border isn't that bad or any different then the rest of them, just have your visa in order then pay the dollar or 2 worth of bribes...
You can buy tickets to Phu Quoc from tourist agencies in Kampot(they take you to border, transfer to Vietnamese tour agency who takes you into Ha Tien and drops you off at the dock). If you want to do this without a tour agency then your best bet is to ask around and find a moto taxi driver that will take you to the dock in Ha Tien. I believe they can take you the entire way although some may only want to/be willing to go to the border.
If you take bus to Ha Tien without booking the boat ride before hand you will have to get transportation to the dock which is on the other side of the river.
Ok made it to Ha Tien. Took a tuktuk from Kampot with a stop at Kep for lunch cost was $20.00 USD. Then a moto to my guesthouse for $3.00 USD. Was not charged any stamping fees. I already had a visa in hand. Guesthouse has taken my passport to register with the police. Is that normal? They said I will get it back later this evening.
Yeah, that's normal. You need to be registered where ever you are staying and they need your passport and visa details to do that.