The very small provincial town of Takeo can be pretty-much totally explored on foot in 30 minutes. Separated from the far busier NH3, Takeo feels like the provincial capital Cambodia forgot it had.
The town has a small waterfront area that faces east towards Vietnam and while it was once a bustling little port town, taking in volumes of trade (legal and otherwise) from Vietnam that would arrive by canal, much now takes the easier and cheaper route by land, leaving the port area a shadow of its former self.
Much of the substantial boat trade appears to have switched to Angkor Borei (which is far closer to Vietnam and from where there are good road connections onwards to Phnom Penh and elsewhere).
Outside of the waterfront area, there are a smattering of colonial villas dispersed across town, some of which are in surprisingly good condition, a lovely old wooden house that now houses the Red Cross, and in a small indication of the Khmer Rouge presence that was once here, Ta Mok has a glistening testament to his paranoia with a large house built to the north of town (reached by walking along an earthen levee and a footbridge).
These minor points aside, the sole remaining attractions here would be to visit some of the outlying villages -- easily done by moto or by boat (in wet season), and to drink a few warm beers at the Stung Takeo Restaurant while watching the waterside life.
A very low key and little visited town, Ta Keo would be worth an overnight stay if you have the time, but otherwise a trip out to Phnom Da and Angkor Borei merits at least consideration, as it can be done as a part of a long day trip from Phnom Penh -- the road to the Cambodian capital is sealed and in good condition.