To reach it, head north on Ruamchit Road and it will eventually turn to dirt and pass through a line of Thai shops, fishmongers, and associated restaurants.
The road tees at the harbour -- to the right is the beach and Shylae Seafood and a couple of other restaurants. To the left, the road continues around the harbour past the docks, more restaurants, and a narrow bridge leads to the other side. It's worth half a day or so just to wander around and take in the bustling scene.
Fishing is an important part of the local economy, and much of the fish served down in Hua Hin actually comes from right here. Try to time your visit for lunch!