When it comes to Cha-am and eating it's all about the seafood. Even on weekdays, when the hordes have picked up and left, there is a staggering amount of seafood for sale along the beachside road (and along Naranthip Road for that matter). You'll see live prawns, crab and fish along with the standards like bugs, squid and various clams and other heavy metal filterers to choose from. The process is simple: point at what you want and then have it cooked to your liking. There is little price variation (if any) between the stalls on any given day, so it's really just a case of wandering along until you find some nice crabs.
Sticking with the seafood for a second, if you head north as far as you can go you'll reach the fishing village where the boats deliver their loads daily. There are a few places to choose from on the beachfront here (just to the south of the pier) but we'd suggest keep walking until you reach the actual unloading dock (you can't miss it), turn left and walk along -- you'll see there are a bunch of seafood places lined up. We had some amazing seafood at the second one along.
Off the beach and away from the fish, plenty of food stalls do everything from satay to som tam and, especially along the southern stretch, there are a few Thai hole-in-the-wall restaurants that have an English menu and will be able to cook up just about anything Thai you want.
For non-Thai offerings, Soi Bus Stop has plenty of small, tourist orientated restaurants, including some bars and a pizzeria to choose from, and we spied an Indian restaurant along the north side of the beach. The quaint blue-coloured Cha Inn Cha-am, on the north side of the railway rd, does good coffee and Western food.
There's a reasonable bakery and cafe on Naranthip, fittingly called the Baguette, which does real bread along with a variety of pasties and sweet things. One last tip: along Naranthip Road, about half way to Phetkasem Road, on your left you'll find Krua Garn Wela. While we didn't try here (it was closed when we passed) we were advised it does the best Thai food in all of Cha-am. Probably worth a go.