Ninh Binh's train station is just behind the bus station across the Van River from the post office. At least five trains a day stop en route to Hanoi and at least four stop heading south, taking around 3 hours. Seats to Hanoi range from 44,000 to 85,000VND. Fares vary depending on the class of train and seating.
Buses take around 1.5 to 2 hours to/from Hanoi, depending on the type of bus and the route it takes. Allow 2 hours to be safe. From Hanoi, buses leave from Giap Bat (southern) bus station. As soon as you arrive you are likely to be accosted by someone asking if you want to go to Ninh Binh -- they are trying to get you on their bus and there's no harm in this, just make sure they take you to buy a ticket at the window in the ticket office (70,000VND) before getting on the bus.
Ninh Binh's bus station is located near the railway station, just across the bridge leading to Phat Diem, on the last section of Tran Hung Dao (Highway 1) going south towards Hue. Buses to all destinations are frequent and you'll rarely have to wait long. If you're not staying near the bus station, you can wait on the main road -- Tran Hung Dao -- and wave down a bus, but you may be stuck sitting on the floor.
Most buses to Hanoi terminate at Giap Bat bus station, although some go to My Dinh. Either is a bit of a trek from Hoan Kiem.
Fares to Hanoi range from 60,000 to 70,000VND. We heard news of a tourist bus that would pick up at hotels in Ninh Binh for around 120,000VND and drop off near Hoan Kiem Lake in Hanoi, but we didn't try it out (preferring the floor of an overcrowded bus of course).
Heading south, you may prefer to book onto an Open Tour bus to ensure you get a seat, as the next stop of particular interest is Dong Khoi, and that's 400 kilometres away. Hotels can help with this.
Bicycles and motorbikes are available to rent from most hotels or guesthouses and the land is flat so not too strenuous. Alternatively you can take a taxi to Tam Coc and from there walk to Bich Dong and even to Hang Mua, if you're keen on walking. Or rent a bicycle in Tam Coc.
To visit Trang An, Hoa Lu and Bai Dinh Pagoda you'll need to either go by bicycle or motorbike or take a taxi or motorbike taxi between venues, as it's a bit far to walk. Otherwise, book onto a tour or arrange a private car through your hotel.