I'm not sure about travel time from Hoi An, but I went from Hue, and I thought they were worth the effort to get there from there. It ended up being an all day trip on the back of a motorbike, but we stopped at various places of interest along the way. I thought the tunnels were fascinating... but do try to get there before the tour buses arrive. Or at least, wait for their group to go through first.
We ended up going through the tunnels at a leisurely pace with an extended Vietnam family and using the tunnel guide as an interpreter and travellers hand signals, the woman was able to tell us a little bit about her time living through the war. Just after we got through, a large tour group arrive. Excellent timing!
I have heard that the Vinh Moc tunnels are better than the ones in Saigon in that more of the tunnels have been left in their original state, for what it's worth.