Welcome aboard the RoadTrip America Forums!
You're probably looking at 5-6 hours of driving to make the trip from Boston to New York by way of Mystic, and while that does leave you some time to walk downtown Mystic and visit Mystic Seaport, it doesn't leave you a lot of time for much else. So my recommendation would be that you stick to the major highway, I-95, tour the Seaport if you have time, and otherwise stroll Mystic. There is a home-made ice cream shop on the north side of US-1 at the west end of the bridge across the Mystic River that has a small deck where you can sit by the river to enjoy your purchase. But if you plan to spend much time in Stamford as well, and if your destination there is away from the highway, you'll have to curtail your time in Mystic.
AZBuck