Welcome aboard the RoadTrip America Forums!
From your description of your drive down, it seems as though your interests are mainly in picturesque small towns: walking them, shopping them, just soaking up the ambience. It also seems as though you have roughly three full days, maybe a little bit more, to make the return trip to New York (City?). With just that to go on, here's a few suggestions...
First, I'd do everything in my power to avoid I-95 between Richmond and NYC. There's absolutely nothing charming about that road. And there's no need for you to take it. After a day (tomorrow) exploring Wilmington and the local beaches, look at using I-40 west to Greensboro and then use US-220 up to Roanoke VA and I-81. This route will take you through Raleigh, but you won't have a lot of time to really explore it, maybe a meal and a short window-shopping walk. Heading northeast out of Roanoke, you can drive a few miles on the Blue Ridge Parkway as time permits, but I-81 is going to be your main route north. A couple of small, charming towns you will want to at least check out for future trips include Lexington VA, Williamsport MD (and nearby Fort Frederick State Park and the Chesapeake and Ohio Canal National Historical Park), and Hershey PA. East of Harrisburg switch over to I-78 to finish the drive back into the NYC area.
You'll need to be behind the wheel for two full days to get home, so that really only leaves you about a day or so to spread out amongst all the places you want to stop, so you'll have to be judicious in your use of that time. Still, you can make this a quite pleasant drive with lots of short stops to see what you'd like to return to at some later date when you have a bit more time.
AZBuck