Thanks for the advice AZBuck and ashank1m!
Unfortunately, I've booked accommodation for all the destinations so there's little I can do in terms of changing locations. However, if it's gonna be tough to drive all the way from Asheville to Charlottesville, perhaps I can split that into 2 days? Any recommendations on where I should stop in between?