You would not need to double back on yourselves from the Giants Forest, you just carry on south and out through Three Rivers and Porterville and then you could cut through on CA155/178 via Lake Isabella to 395. You could also continue to Ridgecrest and take the Trona Wildrose/Panamint Valley road into Death valley as an alternative to 395, this would take you past Ballarat ghost town, although either route will do.

It would be possible to go from Yosemite [or Oakhurst would make a good overnight stop for your last day at Yosemite] to Three rivers the same day and then into Death valley, perhaps Panamint Springs or Stovepipe Wells the following day. On the other hand you could take longer and explore in more depth. If this were the case you could drive through the Sequoia National Forest by taking 190 from Porterville to the Great western divide Highway [107] to Parker Pass and then the Kern River Highway into Kernville north of Lake Isabella. This is a wonderful secluded mountain drive and you would pass the 'Trail of 100 giants' grove of Sequoia's, but it's also slow going. So really there is no single 'best route' but there are options that partly depend on how much time you have to get from Yosemite to Vegas.