... And this has the added benefit of being more "socially acceptable" to mainstream people than wearing the vest itself and looking like a dork...
Just my opinion, but the biggest issue I see here is that you care what other people think
about your choice of motorcycle apparel, i.e. "looking like a dork", and you shouldn't. Motorcycle riding is not a fashion show.
Our only concern regarding other drivers on the road, is that we as motorcycle riders are doing what we can to make ourselves as visible as possible to other drivers. And that means wearing brightly colored helmets and jackets. And they don't necessarily have to be Hi-Vis colors. For example, a solid yellow helmet can be seen in 360 degrees by other motorists...
Which brings me to your idea of wrapping the duffle bag with a Hi-Vis shirt. Aside from safety issues pointed out in other posts... sleeves flapping around in the wind, potentially getting caught in the chain, etc... the fact is that shirt is doing absolutely nothing to improve your ability to be seen from the front
by oncoming drivers. One of the more common types of car - motorcycle collisions is where an oncoming car makes a left turn right in front of the bike. And of course the car driver always says "but I didn't see that motorcycle".
Bottom line: Stop worrying about "looking like a dork" or what anyone else thinks of the choices you make regarding the colors of your riding gear. Those people can take their inconsequential opinions and go pound sand.
At the end of the day it's your neck that is on the line. Do whatever you need to do to give yourself the best chances to be seen by other motorists.