+1 to the alcohol suggestion.
It's the first thing I try with paint transfer issues (accident damage) and it usually works great. Don't leave it on very long.
Another suggestion is to add a bit of abrasive to the WD40. Cigarette ash works well.
That concoction will remove dried permanent ink from a painted surface.
Human spit is another solvent that people overlook. Sometimes, depending on what is stuck to the paint, it works better than WD or alcohol.
Berryman's B12 is the last thing I will try. Don't leave it on for more than 5 seconds. It will damage the original paint if you give it any time. I'll spray it on a paper towel and wipe followed by soapy water immediately.
I will try the B12 only after everything else has failed and I'm to the point of a respray anyways. It's a "Hail Mary" last resort type of application.
Spit and WD have never damaged the original paint for me.
Alcohol will if left on long time. It "dulls" it
B12 will if left on more that 5 seconds. It removes it.