It's not torque steer related to power. It's most likely related to a worn suspension because a few degrees just by releasing hte gas pedal is a lot.
The pull to one side is from an alignment issue but before you get an alignment that I would inspect the suspension since that would cause the alignment issue.
The most likely areas of concern are the lower control arms, tie rods, and struts. To test tie rods, securely lift the corner that you want to test. Hold the wheel at the 3 and 9 o'clock position or as close to the tie rod plane as possible. The wheel should move tight with the tie rod. There is an inner and outer tie rod and both are wear items. Since you noticed the steering wheel move also check the tightness of the steering rack. I forget if the rack has bushings that connect it to the subframe or not but I think it does not have bushings there.
The rear lower control arm bushing is also a common wear item. This page is for an a4 body but your earlier style control arm is very similar:
http://www.myturbodiesel.com/1000q_how_to/a4/LCAbushing.htm . The part number for the rear lower control arm bushing is actually the same. I suggest using the Audi TT bushing - it's still soft enough so that you won't notice any harshness. The front bushings don't seem to wear much.
Worn springs will sag and it's common that the driver's side will sag more because that's the side that gets more weight. If the car is parked on a level road and the tires are inflated equally, see if the car is leaning to one side. This shouldn't cause the effect that you are seeing but is a rough judge of how worn the springs are. If the spring broke than you should hear clunking when the weight shifts. That could cause strange handling. More likely is worn struts that aren't controlling the weight transfer.