The 'jaggy edges' come from the fact that LCD screens suck at displaying anything but the native resolution of the panel (basically you can't display half a pixel on LCDs so you lose contrast). So, when you play the ps3 at 540p you are not sticking to the 1080p native resolution of the panel, and it becomes blurry.

Lucky for you the ps3 is able to display the native resolution of your tv, so yeah like the others said, get a hdmi cable.