Sounds like the relay isn't being triggered properly. Maybe its not wired up correctly?

If you're using the existing fuel pump wiring as a trigger wire for the relay, it should turn on and off with the in-tank pump.