The smaller pipe is the vacuum hose - if it's coming off under any circumstances you have to use a hose clip to fix it to the bov. When it comes off, your engine will suck in air through the open BOV (bypassing the AFM) and the engine will begin to stall etc.

You may have to adjust your spring tension as follows:
http://www.gfb.com.au/videos/StealthFX-pt3-small.wmv

Are you saying that with the vacuum hose disconnected, the car runs at full power under boost and flutters etc at low revs?