I had the same issue when I first installed my valve. I believe it was the characteristic of the valve itself. If you listen to the valves on certain turbocharged gasser engines or 'ricers' they would flutter roughly the same way. Where are you getting your boost reference from? My valve originally got the boost reference right where the valve was mounted to the charged air pipe. (You can see where I plugged it off in one of my pics)
My theory on this is when the valve would open there would be a sudden loss of pressure right at the boost reference point, causing the valve to shut again. Once the valve shuts it would build boost back up in the reference line and cause it to open again. This would all happen really fast which in turn would cause the flutter. How I solved this and sorta proved my theory, was to get a boost reference signal from the other side of the charged air system. Basically as far away from the blow off valve as possible. This way when the valve opened there would still be a boost reference signal in the valve due the charge air having to flow through the intercooler.
One last piece of the puzzle that helped my valve out drastically, was installing a one-way check valve in the boost reference line. When I would let off this valve would keep a boost signal present longer in the valve, therefor keeping it open longer and working a lot more smoothly. The valve was a little worn so I may have only needed it for my setup, but like I said it takes a lot of tuning depending on what valve is used.