I was actually going to ask this same question yesterday but got sidetracked. I know of a few people that have removed theirs and still use their rigs as daily drivers but have not rode with them to see if there is a difference. My thinking is that it would put more stress on the front one, and create a bad situation should that front one somehow fail (as the Antirocks have been known to do), but I could be wrong.
How high is your rig sitting? I pulled mine completely and don't notice any change in stability in off camber situations. I kept my Jeep pretty low though