It's really not about losing the armor, it's just that when you lose a hunter/strider you're basically giving rebels armor at a fight that you're already losing and usually strategically disadvantaged, making them even stronger because now they can drop on officers. I rarely pull armor for banks/prison and feds for this very reason. Basically adding fuel to the fire.
Armor is only really useful if you're going to be dropping on someone or dropping people at a specific location. It's really not that useful at a bank and especially not at the prison. I'm more inclined to pull armor at the fed just because it's little harder for rebels to shoot you out + it can help a lot, especially if you're using nitro to get into the fed, but vans are way more useful usually than armor will ever be and you won't be running the risk of giving rebels armor to further defend the area with.