You have added more void space to the hive by adding the supers. They were probably always there but now how more space to hide from the bees and move in the open on the comb where you see them. If you need the space (and you are probably in your flow and do) leave the super on if not remove it.
I hear folks talking about caulking this and that, but not sure if it accomplishes much. I realize the philosophy of it, getting rid of spaces for them to hid, but just not sure it helps a whole lot.
How many beetles are you seeing? It is not uncommon for me to see 12-16 or so when I pop a top where the bees have them corralled. Something in this area we just have to learn to live with and watch closely. When you begin to see shb move freely on comb with honey and pollen and the bees are ignoring them (not chasing) watch out :'(