Sounds like the guard bee are doing a good bees of guarding the hive from initial attacks. Unless you have very few bees flying, it is very hard to tell that all of those bees flew back into the same hive. I have watched my bees for hours and it is very hard to tell which bees are coming and going once they are in the air.
Next time you see it, have some powered sugar in a shaker ready and sprinkle some on the bees as they fall to the ground. It will make it easier to see where they go.
Jim