What you are experiencing is called drift, bees from one nearby hive landing and entering another. This happens a lot in a bee yard. In my apiary I have 3 hives in a row spaced 7 feet (2 meters) apart, the bees from the middle hive often end up at the hive on one side or the other, making them strong and productive while the middle hive struggles. The closer the hives are together the more pronounced this problem becomes.
To solve this problem face one of the hives southeast and the other southwest, this slight change of entrance allows a more definitive orientation on the hive location by the forager bees and should greatly decrease the drift between hives.
Drift can result in robbing if it occurs in great numbers if the hive of origin of the drifted bee accepts the errant bee back in to the hive and learns of the weakness of the other hive.
I notice a bee carrying pollin trying to get in so I let her in and before I can close up again she comes flying out and heads for my other hive, two seconds later she is back! Now I'm confused because I know they are robbing but why are there pollin carriers flying around? I read up and found that robbers don't carry pollin!
What you describe here is evidence of drift, the worker bee from a different hive with a load of pollen is allowed into a neighboring weaker hive, realizes its mistake and attempts to enter the hive it originated from but the original hive, having a stronger guard force repels any bee with a slightly different odor. The forger bee with the load of pollen picked up a little of pheromones of the hive it temporarily entered so was rejected from its own hive and therefore returned to the other hive.
This could mean that the weaker hive is not yet being robbed and reducing the entrance size does not always prohibit a hive from being robbed, especially if it is already being robbed. The use of a robber screen at the entrance of the weaker hive will redirect the entrance confusing the robbing bees while forcing the weaker hive to more narrowly orient on the hive due to the barrier the screen creates, reduces drifting because the robber screen is a physical and visual barrier that differentiates the hives from each other.
Weaker hives will accept any forager bearing gifts of nectar, pollen, water, or propolis. They need the whatever extra comes their way and will accept any productive forager into their hive, the drift bee becomes a member of the hive to which it errantly entered.