Let's try to put it in simple terms for the layman (neighbor).
What happens when bees swarm:
They have built up enough bees and stores that they believe they can reproduce the colony (the superorganism) so the old queen and half the bees leave. Now that they have left and bivouacked on a limb, they start looking for a place to start the new colony. They check out every hole leading to every hollow in about 1/2 mile radius, including the holes in the neighbors soffet, your soffet, the wall of the barn, the walls of houses, hollow trees etc. The scouts come back from this and dance for their favorite location and eventually the colony makes a decision about where to go. If you have a nice baited (lemongrass essential oil and old comb) bait hive, they will likely move into that. If you don't, they will move into whatever best meets their criteria. Proof of this sequence is the fact that some even get stuck because they can find NO place suitable and they just stay on the limb and perish that winter because they have no home.
What does NOT happen (which the neighbor seems to be thinking):
The bees are sitting around one day and one comes back and reports a nice place to live. Based on this report they decide to move into this new place rather than stay put.