Being winter time my swarm traps that I leave in the trees on my property are seen by anyone looking up. He came over asking me to take them down. That I'd be stealing his bees if they swarm!
If those traps were not in the vicinity of your own hives, then I would have said that he's got a point. Putting up traps close to another beekeeper's hives means stealing his swarms. But at the same time you have to be a prudent beekeeper yourself and try to catch the swarms that issue from your own hives. For that reason I think it should be okay for you to have traps near his hives if you have the traps nearer to your own hives.
Say, are your traps baited with pheromones?
I think it is a beekeeper's responsibility to ensure that his hives don't swarm (if he doesn't want them to swarm), or to be present at the hives when they swarm (if he wants them to swarm).
The murkier question is what happens when one of his hives swarm and you catch the swarm. It depends on local law, but hereabouts the rule is that the swarm belongs to the beekeeper whose hive the swarm had obviously issued from, if that information is reasonably known.
Where I live, this means that when you're called out to a swarm that was reported by the public, you have to try and figure out (by e.g. asking the neighbours) if anyone in the vicinity keeps bees, and if so, then the question is whether the swarm is theirs (and if so, then they have to catch the swarm, and if they don't want to, it means that they abandon their claim to the swarm). The swarming rules in our region are not applied as strictly as they were in the past, because at the time when the rules were written, swarming was a normal part of beekeeping, but these days most local beekeepers try to avoid swarming by making artificial swarms.
Trying to be a good neighbor.
Well, if you are in a compromising mood, try to find out how his queen is marked, see if you can (re-)mark your queens differently, and tell him that if there is a swarm, he can have it if the queen is unmarked or if it turns out to be his queen. It would also be a good idea to make sure he keeps a close watch on his hive so that you know that he will know for certain whether his hive had actually swarmed.