what is the hive set-up like?
They will many times throw a prime and several after swarms. If the brood nest is congested, they will swarm. Extra room in other places will not usually fix the issue.
I would give them some time to get straight and add some frames of foundation (checker boarded) in the brood area then keep looking and see if you can find a queen, odds are this hive has a queen in the hive and if you cant find her then your new queen will be killed (wasted time and money) , you could do like Dallas says and cut out the cells as long as you find them all but the question is (is the old queen gone already or is she still in the hive getting ready to go) see a queen getting ready to swarm will stop laying eggs so she can get a little smaller so she can fly so it might take her a few weeks before she starts laying again, I would split this hive and see which split has a queen, put a new queen in the queenless split , after a week or two after they release the new queen, go to the split with the older queen, find the queen and kill her then merge the two splits together (if you dont want anymore hives) or just leave them alone and have a extra hive...
oh and the bee's will some times always follow the queen a short distance then return, dont worry with it that probably happens to most if not all hives but people never see it...