They'll follow the queen.
If you didn't get them all off the trunk, then likely the queen was still on the trunk, so back the girls troop to her.
If you haven't got a bee brush, get a clean (unused) paint brush (house size - 3-4 inch), a dust brush, even a soft broom and sweep them off. If you're using a heavier/more solid brush than a bee brush, make sure you press it against the trunk and sweep towards the cluster of bees so you roll them all off at once, rather than pushing the brush into the cluster and squishing bees under the brush. There'll be bees in the air, but just make sure you don't leave any bees on the trunk and you can then be pretty sure you'll have the queen.
Don't worry about trying to see the queen - it's a nice bonus if you do, but I never have in a swarm. ONce you get her in the box you'll notice in a few minutes those bees outside the box will start trooping in towards her.