Yes, definately get rid of the excluder until they have significant drawn comb. This is when it is called a honey excluder.
Once they are working it and it is part of the hive, then it will work better as a queen excluder and you can put it back, or at that point it will probably be all honey anyway and you won't need. If you have a frame of drawn comb with brood or honey to put in there, that will work even better.
Rick