The wise men came when Jesus was a “young child” not when he was a newborn, plus the wise men did not come to the manger where he was born but to Joesph and Mary’s house. Here is the verse where it says it in the the new testament.

11 ¶ And when they were come into the house, they saw the young child with Mary his mother, and fell down, and worshipped him: and when they had opened their treasures, they presented unto him gifts; gold, and afrankincense, and myrrh.

Jesus wasn’t actually born December 25th, the Church just synchronized His birth with a close Pagan holiday to make it popular.

The Wise Men got there a few months later.

According to the “official” Christian lore, Jesus was born during the night of December 25th (Christmas). The Wise Men got there on January 6th, the feast today we call “Epiphany”.