Onlookers are said to have heard the remark while the Duke was walking to Sandringham‘s St Mary Magdalene Church on New Year‘s Eve.
The 96-year-old reportedly made the comment when he spotted a tall man with a long ginger beard.
Image: The Queen was also at the church service
A witness, who claimed to have overheard the incident, was quoted by The Sun as saying: “The man appeared to be his with wife and child. Philip was wishing lots of people a Happy New Year and then he spotted this guy with his distinctive beard.
“He pointed at him in a funny way and turned to one of his Royal bodyguards, saying: ‘Is that a terrorist?‘
“He was obviously having a little joke, but he said it within earshot of the man who burst out laughing and appeared to find the whole thing hilarious. I think Philip knew he was going to be overheard.”
Video: Straight-talking Duke in his own words
Buckingham Palace declined to comment on the claims.
The Queen, Princess Anne and Prince Edward also attended the morning service at the Norfolk church, where the Royals traditionally spend Christmas.
Prince Philip stepped down from Royal duties earlier this year after decades of service and more than 22,000 engagements.
Video: Prince Philip retires: A retrospective
Past comments that have caused controversy include asking a group of Aboriginal Australians in 2002: “Do you still throw spears at each other?”
He also told British students in China in 1986: “If you stay here much longer you‘ll all be slitty-eyed”.