James and Oliver Phelps Say Filming Wizards of Baking on the Harry Potter Set Felt Like ‘Returning to High School’ (Exclusive)

James and Oliver Phelps have cooked up some seriously magical nostalgia on the set of their latest TV show, Harry Potter: Wizards of Baking. 

The twin actors, 38, known for portraying Fred and George Weasley in the beloved movie adaptations of the Harry Potter books, have traded pranks for pastries as hosts of the six-episode Harry Potter-themed baking competition on the Food Network, which follows nine teams of bakers as they whip up treats inspired by moments from the series. 

Filmed at Warner Bros. Studios Leavesden — the iconic set where much of the wizarding world in the Harry Potter movie franchise was brought to life — the series finale of Wizards of Baking will air Thursday night and the winners of the competition will be announced.

The brothers recently spoke with PEOPLE about what it was like to return to where the magic began, sharing how filming the finale in the iconic Great Hall of Hogwarts added an extra layer of emotion and nostalgia.

“It was really bizarre,” said James, who played Fred Weasley, of returning to the Great Hall to announce the winner of the baking competition. “I had a quick flashback of 11 years of memories of filming in there, from the first time I ever walked through the doors to filming the in the fourth movie, when they turn into old men and all that. Many memories flashed my mind in a split second. Then, we announced the winner, and some people’s lives changed forever. It was quite a cool moment to be a part of.”

The Great Hall holds a special place in the hearts of Potterheads all over — the site of many a back-to-school feast (and perhaps a magical battle or two) — making it a fitting place to stage the finale of Wizards of Baking. For the Phelps’, it also holds considerable memories from their youth.

“Imagine if you went back to your high school,” James joked about returning to Leavesden. “You have so many memories from there, from years ago, and it’s got a lick of paint now, and it looks a lot nicer than it used to, because the studios are literally the best studios in the world there … Being able to share that with Bonnie Wright, and Evanna Lynch and Warwick Davis — we had a lot of reminiscing whilst we were there.”

As James noted, the brothers weren’t the only Harry Potter alumni to make appearances on the baking show. Wright, who played the twins’ on-screen sister, Ginny Weasley, made a guest judge appearance, as did Lynch, who played Luna Lovegood, and Davis, who played Professor Filius Flitwick.

For Oliver, returning to the sets wasn’t just about reminiscing on the past but also creating new memories — sharing that he brought his oldest daughter to the set for some of the show’s filming.

“I never thought I’d ever be in there with my daughter, you know, so stuff like that was really cool to do,” he said, though he admitted that his daughter was less enthralled with the Potter of it all than she was with seeing the towering final cakes.

“She’s not really too familiar with any of the Potter films or anything, but she’s really into baking shows and really loves the creative side of it. She was more interested in seeing the final cakes than seeing the Great Hall, really,” he said.

For the bakers on the show, unlike Oliver’s daughter, the twins said, both the baking and Harry Potter apsects of the show are special.

“The bakers are big fans, so it means a lot to them,” James told PEOPLE. “We made a point of saying to them, ‘You are the first people to create a bake like this here.’ The only other people that have ever cooked for the Great Hall was in the movies … so this is an honor in itself. And they completely knew that.”

For the brothers, the magic of the original series has yet to fade.

“When you see what it means to the different fan base, it never gets old,” Oliver said. He also shared a lighthearted moment from set, that revealed just how seriously people take Harry Potter.

“On the day we were filming, we walked up to the head table and just pulled up a chair and sat down,” he recalled. “You could see the guys who work at the studio tour going like, ‘Whoa, whoa! No, you can’t.’ It was as if we’d just gone to the British Museum and just pulled out a sarcophagus and just jumped in. So some things never change in that regard.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

The popular eight-movie franchise starring Daniel Radcliffe, Rupert Grint and Emma Watson may have wrapped in 2011, but plenty of new Potter media has emerged in the following years. Warner Bros. Discovery recently confirmed their plans for a brand-new Harry Potter series based on the best-selling novels.

The Harry Potter: Wizards of Baking finale will air on Thursday, Dec. 19, at 8 p.m. ET/PT on Food Network.