Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

In 1999, Po-Shen Loh took home the silver medal as a member of the US International Math Olympiad. Two decades later, much of his life still revolves around math. The team hasn’t been forgotten — if ...