You are a dice.
The "designated panel" on the ground is responsible for opening the goal. To roll to the designated panel, you need to make sure that the eyes on the panel are the same as the eyes on your top, but you change the eyes every time you roll. So, even if you roll honestly, it may not work. When that happens, I recommend that you start over, choose a different puzzle, or get away from "Suikoro" and play another game, or go outside and change your mind.