Perhaps having the key caused the gate to treat the party in a different way than Magus.
Or the party was treated differently due to the big amount of time travels they had been through.
Or this special gate didn't have a standard time moment as target. Perhaps the party fell immediately and got to 65,000,000 BC while Magus managed to keep himself from time travelling (for a few seconds) causing the gate to change its target.
Or, again since this gate was special, it couldn't "connect" to the end of time, so it split its members to two different time periods. It sent three of them (the maximum) to 65,000,000 BC and the one left to 12,000 BC. The three who were sent to 65,000,000 BC were the party, maybe because they were closer to each other (Magus was standing on the other side of the gate) or ...randomly.