There are three active slots and six reserve slots. There is no point in the game where you can access the switch screen and "empty" a slot, meaning you will always have the max number possible in the active party and no more than four in the reserve.
There is exactly one point in the game where the extra reserve slots are used, but you cannot access the switch screen at that time (Lucca's sidequest).
There are never four active characters. When Robo is following your party around at the end of time, he it literally just tagging along (like the cat does at the fair). If you look closely, you will see he is not moving normally.
---T.Geiger