But... even if the timelines were merged, that doesn't mean that they never had existed as different timelines. So the people who has switched between them both, whose body has been modified in order to store that information, that is, Serge and Schala (and the other two people who go with Serge into the tesseract), they should remember.
I think ^_^