Daddy said that he could recall everything from the previous times he'd visited, so perhaps it's something about coming into the City that lets them remember? I suppose that would fit, since people have left the City and come back to it with their memories plenty of times before.

I...can't even imagine what it would be like if you were, to be honest. I just can't picture it at all.