I mirror your brain across the median plane, but don’t change connectivity with the rest of your body (a nerve coming out of the left hemisphere to the right side of the body now comes out of the right hemisphere but still connects to the right side of the body).
I specified that the connectivity between your brain and the rest of the body is unchanged, and the connectivity of the brain itself is unchanged, so ideally nothing would actually change (and thus your memories would not be mirrored).