Dōgen

After around an hour it's off by about 5 frames at 23.98 fps

that's pretty close to what I'd expect for a 24-vs-24/1.001 error; are you sure you're not mixing 23.98 and 24 settings anywhere?

Camera and project are both set to 23.98. Not sure about the audio file.

audio file wouldn't have an associated video frame rate

if that's not it, I'd be inclined to blame QuickTime and suggest a more purpose-built audio recorder application like audio hijack for your existing recording, FCPX has a speed adjustment option, so you can speed up your audio by a factor of about 1.001 (±.0005 or so) there

this assumes the drift occurs at a constant rate, anyway if you run into issues in FCP, you can always put it through ffmpeg, like so: ffmpeg -i input.m4a -af atempo=1.001 output.wav

Will give it a shot! Thank you very much!

any luck, out of curiosity?