Even if something is open source, if it's running on a server you don't control there's no guarantee that the service you're talking to is running that same code as what you've read.
That's the idea, but the problem is verification--there's a black box involved in the process. How do you know the mechanism inside the box produces end-to-end encryption and not merely apparently end-to-end encryption?
There is no black box involved in the process. The Signal app is open source. The whole point of the app is providing end-to-end encryption from client to client. You seem to be confusing end-to-end encryption with transport encryption. It doesn't mean what you seem to think.
It uses authenticated encryption with forward secrecy between instances of the app. It doesn't trust the server. Encrypting connections to the server is not end-to-end encryption. End-to-end means encrypting from one end (Signal app) to the other (Signal app), not to the server.
The app sets up an encrypted session with the other end. It provides forward secrecy rather than using long-lived keys directly for encryption. You either verify the safety number with contacts out-of-band to verify a MITM didn't happen for the initial setup or rely on TOFU.
If you verify the safety number, it starts considering it a secure session and marks it as secure. It will prominently notify you if the encryption has to be renegotiated.
If you don't ever verify a session, it still informs you if it renegotiates (safety number changed).
So, for people who don't care, they still get TOFU and a notice is given if it has to start over. For people willing to put in the minimal effort of scanning the safety number QR code out-of-band, it enforces it.
If you use the encrypted backup/restore, you can move it across phones without losing your data including not requiring new safety numbers. A lot of people don't transfer over their data with the app's backup feature when they switch phones, so their safety numbers change.