At approximately 17:30 UTC, @terencechain observed their clock was adjusted to 4 hours into the future. Shortly after, our pagers started going off, and then user reports started to flood in on discord.
Something was really wrong.
2/n
-
-
Afficher cette discussion
-
We decided to disable the roughtime clock sync by default and replace it with an opt-in feature flag. This prevents the same type of issue to happen on a global scale and now roughtime results are reported as an FYI rather than an automatic clock adjustment. 4/n
Afficher cette discussion -
This graph represents the period in time that Prysm nodes were reporting a clock skew of over 2 seconds. It lasted approximately 90 minutes between 17:30 to 19:00 UTC. Note: this screenshot is in PST. 5/npic.twitter.com/Z0vxvnDbaw
Afficher cette discussion -
So now we look at the data and ask “what roughtime server was giving us such a hard time?”. The data shows that all of the servers were reporting 0.1s offset or less. So how did we end up with a 4 hour offset?? 6/npic.twitter.com/ejqQllmRKZ
Afficher cette discussion -
We are still trying to answer that question! There must be some bug in the roughtime delta calculation and we hope to uncover it soon. Regardless of the results of the investigation, we think that making automatic clock adjustments should be opt-in or maybe removed entirely 7/n
Afficher cette discussion -
Read our full post mortem report and follow the report for updates as we dig deep into the root cause analysis. Uncovering these issues is why we test before mainnet. Having alternative client options benefits users in this type of scenario. https://docs.google.com/document/d/11RmitNRui10LcLCyoXY6B1INCZZKq30gEU6BEg3EWfk/edit … 8/8
Afficher cette discussion
Fin de la conversation
Nouvelle conversation -
-
-
get that bug lads
Merci. Twitter en tiendra compte pour améliorer votre fil. SupprimerSupprimer
-
-
-
I wanted to say that Prysm team did a fantastic job at tackling the issue quickly and swiftly. This also shows the power of
#Decentralization no single point of failure for the network#Ethereum 2.0$ETH#ETH - Fin de la conversation
Nouvelle conversation -
Le chargement semble prendre du temps.
Twitter est peut-être en surcapacité ou rencontre momentanément un incident. Réessayez ou rendez-vous sur la page Twitter Status pour plus d'informations.