JavaScript is not available.

May 27, 2022

And that was it for today! Note that we'll have a CL call next Thursday, a Merge Community Call next Friday (github.com/ethereum/pm/is) and AllCoreDevs two weeks from today, June 10 at 14:00 UTC

Show this thread

5:20 PM · Jun 10, 2022

Replying to

and

First on the call, we discussed the Ropsten Merge, which happened earlier this week.

@dannyryan

gave an update, based in part on

@parithosh_j

's analysis from earlier this week:

Quote Tweet

parithosh |

@parithosh_j

Jun 8, 2022

Summary of the Ropsten testnet merge:

Show this thread

Pre-merge, some CL clients had issues tracking deposits to the beacon chain. This was because they made specific assumptions around block times which did not hold under Ropsten's chaotic block times due to its variable hash rate as a testnet.

These issues were all fixed before hitting TTD and deposits were all processed normally pre-merge.

After The Merge on Ropsten, we saw a ~14% participation drop on the network. Danny has tweeted about this:

Quote Tweet

dannyryan

@dannyryan

Jun 9, 2022

~14% of validators had downtime at the transition 1. 9% -- nimbus-team config issue, fixed with CLI change and redeploy 2. 1.8% -- known concurrency bug in nethermind, required a reboot for some nodes 3. 2.5-3% -- nimbus-besu web-sockets problem, swapped to use http 2/3

Show this thread

With the tweaks mentioned in the above tweet, participation rate went back up to ~99%, which is similar to mainnet. It's now a bit lower as teams are testing fixes and running other tests on live validators.

We've also started running

@vdWijden

's fuzzer on Ropsten to make sure we have a good transactional load on the network.

We dove into the client issues in more details on the call. Again, strongly recommend watching the livestream to get all the nuance.

For

the concurrency issue occurred because, at TTD, some nodes would receive new blocks both from the EL p2p network _and_ their CL client. This was made worse on Ropsten because of the high uncle block rate due to the bad mining conditions.

The team is currently working on a fix for the issue 🛠

The

issue had to do with the interaction of websockets, JWT tokens and missing eth_* API endpoints exposed at a specific port.

The Nimbus team shared a workaround with Besu which is working now, but it requires disabling JWT. The Besu team is currently working on a proper fix for the issue.

In addition to issues leading to lower validator participation, there was also an issue where

@ErigonEth

was producing blocks with 0 transactions in it.

This specific bug was fixed, but the "deeper" cause is that Erigon does not support PoW mining. This means their code for block creation isn't s robust as other parts of the client and they are now focusing heavily on that part of the codebase.

Next up, we discussed an issue around CLs asking ELs for blocks before these have had time to create a block. There was a lot of talk about the intricacies of this interaction on the call (

@ajsutton

even showed up!) so, again, recommend watching the livestream for the details!

In short, some CLs sometime ask ELs for a block before these have had a time to create one. Per the spec, the correct behaviour is for clients to return an empty block. This obviously isn't ideal because the EL can usually create _some_ block pretty quickly.

Because of this, Geth & Nethermind have implemented a workaround where, if this happens, they simply create a block "ASAP" and send that back. This isn't great, because it doesn't fix the root cause of CLs sending asking for the block too soon.

Extend `PayloadAttributes` with configurable `gas_limit` · Issue #29 · ethereum/builder-specs

Again, most CLs have this fixed already, and others are focusing on it now. Once this is fixed, ELs will likely revert this functionality to instead follow the spec. In short, when they get asked for a block, they'll stop iterating on its construction, seal it, and send it back!

Next up,

had a change to the Engine API he wanted feedback on: github.com/ethereum/build In short, this would allow CLs to pass a specific gas limit as part of Engine API calls, which would be useful for external block builders.

github.com

If we refer to the current execution Engine API spec for the build inputs: https://github.com/ethereum/execution-apis/blob/main/src/engine/specification.md#payloadattributesv1 we see that there is ...

To unpack that a bit more: post-merge, validators control the gas limit on Ethereum, like miners do today. This parameter is set as part of the EL, when producing a block. Each time a block is chosen, you can raise/lower the gas limit by 1/1024th, or leave it as is.

If you assume that a staker is the one producing their block, then the staker passes their desired gas limit to their EL, and that's it! Things get more complex in a world where the staker may not be building the block themselves!

In practice, this is likely due to MEV. Validators who want to benefit from MEV fees need to run additional software to receive these blocks from searchers/relays.

(side note: the entire MEV design around the merge is out of scope, but

@lightclients

had a great thread about it recently twitter.com/lightclients/s)

Quote Tweet

@lightclients

Jun 6, 2022

After the merge, there will be a much larger faction of "at home" block producers than we've seen under proof-of-work. In order for these independent stakers to extract MEV, we need to think differently about our approach to block production (thread).

Show this thread

So, for external block builders to be able to follow validators' gas limit preferences, it would be useful to have them be passed explicitly as part of Engine API calls. This is what the proposed change does :-)

We agreed on the call that this should _not_ come as a change to the current API, but rather as a separate version for it. There were also questions about whether other info should be passed to builders, such as the `extraData` field.

Everyone seems to agree this is useful, but there are still some details to iron out, which we'll keep doing over the coming weeks!

After this, we discussed the launch of the Sepolia Beacon Chain. With Ropsten now merged, Sepolia also needs a PoS chain to merge with 😁 The configs have been mostly finalized over the past week: github.com/ethereum/pm/is

Similarly to Clique, the validator set on Sepolia will be permissionned. The Beacon Chain will go live in 10 days, and quickly run through Altair + Bellatrix with an artificially high TTD. When we are ready to merge Sepolia, we will perform a TTD override, like for Ropsten.

After this, I asked the various EL teams what they'd like to see before being confident in moving forward with testnets. Once more: there was plenty shared on the call, these are just rough highlights, please watch the livestream 📺!

Nethermind went first: for them, it was seeing hive/kurtosis tests all passing, block proposal issues fixed, and obviously having their current concurrency issue fully resolved. They said they'd want at least 1/2 of the testnets to have code which is basically final.