Welcome back to our blog series on “What is Danksharding”! In our previous post, we delved into the Ethereum scalability roadmap and discussed its significance in tackling the challenges of scalability. Today, we will continue our exploration by diving into the fascinating world of Sharding and its cutting-edge evolution, Danksharding.
Sharding
Sharding is a term used in databases and distributed system field. When data is stored in a single location, it can become increasingly large as the data grows. This can make data processing cumbersome and the managment of the data itself burdensome. To address this, t continues to grow over time. As a result, it becomes heavy to process this data, and managing the data itself becomes burdensome. Sharding is a technology that splits data into several pieces for storage or processing to mitigate these issues.
- Sharding: A technique that divides data into multiple pieces for storage or processing.
Ethereum’s Sharding
Sharding in Ethereum refers to the idea of dividing the blockchain into several shard chains.
In the diagram below, the main Beacon Chain acts as the Coordination Layer, with several Shard Chains that depend on the Beacon Chain. The proposed idea was to divide the chains into 64 shard chains, each processing transactions separately. For example, if there are 64,000 transactions to be processed, it’s more efficient to have 64 chains that each processes 1,000 transactions than to have one chain processing all 64,000 transactions. The goal of Ethereum sharding was to distribute the processing of transactions and data storage into each shard chain rather than processing everything in a single chain.
Before sharding, when a transaction occurred, all validators had to verify all incoming blocks. After sharding, validators are divided among shard chains, and each validator only needs to verify the shard chain it is belonged to, reducing the amount of data they need to process. This could potentially allow nodes to operate and perform validation with lower hardware specifications. However, the initial idea of dividing the network into 64 shard chains has been abandoned. The new idea that emerged after is called Danksharding.
One of the reasons for moving away from the previous approach and adopting Danksharding is the shift towards a rollup-centric approach to enhance Ethereum’s scalability. As explained, shard chains not only process transaction directly but also are used as storage space. However, with the rollup-centric approach, all data processing happens on Layer 2 rollup hence reducing the need of using shard chains. Also, many other issues arose during the implementation of this sharding method, leading to its abandonment.
Summary
- The idea of splitting the L1 blockchain into several shard chains.
- Transactions and data storage are distributed to each shard.
- Before sharding
- All validators verified all blocks
- After sharding
- Validators only need to verify the shard chain they belong to.
- Lower hardware specifications are sufficient.
However, this method was abandoned ⇒ the focus shifted to rollups.
PBS (Proposer-Builder Separation)
To understand DankSharding, you need to know about PBS first. PBS stands for Proposer-Builder Separation. Simply put, it means separating the roles of proposers and builders.
First, let’s examine what validators do after transitioning to a POS chain. Validators collect various transactions, such as sending Ether or depositing it into DeFi. For example, if they propose a block containing three transactions as the next block, other validators verify whether the transactions contained in the proposed block are valid or not, and vote on it. If it passes, the next block is generated.
Gather transactions. Validate and vote new proposed block. Then create block.
As mentioned earlier, the role of a validator is either proposing blocks or validating those blocks.
Among the validators, a committee is randomly selected. For example, when 150 people are chosen as committee members out of 10,000 validators, one of them is designated as the proposer, and the rest are categorized as attesters. The proposer selects transactions to create a new block and proposes it as the next block. The remaining attesters verify the proposed block and vote on its validity to ensure the generation of the next block. Proposer’s job is much larger than that of attesters, so proposer receives a higher fee.
If the proposer is powerful, which means this proposer has a higher probability to be chosen during the block proposal process, problems can arise during the process of making decentralized decisions in the blockchain.
In such cases, when the proposer collects transactions to propose a new block, they can gain an advantage if they notice a transaction involving a large-scale purchase of Ethereum. Knowing that there is a large-scale purchase of Ethereum implies that the price of Ethereum may change before and after the purchase. Therefore, before the transaction involving the purchase is processed, the proposer can include a transaction in which they buy Ethereum. Later, when the price of Ethereum increases due to the large-scale purchase, the proposer can include a transaction to sell Ethereum without any risk, thus capturing the price difference. These actions are referred to as MEV, which stands for Maximum Extractable Value. Miners can engage in actions that allow them to profit. The centralization of proposers can lead to such unintended consequences.
What a strong validator can do: MEV(Maximal extractable value) centralization issue
To address these issues, the concept of PBS (Proposer-Builder Separation) has been proposed. It suggests dividing the original role of the proposer into two roles: the proposer and the builder. Although it has not been implemented yet, it is a proposed solution that is currently under discussion.
If PBS is introduced, the task of collecting transactions and bundling them into blocks would fall to the builders. These builders can still create blocks as a collection of transactions for their own gain, similar to the MEV mentioned earlier. In this case, there can be multiple builders. Each builder creates their desired block and sends it to the proposers. The proposers then choose one block from the candidates received and send it to the attesters. When the proposer receives multiple blocks to propose, the selection criteria for choosing one block are as follows: the builders also deliver proposer fees along with the block they send to the proposer. From the proposer’s perspective, they will propose the block with the highest potential fees they can receive for the next block.
Amidst discussions to introduce such a model, the concept of Danksharding has emerged.
Problems with the existing sharding model
The shortcomings of the existing sharding model were briefly mentioned earlier as the reason for its abandonment. In the previous model, to process the sharding chains, each separate chain had to process its own transactions, which would then enter the consensus layer of the beacon chain. However, during the process of aggregating data from all 64 shards, there was a problem of nodes becoming heavy as they had to read data from all shards. Additionally, with Ethereum transitioning to Proof-of-Stake (POS), the beacon chain became the consensus layer, creating slots that allowed for block generation every 12 seconds. However, it was challenging for all shards to create their blocks simultaneously every 12 seconds, which made extra technical complexity needed to synchronize everything. According to the rollup-focused roadmap, it is now possible to execute transactions in Layer 2 rollup and only store data in Layer 1 Ethereum. Therefore, there is no longer a need to execute transactions in the shard chain.
- In the process of compiling transactions from each shard, it is necessary to access and read data from all shards. This prerequisite imposes a substantial computational load, presenting a challenge for efficient node operation.
- Another significant issue pertains to the difficulty in synchronising the timing of block production across all shards. Given that we are dealing with 64 shards, coordinating the simultaneous creation of blocks every 12 seconds is a far from trivial task.
- A developing trend in the space is to execute transactions within rollups, with Layer 1 Ethereum being used merely for data storage.
Conclusion: Given these factors, there is a diminishing necessity for transaction execution within the shards themselves.
Danksharding
An improved model from the traditional sharding is called Danksharding. Danksharding is a sharding approach proposed by Dankrad Feist, named after him. The core idea of Danksharding is to avoid creating separate shard chains for transaction execution and data storage, and instead expand the space available for storing data within the consensus layer of the beacon chain.
In the previous discussion, it was mentioned that rollups typically store their data in calldata. This data refers to such information as the account balances and transactions executed within the rollup. Danksharding suggests to create data blobs and store this rollup data there, rather than storing it directly in the calldata.
In the diagram, each data blob contains transactions. Regular tx is the regular transactions that we are familiar with. Data tx, the new method that is not applied yet, suggests to make separate transaction and connect the positions of data blobs.
By incorporating calldata into the data blobs, Danksharding aims to achieve a structure where more data can be included at a lower cost. This enables the goal of storing more data at a cheaper price through the use of data blobs.
Summary
Sharding proposed by Dankrad Feist
- Key ideas:
- Don’t create a separate shard chain for transaction execution and data storage.
- Instead, create shards within the Beacon chain that only store data.
- Use data blob instead of calldata
- Data blob
- Regular TX
- Data TX
- Data blob
- Data blob is not used for transaction execution hence cheaper gas fees required.
- This does, indeed, increase the size of the blocks. This could prompt the question, why not simply augment the existing block size?
- A surge in block size can potentially lead to a centralisation of nodes and validators.
Yes, you’re correct. With the introduction of data blobs in Dank Sharding, the burden of processing the growing amount of data would fall on the builders. As the data blobs increase in size, builders would need to utilize high-performance hardware to handle the computational requirements. On the other hand, proposers and attesters would be less affected by the increase in data size, as their roles primarily involve the validation and proposal processes rather than direct data processing.
- PBS
Vitalik Buterin, the co-founder of Ethereum, recently shared his views on the ‘endgame’ of Ethereum and blockchain networks. He anticipates that the key elements of the future blockchain network will be composed as follows:
- Centralized Block Production
- Decentralized Block Validation
- Strengthened anti-censorship protection
The illustration below shows that as Ethereum expands based on rollups, the block creation process will increasingly become centralized, while block validation will become decentralized. This is why Danksharding has risen as a key element in the roadmap.
Strong Anti-censorship Protection
To strengthen the prevention of transaction censorship, a Censorship Resistance List (CRL) is utilized. The CRL is a list of transactions provided by the proposer to the builders during the block creation process, ensuring that the builders cannot exclude specific transactions.
Decentralized Block Validation
Considering that data cannot be infinitely heavy, methods to make the data blobs lightweight are needed.
The first proposal to lightweight data blobs is the idea of deleting them after a certain period, as defined in the Shard Blob Transactions EIP-4844. This specification is referred to as Proto-Danksharding. Since the implementation of Danksharding requires PBS and support from the consensus layer, the idea of EIP-4844 suggests exploring feasible methods rather than waiting for all the necessary tasks to be completed.
EIP-4844 establishes a specification that allows for the deletion of data blobs after a certain period.
Deleting the data after a certain period may result in the absence of data to be verified during transaction validation. To address this, the concept of Data Availability Sampling (DAS) has been introduced. DAS enables the verification of whether a sampled portion of the data blob represents the entire data. Additionally, the concept of KZG commitment is utilized to verify the existence of specific data within the data blob. This technique enables verification with only a portion of the data.
By using DAS to confirm data availability and leveraging KZG commitment to enable verification with partial data, the verification process can be lightweight and decentralized.
- Methods to lightweight the data blobs
- EIP-4844: Shard Blob Transactions
Consider deleting Blob data after a certain period.
Plan to delete after 30~60 days
(Proto-Danksharding)
Data Blob
- A technique that allows checking the availability of data by sampling only a portion of it.
- It should be designed to allow deletion after a certain period.
- When validating, data may be absent, so it is necessary to check if the data is available. ⇒
DASData Availability Samplin
KZG commitment
Allows verification of the existence of specific data within a Blob.
→ Enables verification through just a subset of data.
Data availability is confirmed through DAS, and KZG Commitment allows verification using just a subset of data, thereby lightening the verification process. ⇒ Decentralization of the verification process
In Conclusion
Ethereum is striving to secure scalability, higher throughput, and lower fees by blending Layer 2 technology based on Rollup and Danksharding techniques. Through these endeavours, Ethereum is expected to evolve into a more mature smart contract platform, in tandem with the advancement of blockchain technology.
