v0.8.4

Summary

This release is optional and contains minor fixes, updates to logging, and devops improvements.

• Latency observability into encoding and retrieval endpoint
• Reduce metrics cardinality
• EigenDA client:
  • Logging update
  • Ability to initialize without configuring signer
• Improvement in protobuf compilation & docker build pipeline

NOTE: The encoding optimization introduced in v0.8.0, which is not compatible with the previous versions, will be enabled on 10/13 11AM PDT. Please ensure that DA node is upgraded to v0.8.0+ by 10/11. This breaking change only affects DA nodes and does not affect rollups or any users posting blobs.

What's Changed

• chore: refactor to single multistage dockerfile with bake file by @samlaf in #754
• feat: dockerfile for deploying contracts by @samlaf in #755
• Lightnode docker by @cody-littley in #726
• Bump actions/upload-artifact action by @ian-shim in #758
• [ci] Generate unit test coverage on PR by @ian-shim in #759
• Add registry and repository variables to docker bake for devops builds by @pschork in #760
• Fix docker-bake issues with devops builds by @pschork in #761
• doc: remove duplicate word by @xiaobei0715 in #753
• DAS protobuffs by @cody-littley in #710
• chore: cleaner eigenda_client logs by @samlaf in #764
• Make the latency accounting more accurate for encoder by @jianoaix in #766
• Lightnode kv store by @cody-littley in #721
• Add headers for routing on encoding streamer by @dmanc in #765
• 'make protoc' now utilizes docker workflow by default. by @cody-littley in #748
• Reduce the operator latency metric cardinality by @jianoaix in #768
• Instrument the retrieval latency stages by @jianoaix in #771
• rm sig check by @bxue-l2 in #770
• Fix docker-publish-release docker bake by @pschork in #772

New Contributors

• @xiaobei0715 made their first contribution in #753

Full Changelog: v0.8.3...v0.8.4

[Holesky] v0.8.4-rc.0

Summary

This pre-release is optional and contains minor fixes, updates to logging, and devops improvements.

• Latency observability into encoding and retrieval endpoint
• Reduce metrics cardinality
• EigenDA client:
  • Logging update
  • Ability to initialize without configuring signer
• Improvement in protobuf compilation & docker build pipeline

NOTE: The encoding optimization introduced in v0.8.0, which is not compatible with the previous versions, will be enabled on 10/13 11AM PDT. Please ensure that DA node is upgraded to v0.8.0+ by 10/11. This breaking change only affects DA nodes and does not affect rollups or any users posting blobs.

What's Changed

• chore: refactor to single multistage dockerfile with bake file by @samlaf in #754
• feat: dockerfile for deploying contracts by @samlaf in #755
• Lightnode docker by @cody-littley in #726
• Bump actions/upload-artifact action by @ian-shim in #758
• [ci] Generate unit test coverage on PR by @ian-shim in #759
• Add registry and repository variables to docker bake for devops builds by @pschork in #760
• Fix docker-bake issues with devops builds by @pschork in #761
• doc: remove duplicate word by @xiaobei0715 in #753
• DAS protobuffs by @cody-littley in #710
• chore: cleaner eigenda_client logs by @samlaf in #764
• Make the latency accounting more accurate for encoder by @jianoaix in #766
• Lightnode kv store by @cody-littley in #721
• Add headers for routing on encoding streamer by @dmanc in #765
• 'make protoc' now utilizes docker workflow by default. by @cody-littley in #748
• Reduce the operator latency metric cardinality by @jianoaix in #768
• Instrument the retrieval latency stages by @jianoaix in #771
• rm sig check by @bxue-l2 in #770

New Contributors

• @xiaobei0715 made their first contribution in #753

Full Changelog: v0.8.3...v0.8.4-rc.0

v0.8.3

Summary

This release is optional for users and operators to upgrade.

Notable changes

• Performance improvement: Zero serialization of chunks after being encoded in the dispersal flow, eliminating 20s to minutes overhead on dispersal latency
• Performance observability: Added metrics for blob age, blob encoding latency and system throughput
• Robustness fixes:
  • Added blob state transition invariant checking to make sure blob state changes are correct
  • Added blob state recovery for handling failed blobs stuck in DISPERSING state

What's Changed

• Downlevel more encoding streamer error log by @ian-shim in #689
• Check if blobs are expired before returning by @mooselumph in #695
• Add log for chunk encoding format by @jianoaix in #696
• Add BlobMinibatchMapping in minibatch store by @ian-shim in #683
• Create blob minibatch mappings by @ian-shim in #687
• [BatchConfirmer] Apply state transitions for full batches by @ian-shim in #691
• [1/N][zero serialization] Create data structure for passing encode chunks by @jianoaix in #690
• Add an API to fetch blobs from a given batch header hash by @dmanc in #688
• Update README.md by @Ocheretovich in #694
• [node] AttestBatch endpoint by @ian-shim in #676
• Split blob writer code out of larger PR. by @cody-littley in #685
• [2/N][zero serialization] Make Batcher operate on chunks without ser/deser by @jianoaix in #700
• use insecure credential by @bxue-l2 in #711
• Simplify minibatch data model by @ian-shim in #707
• [node] Refactor blob expiration key by @ian-shim in #693
• Compress signatures in minibatch response by @ian-shim in #699
• [node] Make GetBlobHeader compatible with minibatches by @ian-shim in #698
• [1/N][GPU encoder] Add benchmarking code and refactor encoding module by @dmanc in #715
• Add GetBulkBlobMetadata method in BlobMetadataStore by @ian-shim in #703
• Perf observability: Add blob age at key stages in blob's lifecycle by @jianoaix in #719
• Perf observability: Encoder data throughput and blob size breakdown by @jianoaix in #716
• Perf observability: Add blob encoding latency at client side by @jianoaix in #722
• Fix stringification of uint8 by @jianoaix in #723
• [node] Make GetChunks compatible with minibatches by @ian-shim in #712
• Send blob headers to all quorums on minibatch dispersal by @ian-shim in #717
• [3/N][zero serialization] Make Encoder produce and pass chunks in desired format by @jianoaix in #735
• [4/N][zero serialization] Add fallback to the encoder client when it cannot tell chunk encoding format by @jianoaix in #738
• More fixes of quorum (uint8) stringification by @jianoaix in #731
• Perf observability: Add attestation and encoding throughput by @jianoaix in #725
• Created blob verifier worker. by @cody-littley in #708
• Add dispersing transistion check by @bxue-l2 in #741
• Recover failed blobs in encoding streamer by @ian-shim in #733
• Fix: Append metadata when constructing batch by @ian-shim in #746
• Minor updates to error logs by @ian-shim in #747
• Created dockerized workflow for building protobufs. by @cody-littley in #734
• chore(inabox): multiplex inabox logs to stdout to help debug easier by @samlaf in #752
• Expose public ip service option on node plugin by @pschork in #756

New Contributors

• @Ocheretovich made their first contribution in #694

Full Changelog: v0.8.2...v0.8.3

[Holesky] v0.8.3-rc.0

Summary

This pre-release is optional for users and operators to upgrade.

Notable changes

• Performance improvement: Zero serialization of chunks after being encoded in the dispersal flow, eliminating 20s to minutes overhead on dispersal latency
• Performance observability: Added metrics for blob age, blob encoding latency and system throughput
• Robustness fixes:
  • Added blob state transition invariant checking to make sure blob state changes are correct
  • Added blob state recovery for handling failed blobs stuck in DISPERSING state

What's Changed

• Downlevel more encoding streamer error log by @ian-shim in #689
• Check if blobs are expired before returning by @mooselumph in #695
• Add log for chunk encoding format by @jianoaix in #696
• Add BlobMinibatchMapping in minibatch store by @ian-shim in #683
• Create blob minibatch mappings by @ian-shim in #687
• [BatchConfirmer] Apply state transitions for full batches by @ian-shim in #691
• [1/N][zero serialization] Create data structure for passing encode chunks by @jianoaix in #690
• Add an API to fetch blobs from a given batch header hash by @dmanc in #688
• Update README.md by @Ocheretovich in #694
• [node] AttestBatch endpoint by @ian-shim in #676
• Split blob writer code out of larger PR. by @cody-littley in #685
• [2/N][zero serialization] Make Batcher operate on chunks without ser/deser by @jianoaix in #700
• use insecure credential by @bxue-l2 in #711
• Simplify minibatch data model by @ian-shim in #707
• [node] Refactor blob expiration key by @ian-shim in #693
• Compress signatures in minibatch response by @ian-shim in #699
• [node] Make GetBlobHeader compatible with minibatches by @ian-shim in #698
• [1/N][GPU encoder] Add benchmarking code and refactor encoding module by @dmanc in #715
• Add GetBulkBlobMetadata method in BlobMetadataStore by @ian-shim in #703
• Perf observability: Add blob age at key stages in blob's lifecycle by @jianoaix in #719
• Perf observability: Encoder data throughput and blob size breakdown by @jianoaix in #716
• Perf observability: Add blob encoding latency at client side by @jianoaix in #722
• Fix stringification of uint8 by @jianoaix in #723
• [node] Make GetChunks compatible with minibatches by @ian-shim in #712
• Send blob headers to all quorums on minibatch dispersal by @ian-shim in #717
• [3/N][zero serialization] Make Encoder produce and pass chunks in desired format by @jianoaix in #735
• [4/N][zero serialization] Add fallback to the encoder client when it cannot tell chunk encoding format by @jianoaix in #738
• More fixes of quorum (uint8) stringification by @jianoaix in #731
• Perf observability: Add attestation and encoding throughput by @jianoaix in #725
• Created blob verifier worker. by @cody-littley in #708
• Add dispersing transistion check by @bxue-l2 in #741
• Recover failed blobs in encoding streamer by @ian-shim in #733
• Fix: Append metadata when constructing batch by @ian-shim in #746
• Minor updates to error logs by @ian-shim in #747
• Created dockerized workflow for building protobufs. by @cody-littley in #734

New Contributors

• @Ocheretovich made their first contribution in #694

Full Changelog: v0.8.2...v0.8.3-rc.0

v0.8.2

Summary

This is a patch release to fix a bug related to the blob state transition (#741). Without this fix, the blob may have wrong state transition from Confirmed back to Dispersing, which should never happen.

Users/rollups: No action needed. This is a system internal fix and should just work for users.

Operators: Not affected. No action needed.

Full Changelog: v0.8.1...v0.8.2

v0.8.1

Summary

This is a patch release to fix a bug related to gRPC dial option (#711) in dispersal client library in v0.8.0, so users can retrieve the 4MiB blobs correctly.

Users/rollups: if you are using dispersal client to retrieve 4MiB blobs, please upgrade to v0.8.1. No impact otherwise.

Operators: no impact, and no action is needed.

Full Changelog: v0.8.0...v0.8.1

v0.8.0

Summary

This release is optional for now, but strongly recommended.

In particular, it’ll become required for operators to upgrade to v0.8.0 or higher before 2024/09/20 on Mainnet.

Two changes are needed to complete the upgrade:

• Upgrade image to v0.8.0 or higher
• Set the node environment variable “NODE_SRS_LOAD=8388608”

Both are included in operator setup repo: https://github.com/Layr-Labs/eigenda-operator-setup/releases/tag/v0.8.0

Notable features

• EigenDA users - Max Blob size raised from 2MiB to 4MiB
  • Not enabled for custom quorums with less than 8 operators
• Operators - Gnark chunk encoding to reduce data size by about 30%
  • Not enabled for now
  • Scheduled to enable after operators upgrade to v0.8.0 or later version
• NodeInfo API at Node: Operators will start making the Node software version information available to the Disperser
• Churn approval expiration reduced from 60 mins to 15 mins: Churner can approve a churn request every 15 mins now

Other notable changes

• Fixed a bug in blob state transition in Batcher (reset DISPERSING blobs back to PROCESSING at Batcher start)
• Improved performance logging

Extra details for blob size limit and chunk encoding

Increasing max blob size is a notable feature enabled in the release. This feature allows EigenDA users to send blobs of 4MiB, previously 2MiB. For operators, they will need to update the NODE_SRS_LOAD to support larger blobs. The resource utilization on the node is expected to be almost identical under the same level of traffic.

This release also contains chunk encoding changes that are not enabled for now, but are scheduled to enable at some future release, to give operators ample time to upgrade. When enabled after the release post 2024/09/20, the nodes with versions before v0.8.0 will not be able to parse the chunks from Disperser.

We will make an announcement in that future release when the changes are enabled.

What's Changed

• Simplify the bytes decoding of chunks by @jianoaix in #607
• Pull ejector out of dataapi by @ian-shim in #608
• Update eigenlayer-middleware by @0x0aa0 in #611
• Record args in mock transactor by @ian-shim in #616
• [1/N] Chunk encoding optimization: Support encoding format with gnark by @jianoaix in #617
• fix dataapi docker build by @ian-shim in #618
• Use variable rate interval for blob thoughput metrics based on time interval by @pschork in #621
• Add configuration setting for high-level EigenDA client for returning as soon as the blob has confirmed by @teddyknox in #620
• Log the batch validation latency breakdown by @jianoaix in #619
• [TxnManager] Make pk wallet compatible with broadcast check by @ian-shim in #626
• Log request serialization time by @jianoaix in #625
• [inabox] Set BATCHER_FINALIZATION_BLOCK_DELAY to 0 by @ian-shim in #631
• [4/N] Chunk encoding optimization: Add bundle encoding by @jianoaix in #632
• Adds NodeInfo endpoint to retriever and dispersal services for observability by @pschork in #630
• Refactor cpu prover by @bxue-l2 in #629
• [2/N] Chunk encoding optimization: bench mark the gnark based encoding by @jianoaix in #624
• [3/N] Chunk encoding optimization: Add support of new encoding format… by @jianoaix in #628
• [minibatch] Add feature flag on batcher by @ian-shim in #634
• Ignore project files for the Jetbrains GoLand IDE. by @cody-littley in #639
• Fix fft_fr.IsPowerOfTwo(). (#83) by @cody-littley in #638
• Remove ejector from dataapi by @ian-shim in #633
• [5/N] Chunk encoding optimization: Add support of new encoding at Node by @jianoaix in #636
• Refactor dispatcher timeout by @ian-shim in #641
• Collect Node Info during reachability check if port scan was successful by @pschork in #640
• Add a metric for DB writing throughput by @jianoaix in #644
• Inabox readme improvements by @cody-littley in #643
• Revert "Refactor dispatcher timeout" by @ian-shim in #645
• Minibatch store interface by @ian-shim in #635
• Update node interface to enable minibatching by @ian-shim in #646
• [minibatch] Create minibatch from encoding streamer by @ian-shim in #637
• Dispatcher method to disperse blobs to node via StoreBlobs endpoint by @ian-shim in #648
• Generate inclusion proof with blob index instead of hash by @ian-shim in #649
• Clean up EnableDualQuorums flag by @ian-shim in #656
• Add Minibatcher by @ian-shim in #655
• Fix typos by @nnsW3 in #659
• Code hygiene: return error instead of bool by @jianoaix in #660
• Doc hygiene: api docs, repo readme, codebase structure by @jianoaix in #662
• [dispatcher] Add method to send AttestBatch request to nodes by @ian-shim in #664
• [minibatcher] Fullbatch state management by @ian-shim in #665
• Minibatch dynamodb client by @pschork in #652
• [6/N] Chunk encoding optimization: Disperser/Retriever support for new chunk encoding by @jianoaix in #650
• Average blob throughput over 9m window by @pschork in #667
• Enable shadow PutItem blob metadata requests to separate dynamo table for minibatch preprod testing by @pschork in #669
• Added metrics utility for generator. by @cody-littley in #673
• Added test utilities. by @cody-littley in #672
• add configurable max blob size by @bxue-l2 in #675
• Log blob size on request by @ian-shim in #678
• [batcher] Recover blob state by @ian-shim in #657
• add grpc recv buffer by @bxue-l2 in #679
• Configuration for new generator functionality. by @cody-littley in #674
• Downgrade encoder error logs by @ian-shim in #682
• [7/N] Chunk encoding optimization: Node support for mixed encoding by @jianoaix in #654
• Added blob table data structure. by @cody-littley in #677
• Keep batch state in Minibatcher by @ian-shim in #668
• Code hygiene: more properly name the chunk encoding format field by @jianoaix in #661
• Fix the operator stake share basis point calculation by @jianoaix in #658
• [node] Add StoreBlobs endpoint for minibatch streaming by @ian-shim in #671
• Add BatchConfirmer by @ian-shim in #670
• Add batch read operation in dynamodb client by @ian-shim in #684
• Adjust the churn approval interval by @jianoaix in #651
• Downlevel more encoding streamer error log by @ian-shim in #689
• Check if blobs are expired before returning by @mooselumph in #695
• Add log for chunk encoding format by @jianoaix in #696

New Contributors

• @nnsW3 made their first contribution in #659

Full Changelog: v0.7.4...v0.8.0

[Holesky] v0.8.0-rc.1

Summary

This pre-release is optional for now, but strongly recommended.

In particular, it’ll become required for operators to upgrade to v0.8.0 or higher before 2024/09/15 on testnet.

Two changes are needed to complete the upgrade:

• Upgrade image to v0.8.0 or higher
• Set the node environment variable “NODE_SRS_LOAD=8388608”

Both are included in release https://github.com/Layr-Labs/eigenda-operator-setup/releases/tag/v0.8.0-rc.1

Notable features

• EigenDA users - Max Blob size raised from 2MiB to 4MiB
  • Not enabled for custom quorums with less than 8 operators
• Operators - Gnark chunk encoding to reduce data size by about 30%
  • Not enabled for now
  • Scheduled to enable after operators upgrade to v0.8.0 or later version
• NodeInfo API at Node: Operators will start making the Node software version information available to the Disperser
• Churn approval expiration reduced from 60 mins to 15 mins: Churner can approve a churn request every 15 mins now

Other notable changes

• Fixed a bug in blob state transition in Batcher (reset DISPERSING blobs back to PROCESSING at Batcher start)
• Improved performance logging

Extra details for blob size limit and chunk encoding

Increasing max blob size is a notable feature enabled in the release. This feature allows EigenDA users to send blobs of 4MiB, previously 2MiB. For operators, they will need to update the NODE_SRS_LOAD to support larger blobs. The resource utilization on the node is expected to be almost identical under the same level of traffic.

This pre-release also contains chunk encoding changes that are not enabled for now, but are scheduled to enable at some future release, to give operators ample time to upgrade. When enabled after the release post 2024/09/15, the nodes with versions before v0.8.0 will not be able to parse the chunks from Disperser.

We will make an announcement in that future release when the changes are enabled.

What's Changed

• Simplify the bytes decoding of chunks by @jianoaix in #607
• Pull ejector out of dataapi by @ian-shim in #608
• Update eigenlayer-middleware by @0x0aa0 in #611
• Record args in mock transactor by @ian-shim in #616
• [1/N] Chunk encoding optimization: Support encoding format with gnark by @jianoaix in #617
• fix dataapi docker build by @ian-shim in #618
• Use variable rate interval for blob thoughput metrics based on time interval by @pschork in #621
• Add configuration setting for high-level EigenDA client for returning as soon as the blob has confirmed by @teddyknox in #620
• Log the batch validation latency breakdown by @jianoaix in #619
• [TxnManager] Make pk wallet compatible with broadcast check by @ian-shim in #626
• Log request serialization time by @jianoaix in #625
• [inabox] Set BATCHER_FINALIZATION_BLOCK_DELAY to 0 by @ian-shim in #631
• [4/N] Chunk encoding optimization: Add bundle encoding by @jianoaix in #632
• Adds NodeInfo endpoint to retriever and dispersal services for observability by @pschork in #630
• Refactor cpu prover by @bxue-l2 in #629
• [2/N] Chunk encoding optimization: bench mark the gnark based encoding by @jianoaix in #624
• [3/N] Chunk encoding optimization: Add support of new encoding format… by @jianoaix in #628
• [minibatch] Add feature flag on batcher by @ian-shim in #634
• Ignore project files for the Jetbrains GoLand IDE. by @cody-littley in #639
• Fix fft_fr.IsPowerOfTwo(). (#83) by @cody-littley in #638
• Remove ejector from dataapi by @ian-shim in #633
• [5/N] Chunk encoding optimization: Add support of new encoding at Node by @jianoaix in #636
• Refactor dispatcher timeout by @ian-shim in #641
• Collect Node Info during reachability check if port scan was successful by @pschork in #640
• Add a metric for DB writing throughput by @jianoaix in #644
• Inabox readme improvements by @cody-littley in #643
• Revert "Refactor dispatcher timeout" by @ian-shim in #645
• Minibatch store interface by @ian-shim in #635
• Update node interface to enable minibatching by @ian-shim in #646
• [minibatch] Create minibatch from encoding streamer by @ian-shim in #637
• Dispatcher method to disperse blobs to node via StoreBlobs endpoint by @ian-shim in #648
• Generate inclusion proof with blob index instead of hash by @ian-shim in #649
• Clean up EnableDualQuorums flag by @ian-shim in #656
• Add Minibatcher by @ian-shim in #655
• Fix typos by @nnsW3 in #659
• Code hygiene: return error instead of bool by @jianoaix in #660
• Doc hygiene: api docs, repo readme, codebase structure by @jianoaix in #662
• [dispatcher] Add method to send AttestBatch request to nodes by @ian-shim in #664
• [minibatcher] Fullbatch state management by @ian-shim in #665
• Minibatch dynamodb client by @pschork in #652
• [6/N] Chunk encoding optimization: Disperser/Retriever support for new chunk encoding by @jianoaix in #650
• Average blob throughput over 9m window by @pschork in #667
• Enable shadow PutItem blob metadata requests to separate dynamo table for minibatch preprod testing by @pschork in #669
• Added metrics utility for generator. by @cody-littley in #673
• Added test utilities. by @cody-littley in #672
• add configurable max blob size by @bxue-l2 in #675
• Log blob size on request by @ian-shim in #678
• [batcher] Recover blob state by @ian-shim in #657
• add grpc recv buffer by @bxue-l2 in #679
• Configuration for new generator functionality. by @cody-littley in #674
• Downgrade encoder error logs by @ian-shim in #682
• [7/N] Chunk encoding optimization: Node support for mixed encoding by @jianoaix in #654
• Added blob table data structure. by @cody-littley in #677
• Keep batch state in Minibatcher by @ian-shim in #668
• Code hygiene: more properly name the chunk encoding format field by @jianoaix in #661
• Fix the operator stake share basis point calculation by @jianoaix in #658
• [node] Add StoreBlobs endpoint for minibatch streaming by @ian-shim in #671
• Add BatchConfirmer by @ian-shim in #670
• Add batch read operation in dynamodb client by @ian-shim in #684
• Adjust the churn approval interval by @jianoaix in #651
• Downlevel more encoding streamer error log by @ian-shim in #689
• Check if blobs are expired before returning by @mooselumph in #695
• Add log for chunk encoding format by @jianoaix in #696

New Contributors

• @cody-littley made their first contribution in #639
• @nnsW3 made their first contribution in #659

Full Changelog: v0.7.4...v0.8.0-rc.1

hot fix disperser grpc recv buffer

v0.7.4-4

fix

v0.7.4

Summary

This release is optional and contains performance improvements as well as bug fixes.

Notable features that directly affect users are:

• Operators - ARM support: We now publish ARM docker images for the operator node.

• Operators - Improvements to storage: Added performance improvements on the node DB side resulting in 4x performance increase.

Other changes:

• Traffic generator now supports authenticated dispersals
• Fixes to the retriever service
• Improved logging on the operator node
• Preparing rewards integrations at the smart contract layer

What's Changed

• Add decode flag for GetBlob() by reworking codec layer by @teddyknox in #592
• Add nonsigning rate threshold control to ejector by @jianoaix in #601
• Few fixes to retriever server by @ian-shim in #599
• Auth traffic gen by @mooselumph in #566
• Adds support for multi-platform linux/amd64 & linux/arm64 docker images by @pschork in #603
• Add log for request data size breakdown by @jianoaix in #604
• Use accountKey as AccountID by @dmanc in #602
• Fix GitVersion regex to properly identify versioned tags when building in detached HEAD mode by @pschork in #605
• [finalizer] Update confirmation block number in finalizer by @ian-shim in #600
• Optimize the bytes encoding at node by @jianoaix in #606
• Rewards Integration by @0x0aa0 in #588

Full Changelog: v0.7.3...v0.7.4