- The Sui network migrated from AWS S3 to Cloudflare R2 for hosting state snapshots, improving user experience and reducing costs.
- Mysten Labs implemented signed and unsigned object store operations and a custom index file to facilitate the transition and enhance reliability.
- The switch to R2 hosting contributed to the scalability and high throughput of the Sui network.
The Sui network relies on a robust infrastructure of validators and full nodes to maintain the highest levels of reliability and uptime, enabling a high throughput scalable blockchain. Running a stateful application like the Sui network requires a seamless failover mechanism and the ability to migrate validators in the event of hardware failures or outages without reprocessing the entire chain history. This is where state snapshots play a crucial role.
State Snapshots: Formal and Database Forms
In the Sui network, state snapshots come in two forms: formal and database snapshots. Formal snapshots contain the minimal state that encompasses all the validator consensus information at the end of an epoch. On the other hand, database snapshots provide a full copy of a node’s database.
Snapshots, however, only become truly useful when they are stored in easily accessible and reliable locations. Initially, the Sui network chose Amazon Web Services’ (AWS) S3 as the ideal backend for snapshot storage, allowing early node operators to share and sync their full nodes or validators quickly. Mysten Labs, the driving force behind the Sui network, took the initiative to host public snapshots on S3, providing a convenient resource for the network participants.
Challenges with S3 Hosting and Growing Snapshots
Unfortunately, hosting an S3 bucket that allowed public downloads of state snapshots turned out to be a less-than-optimal user experience. The format of the state snapshot necessitated the use of the AWS Command Line Interface (CLI) to download multiple files contained within a single snapshot. If users didn’t have pre-existing AWS credentials, they had to employ a sparsely documented incantation of the AWS CLI: aws s3 cp --no-sign-request.
Moreover, the size of Sui’s snapshots grew exponentially due to its ultra-high throughput nature. As a result, downloading a state snapshot from S3 to a node operator’s host could take hours due to the over 800 gigabytes of data involved. This rapid growth led to significant expenses as S3 charges per gigabyte for data transferred out of the service, resulting in substantial monthly hosting costs for the public snapshot resource.
Enter Cloudflare R2: A Game-Changer
To address these challenges, Mysten Labs turned to Cloudflare’s R2, a competitor to AWS S3, with a unique pricing model: zero egress costs. This pricing model made R2 an ideal choice for hosting a dataset like the Sui network’s snapshots, which are frequently fetched by node operators. The absence of egress costs provided a significant advantage over S3, making R2 a clear winner in terms of affordability.
Rather than performing a complete migration from S3 to R2, Mysten Labs decided to make R2 an alternative source while transitioning S3 to a Requester Pays model. This decision was driven by S3’s excellent performance and features, such as global transfer acceleration, which the team didn’t want to relinquish. The migration primarily involved modifying the Sui application to facilitate easy reading from R2, as R2 is API-compatible with S3.
Enabling Permissionless Downloads on R2
To ensure a smooth user experience, Mysten Labs aimed to eliminate the need for users to employ the AWS CLI against R2. Instead, they wanted users to be able to point their tooling at db-snapshot.mainnet.sui.io and read the files hosted there without requiring any authentication. This zero authentication option was critical to the team, as they wanted to make it as easy as possible for anyone to run a Sui full node.
AWS S3 request signing plays a vital role in securely interacting with Amazon S3 resources. For publicly accessible files or objects, it is technically possible to bypass request signing for reading resources, but this capability is not supported in the Rust object store library used by Mysten Labs. Consequently, the team decided to add support for zero authentication in their codebase, allowing users to choose between restoring snapshots from S3 with signed requests (due to the requester pays mode) or from R2 without signing.
Abstracting Object Store Operations
To smoothly switch between signed and unsigned implementations, Mysten Labs introduced abstractions for common object store operations in their codebase. This abstraction layer allowed them to decouple the specific implementation details and seamlessly switch between different cloud providers.
Implementing Signed and Unsigned Implementations
Following the abstraction layer, Mysten Labs implemented both the signed and unsigned implementations of the object store operations. The signed implementation relied on the existing object store library, while the unsigned implementation leveraged the REST APIs of individual cloud providers.
Configuring User Preferences
With the signed and unsigned implementations in place, Mysten Labs incorporated user configuration options to cleanly switch between the two approaches. This flexibility allowed users to choose whether to restore snapshots from S3 with signed requests or from R2 without any authentication, based on their specific requirements.
Overcoming the Lack of Unsigned List Access in R2
One final challenge remained: the inability to listobjects in a Cloudflare R2 bucket without authentication. This lack of functionality posed a significant hurdle as it prevented users from easily discovering the available snapshots on R2.
To address this limitation, Mysten Labs developed a custom solution. They created a separate index file that contained a list of all the available snapshots hosted on R2. This index file was publicly accessible and didn’t require any authentication. By providing this index file, users could easily see the available snapshots and select the one they wanted to download.
Conclusion
The Sui network’s migration from AWS S3 to Cloudflare R2 for hosting state snapshots has significantly improved the user experience and reduced costs. With R2’s zero egress costs and the ability to access snapshots without authentication, node operators can seamlessly fetch the required data to run a Sui full node. Mysten Labs’ implementation of signed and unsigned object store operations, along with the custom index file, have overcome the challenges and ensured a smooth transition to R2 hosting. These efforts have contributed to the reliability and scalability of the Sui network, enabling it to maintain its high throughput and accommodate its rapidly growing snapshot sizes.
Disclaimer
The information provided in this article is for informational purposes only and should not be considered financial advice. The article does not offer sufficient information to make investment decisions, nor does it constitute an offer, recommendation, or solicitation to buy or sell any financial instrument. The content is opinion of the author and does not reflect any view or suggestion or any kind of advise from CryptoNewsBytes.com. The author declares he does not hold any of the above mentioned tokens or received any incentive from any company.
image source
