Experience faster, more reliable builds with Cargo's new sparse registry protocol

With the Rust team's announcement of the 1.68.0 version of Rust, Cloudsmith is happy to announce our support for the new "sparse" registry protocol, which has been stabilized as part of version 1.68.0.

Cargo is the package manager for Rust, a programming language empowering everyone to build reliable and efficient software. Cloudsmith customers who migrate to the new beta "sparse" registry implementation will dramatically improve the speed and reliability of their builds.

What's changed with Cargo's sparse protocol

Cargo's "sparse" registry protocol defines a new mechanism for communication with remote registries such as crates.io and Cloudsmith. The sparse protocol switches to an HTTP-based transport from the previously supported (and still default) Git-based transport. The Git-based protocol works by cloning a repository that includes JSON metadata for all packages contained within the registry.

Advantages of the Git-based protocol

The Git-based method of serving the registry has several advantages, including:

• Crates.io could rely on GitHub to store metadata, offloading some infrastructure requirements
• Incremental updates (determining what's changed) are trivial with Git
• Git has a well-understood API that allows for simple integration with other tooling

Drawbacks of the Git-based protocol

However, because Git can be highly resource intensive and because Git expects repositories to maintain a clean history over time, this approach began to fail for providers as repositories grew in size. Cloudsmith customers saw this impact, as did the main crates.io registry users. To deal with the problems before they got out of control, The Cargo team implemented an HTTP-based protocol that Cargo could use to pull crate metadata from registries (either public or private).

Advantages of the sparse protocol

This new "sparse" protocol allows Cargo to fetch package metadata via simple HTTP GET requests that return JSON for only the packages that the client requests. This means:

• Only metadata for packages that the build uses need to be pulled
• Registries no longer need to maintain long-lived state (git repositories) to serve to clients
• HTTP caching and other techniques can be used to speed up performance dramatically
• Registries can handle much larger sets of packages due to no longer needing to deal with them all at once

Migrating to the new sparse registry implementation

We recommend Cloudsmith customers migrate to the new beta "sparse" registry implementation as soon as possible. Customers with 1000s of packages in their registries will see a dramatic improvement in the speed and reliability of their builds. The only migration requirement is that you're using Rust 1.68 or nightly.

Please have a look at the documentation in your repositories for setup instructions.

Deprecating support for the Git-based method

It is paramount to Cloudsmith that we maintain a performant and reliable platform for our customers. To ensure we do so for Cargo users, we intend to implement a limit on the number of packages a customer can have in a single repository, and to deprecate support for the Git-based protocol later this year. Currently, the sparse protocol is planned to become the default for crates.io in the 1.70.0 release coming in a few months.

Liked this article? Don\'t be selfish (:-), share with others:   Tweet