![]() ![]() The effect was immediately apparent, reducing the simulation build time by 2.5 minutes and the non-bottleneck debug build time by 4 minutes. I had been thinking about whether I could echo the key out in CI before, but it seems there’s no need to worry about that. Buildkite can obtain permissions through IAM roles, so I just need to a policy for the role to access an S3 bucket, without the need to configure things like secret keys. (Well, behind the scenes, you actually need to study Buildkite and AWS configurations - which are also very simple. Just add two environment variables to start it up:ĮNV RUSTC_WRAPPER=sccache ENV SCCACHE_BUCKET=ci-sccache-bucket Why should you give Sccache a try? With xuanwo’s strong recommendation, I was very tempted to try sccache, which was also a major trigger for my optimization efforts this time. If you think about it, it’s pretty obvious how a good caching strategy for CI should work. We can see that the compilation times for some dependencies such as zstd-sys and protobuf-src are very long, so we should try to optimize them. cargo build -timingsĬargo comes with built-in support for profiling build times (it was stabilized last year), which can be enabled by running cargo build -timings. The DAG graph shows that recovery test depends only on simulation build, so we can forget about the check step for now, and conclude the biggest bottleneck is in the path of simulation build -> recovery test. Two large steps finish before it: build (deterministic simulation) and check. We use Buildkite for our CI, and the normal view of a page (such as Build #17099) looks like this:īuildkite has two very useful hidden pages, located at /waterfall and /dag, respectively, which show:įrom the waferfall graph, we can see recovery test finishes last. Luckily, we do have some nice ones for optimizing CI time. When trying to optimize anything, it would be good to have some profiling data and charts to find out the bottlenecks. Valuable data and charts to find the bottlenecksīuild times are a fairly easy optimization problem: it’s trivial to get direct feedback (just time the build), there are a bunch of tools for profiling, and you don’t even need to come up with a representative benchmark. Results after optimization: main workflow is now 27 minutes at the fastest, and PR workflow is now 16 minutes at the fastest, with most taking around 17-19 minutes. Each optimization point comes with a corresponding PR, and you can combine the commit history to see the effect of each optimization point by comparing the CI pages before and after its PR. If not otherwise indicated, all quotes in this article come from there.Īlthough there are quite some articles talking about how to optimize Rust compilation speed (e.g., Tips for Faster Rust Compile Times), I still want to write another one to share my step-by-step process. Most of the methods I used are discussed there, and he explains them clearly. I highly recommend matklad’s blog (He is the original author of IntelliJ Rust and rust-analyzer): I feel like I can describe them as low-hanging fruits, silver bullets, or even free lunch □. What shocked me was that there were some very simple methods that, with just a little effort, produced astonishing results. So a few days ago, I decided to spend some time researching whether I could optimize the compilation speed a bit more. Although it is still not intolerably slow, it is already noticeably slower than before. The main workflow now takes about 40 minutes, while the PR workflow takes about 25 minutes 30 seconds. ![]() However, over time, more and more things have been added to our CI, making it increasingly bloated. A full debug compilation only takes 2-3 minutes. After using an M1 MacBook Pro, compiling is not a problem at all. Step 3: Disable incremental compilationĪlthough there are often complaints saying Rust’compilation speed is notorious slow, our project RisingWave is not very slow to compile, especially since previously contributors like ( skyzh, BugenZhao) have put in a lot of effort.Valuable data and charts to find the bottlenecks.Stupidly effective ways to optimize Rust compile time ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |