Optimizing build times with Bazel and Buildfarm

Introduction

Optimizing build times is crucial for maintaining a fast and efficient development pace in continuous integration pipelines.

On a client project, we created a mono-repository that now contains ~40 micro-services, mainly in Go such as APIs, Kafka consumers, Temporal workers, and also 4 front-end projects in React and TypeScript.

Bazel handles everything from building container images to pushing them to a registry and creating Kubernetes deployment manifests. We follow the GitOps philosophy 100%.

To build these projects and deploy them in our CI/CD, we use Bazel, an open-source build tool developed by Google.

It offers powerful features to accelerate this process.

When combined with Buildfarm, a distributed build and cache solution, the performance gains can be significant.

In this article, I'll share some configuration options that we added to our .bazelrc file to achieve faster builds.

How does Bazel work?

Without going into too much detail, Bazel is a build system and dependency management tool that supports multiple programming languages.

It is designed to be fast, reliable, and capable of handling various large-scale projects.

Here are some of its main features:

• Incremental Builds: Bazel only rebuilds the parts of the project that have changed, the rest is cached,
• Parallelization: It executes build actions in parallel to speed up the process,
• Reproducibility: Builds are reproducible, ensuring that the same code produces the same results every time.

What is Buildfarm?

Buildfarm is a distributed cache and build solution that allows you to run Bazel builds across multiple machines.

By distributing build tasks, Buildfarm can significantly reduce the time needed to compile large projects.

It integrates with several tools and particularly well with Bazel.

Optimizing the .bazelrc File

Let's get to the heart of the matter, here are some options I was able to identify among the available options to speed up build times.

In the context of our project, when a minimal change is made, our CI/CD job takes only 3 minutes to rebuild all 40 services.

The .bazelrc file is essential for configuring build options in Bazel. Here are some key options to include to improve build times:

Fast compilation mode

build --compilation_mode=fastbuild

Use fastbuild compilation mode to speed up the build process by sacrificing some optimizations.

Reuse sandbox directories

build --reuse_sandbox_directories

Reuse sandbox directories to avoid recreating execution environments for each build.

Disable external repository files check

build --noexperimental_check_external_repository_files

Disable experimental checking of external repository files to speed up the build process.

Disable action cache metadata storage

build --noexperimental_action_cache_store_output_metadata

Disable experimental storage of action cache metadata to reduce build time.

Parallel Skyframe analysis execution

build --experimental_merged_skyframe_analysis_execution

Enable parallel execution of Skyframe analyses to speed up dependency analysis.

Use distributed Merkle cache

build --experimental_remote_merkle_tree_cache

Use a distributed Merkle cache to efficiently store and retrieve build artifacts.

Use hardlinks for repository cache

build --experimental_repository_cache_hardlinks

Use hardlinks for repository cache to reduce disk space usage and speed up file access.

Protection against concurrent changes

build --experimental_guard_against_concurrent_changes

Protect against concurrent changes that can interfere with the build process.

Reduce downloads from remote cache

build --remote_download_outputs=minimal

Reduce the number of files downloaded from the remote cache to speed up builds.

For tests

test --build_tests_only

Build only what's necessary for tests, which greatly speeds up when you target the entire project.

Strict action environment and complete test output

test --incompatible_strict_action_env --test_output=all

Use a strict action environment and generate complete test output for better traceability and increased performance.

General Best Practices

These options improve builds, but it's important to keep these rules in mind to maintain fast builds:

• Dependency Reduction: Simplify your BUILD and WORKSPACE files to minimize dependencies,
• Powerful Machines: Run builds on machines with good CPU and RAM, as Bazel uses a lot of resources,
• Profiling: Use Bazel's profiling tools (--profile) to identify and resolve bottlenecks.

Conclusion

By properly configuring Bazel and Buildfarm, and optimizing your .bazelrc file, you can significantly reduce build times.

These optimizations allow you to maintain a fast and efficient development pace, even for large-scale projects.

Don't hesitate to experiment and adapt these configurations based on your project's specific needs.