Go modules are a new feature in Go for versioning packages and managing dependencies. It has been almost 2 years in the making, and it’s finally production-ready in the Go 1.14 release early this year. Go recommends using single-module repositories by default, and warns that multi-module repositories require great care.
At Grab, we have a large monorepo and changing from our existing monorepo structure has been an interesting and humbling adventure. We faced serious obstacles to fully adopting Go modules. This series of articles describes Grab’s experience working with Go modules in a multi-module monorepo, the challenges we faced along the way, and the solutions we came up with.
To fully appreciate Grab’s journey in using Go Modules, it’s important to learn about the beginning of our vendoring process.
With Go 1.5 came the concept of the vendor
folder, a new package discovery method, providing native support for vendoring in Go for the first time.
With the vendor
folder, projects influenced the lookup path simply by copying packages into a vendor
folder nested at the project root. Go uses these packages before traversing the GOPATH
root, which allows a monorepo structure to vendor packages within the same repo as if they were 3rd-party libraries. This enabled go build
to work consistently without any need for extra scripts or env var modifications.
Initial Obstacles
There was no official command for managing the vendor
folder, and even copying the files in the vendor
folder manually was common.
At Grab, different teams took different approaches. This meant that we had multiple version manifests and lock files for our monorepo’s vendor folder. It worked fine as long as there were no conflicts. At this time very few 3rd-party libraries were using proper tagging and semantic versioning, so it was worse because the lock files were largely a jumble of commit hashes and timestamps.
As a result of the multiple versions and lock files, the vendor directory was not reproducible, and we couldn’t be sure what versions we had in there.
Temporary Relief
We eventually settled on using Glide, and standardised our vendoring process. Glide gave us a reproducible, verifiable vendor
folder for our dependencies, which worked up until we switched to Go modules.
Vendoring Using Go Modules
I first heard about Go modules from Russ Cox’s talk at GopherCon Singapore in 2018, and soon after started working on adopting modules at Grab, which was to manage our existing vendor
folder.
This allowed us to align with the official Go toolchain and familiarise ourselves with Go modules while the feature matured.
Switching to Go Modules
Go modules introduced a go mod vendor
command for exporting all dependencies from go.mod
into vendor
. We didn’t plan to enable Go modules for builds at this point, so our builds continued to run exactly as before, indifferent to the fact that the vendor directory was created using go mod
.
The initial task to switch to go mod vendor
was relatively straightforward as listed here:
- Generated a
go.mod
file from ourglide.yaml
dependencies. This was scripted so it could be kept up to date without manual effort. - Replaced the vendor directory.
- Committed the changes.
- Used
go mod
instead of glide to manage the vendor folder.
The change was extremely large (due to differences in how glide and go mod
handled the pruning of unused code), but equivalent in terms of Go code. However, there were some additional changes needed besides porting the version file.
Addressing Incompatible Dependencies
Some of our dependencies were not yet compatible with Go modules, so we had to use Go module’s replace directive to substitute them with a working version.
A more complex issue was that parts of our codebase relied on nested vendor directories, and had dependencies that were incompatible with the top level. The go mod vendor
command attempts to include all code nested under the root path, whether or not they have used a sub-vendor directory, so this led to conflicts.
Problematic Paths
Rather than resolving all the incompatibilities, which would’ve been a major undertaking in the monorepo, we decided to exclude these paths from Go modules instead. This was accomplished by placing an empty go.mod file in the problematic paths.
Nested Modules
The empty go.mod
file worked. This brought us to an important rule of Go modules, which is central to understanding many of the issues we encountered:
A module cannot contain other modules
This means that although the modules are within the same repository, Go modules treat them as though they are completely independent. When running go mod
commands in the root of the monorepo, Go doesn’t even ‘see’ the other modules nested within.
Tackling Maintenance Issues
After completing the initial migration of our vendor directory to go mod vendor however, it opened up a different set of problems related to maintenance.
With Glide, we could guarantee that the Glide files and vendor directory would not change unless we deliberately changed them. This was not the case after switching to Go modules; we found that the go.mod
file frequently required unexpected changes to keep our vendor directory reproducible.
There are two frequent cases that cause the go.mod
file to need updates: dependency inheritance and implicit updates.
Dependency Inheritance
Dependency inheritance is a consequence of Go modules version selection. If one of the monorepo’s dependencies uses Go modules, then the monorepo inherits those version requirements as well.
When starting a new module, the default is to use the latest version of dependencies. This was an issue for us as some of our monorepo dependencies had not been updated for some time. As engineers wanted to import their module from the monorepo, it caused go mod vendor
to pull in a huge amount of updates.
To solve this issue, we wrote a quick script to copy the dependency versions from one module to another.
One key learning here is to have other modules use the monorepo’s versions, and if any updates are needed then the monorepo should be updated first.
Implicit Updates
Implicit updates are a more subtle problem. The typical Go modules workflow is to use standard Go commands: go build
, go test
, and so on, and they will automatically update the go.mod
file as needed. However, this was sometimes surprising, and it wasn’t always clear why the go.mod
file was being updated. Some of the reasons we found were:
- A new import was added by mistake, causing the dependency to be added to the
go.mod
file - There is alocal replacefor some module B,