It all started, as most good stories do, with:
Entering directory 'C:\it\sbx\wave\x86_64-windows64\infer-edge\src\infer' ** Cannot resolve symbols for C:\it\sbx\wave\x86_64-windows64\tp-opam\install\r\s\lib/ocaml\libasmrun.a(startup_aux.n.o): swscanf ** Cannot resolve symbols for descriptor object: _vscwprintf File "caml_startup", line 1: Error: Error during linking (exit code 2) -> required by _build/default/.ppx/a5fc69afc6265736fea9df46821bec87/ppx.exe -> required by _build/default/src/quandary/ClangTaintAnalysis.pp.ml -> required by _build/default/src/quandary/.Quandary.objs/native/quandary__ClangTaintAnalysis.cmx -> required by _build/default/src/quandary/Quandary.cmxa -> required by _build/default/src/infer.exe Leaving directory 'C:\it\sbx\wave\x86_64-windows64\infer-edge\src\infer' make[1]: *** [Makefile:124: /it/sbx/wave/x86_64-windows64/infer-edge/src/infer/bin/infer.exe] Error 1 make[1]: Leaving directory '/it/sbx/wave/x86_64-windows64/infer-edge/src/infer/src' make: *** [Makefile:377: src_build] Error 2
We hadn't touched anything, not a single commit the day before, and yet there we were, with our builds not building, our tests not testing, our prod not proding. It turns out that the team in charge of GCC had upgraded the windows runtime GCC is compiled with, going from MinGW to UCRT. It also turns out that our Ocaml compiler was built with a pretty old GCC, and wasn't even rebuilt every day. This resulted in our poor Infer being built with a UCRT GCC, against a MinGW Ocaml runtime, hence the complaints about the missing symbols. The solution was just to recompile Ocaml with the newer GCC, which would reconcile the runtime needs of Infer and Ocaml's runtime.
Figuring this out was the easy part though. When we started building our static analyzer on Infer, Ocaml was not relocatable on windows, and the solution to that was to heavily patch Ocamlc. These patches, being heavy, made Ocaml hard to upgrade - we were stuck on version 4.12.Something of the compiler while the rest of the world moved on to Glorious Ocaml 5 with Parrallelism and Effects.
Ocaml 4.12.0 came out in early 2021. In 2021, GCC's most up to date release was GCC 10. Between 2021 and 2026, GCC had had plenty of time to improve, and in particular, to find plenty of issues with C code. Guess what's written in C? That's right, Ocamlc. This is why, when we tried to recompile our Ocaml 4.12.0 with a GCC based on GCC 15, we were met with:
### output ###
# [...]
# caml/mlvalues.h:228:18: note: previous declaration of 'caml_get_public_method' with type 'value(value, value)' {aka 'long long int(long long int, long long int)'}
# 228 | CAMLextern value caml_get_public_method (value obj, value tag);
# | ^~~~~~~~~~~~~~~~~~~~~~
# prims.c:338:14: error: conflicting types for 'caml_set_oo_id'; have 'value(void)' {aka 'long long int(void)'}
# 338 | extern value caml_set_oo_id();
# | ^~~~~~~~~~~~~~
So our crummy oldcaml 4.12.Something couldn't be compiled with a recent GCC. The only way to stop GCC from detecting these issues was to pass -std=c17 to GCC, but that caused more issues than it solved because Oldcaml 4.12. wasn't written for c17.
It sounded like we had no choice, we'd have to upgrade our Ocaml to a more recent version, that could be compiled with a more recent GCC, and I tried this hard, very hard, and failed miserably. The reason for that is that we're not only carrying patches for relocations - we also have some to make Ocaml's runtime compatible with GNAT's (some of our libraries being written in Ada), and to improve the performance of the Ocaml typer, which chokes miserably on the crazy things we do with types in our Infer front-end.
In the end, I gave up and settled for a much less ambitious approach: backport Ocaml pull request #12577 and #9919. These patches were not too compilicated to apply, but then we started experiencing failures in the packages we used, e.g.
#=== ERROR while compiling ANSITerminal.0.8.1 =================================#
# context 2.0.4 | win32/x86_64 | ocaml-variants.4.12.1+flambda+reloc | file://C:/Users/itmgr/thirdparties/x86_64-windows64/opam-4.12.1-2/src/opam-repo
# path ~/thirdparties/x86_64-windows64/opam-4.12.1-2/pkg/r/s/.opam-switch/build/ANSITerminal.0.8.1
# command C:\Users\itmgr\thirdparties\x86_64-windows64\opam-4.12.1-2\pkg\r\s\bin\dune.exe build -p ANSITerminal -j 7
# exit-code 1
# env-file ~/thirdparties/x86_64-windows64/opam-4.12.1-2/pkg/r/log/ANSITerminal-5488-876df3.env
# output-file ~/thirdparties/x86_64-windows64/opam-4.12.1-2/pkg/r/log/ANSITerminal-5488-876df3.out
### output ###
# ANSITerminal_stubs.c:240:44: error: passing argument 5 of 'FillConsoleOutputCharacterA' from incompatible pointer type [-Wincompatible-pointer-types]
# [...]
# | int *
# In file included from C:/Users/itmgr/thirdparties/x86_64-windows64/stable-gcc_20260203/install/x86_64-w64-mingw32/include/windows.h:74,
# from ANSITerminal_stubs.c:30:
# C:/Users/itmgr/thirdparties/x86_64-windows64/stable-gcc_20260203/install/x86_64-w64-mingw32/include/wincon.h:231:136: note: expected 'LPDWORD' {aka 'long unsigned int *'} but argument is o
f type 'int *'
# 231 | WINBASEAPI WINBOOL WINAPI FillConsoleOutputCharacterA(HANDLE hConsoleOutput,CHAR cCharacter,DWORD nLength,COORD dwWriteCoord,LPDWORD lpNumberOfCharsWritten);
# | ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
# In file included from C:/Users/itmgr/thirdparties/x86_64-windows64/stable-gcc_20260203/install/x86_64-w64-mingw32/include/windef.h:9,
# from C:/Users/itmgr/thirdparties/x86_64-windows64/stable-gcc_20260203/install/x86_64-w64-mingw32/include/windows.h:69:
# C:/Users/itmgr/thirdparties/x86_64-windows64/stable-gcc_20260203/install/x86_64-w64-mingw32/include/minwindef.h:152:18: note: 'LPDWORD' declared here
# 152 | typedef DWORD *LPDWORD;
# | ^~~~~~~
Patching our packages was easier than patching the compiler though, and got me pretty close to the end of the ordeal. In fact, I thought I was done when everything worked On My Machine™, except it turns out that the nightly build machines rebuilding our Ocaml compiler were running CentOS7 and failed with
opam init --bare -n adacore --disable-sandboxing --bypass-checks -vv --keep-build-dir /it/sbx/thirdparties/x86_64-linux/opam-4.14.3-1/src/opam-repo opam: /lib64/libc.so.6: version `GLIBC_2.26' not found (required by /it/sbx/thirdparties/x86_64-linux/stable-gcc_20260203/install/lib64/libstdc++.so.6) opam: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /it/sbx/thirdparties/x86_64-linux/stable-gcc_20260203/install/lib64/libstdc++.so.6) opam: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /it/sbx/thirdparties/x86_64-linux/stable-gcc_20260203/install/lib64/libstdc++.so.6)
When asked to run a pretty recent GCC. The solution was simple though - upgrade the machine to centos8 and forget about it.
Moral of the story: