Table of contents of the article:
In an enterprise Linux migration, the problem is rarely simply upgrading the operating system version. The real problem is everything that's been built on top of that system over the years: legacy packages, no longer available dependencies, historical init scripts, application configurations, implicit assumptions, and components that, despite being dated, continue to play a critical role in production.
This is the case of a customer who needed to migrate 43 systems from CentOS 7 to RHEL 9, while maintaining a very binding technical requirement: to continue using Varnish Cache 3.0.7.
The reason was not an arbitrary preference for legacy software. The infrastructure used numerous Custom VCL 3.x, developed over the years, often thousands of lines long and deeply integrated with the application logic. These VCLs handled caches, bypasses, backends, headers, request normalization, exceptions, special behaviors, and specific business logic.
Rewriting them for Varnish 4, Varnish 6, or later didn't mean changing any guidelines. It meant starting a separate project for analysis, porting, functional testing, load testing, application validation, and a thorough comparison of the old and new behavior.
For this reason we have chosen a pragmatic approach: Migrate your operating system to RHEL 9 without changing Varnish 3 behaviorTo achieve this, however, simply installing the old EL7 RPMs wasn't enough. On EL9, those packages no longer worked properly. There was only one solution: recover the official source code and rebuild the package cleanly for RHEL 9 and derivatives.
The Varnish 3 rebuild GitHub repository for EL9
The result of this work has been published in a dedicated GitHub repository:
Repositories: https://github.com/MarcoMarcoaldi/VARNISH3-EL9
The repository contains the rebuild of the official RPM packages of Varnish Cache 3.0.7 for RHEL 9, AlmaLinux 9 and Rocky Linux 9, based on the original Source RPM varnish-3.0.7-1.el7.centos.src.rpm published by Varnish Software.
This is not a functional fork of Varnish, nor is it a modernized version of the software. It's a targeted rebuild, with minimal patches and changes to the RPM spec, designed for a specific use case: Maintain compatibility with legacy VCL 3.x configurations during a migration to EL9.
This distinction is fundamental. Varnish 3 is software End of life Since 2015, it hasn't received any upstream security fixes and shouldn't be used for new deployments. For new projects, it's OK to use Varnish 6.0 LTS or later. In this case, however, the requirement wasn't to design a new caching architecture, but to allow 43 existing systems to migrate from CentOS 7 without breaking thousands of lines of VCL already in production.
Why Varnish 4 broke compatibility with Varnish 3
The main technical issue is the compatibility breach introduced starting from Varnish 4The VCL language has not remained unchanged: it has been substantially restructured, with incompatible changes from the Varnish 3 syntax and logical model.
Some examples make the scope of the change immediately clear:
| Varnish 3 | Varnish 4 and later | Impact |
|---|---|---|
vcl_fetch | vcl_backend_response | Change the point in the flow where the response is handled by the backend |
req.request | req.method | Change the variable used to read the HTTP method |
error | synth | Change the logic for generating synthetic responses |
Actions return VCL 3 | Actions return rearranged | Change the behavior of the decision flow |
In a simple configuration, these changes can be addressed with relatively little manual porting. In an enterprise configuration, with thousands of VCL lines and years of layered logic, the situation is different.
A production VCL isn't just a configuration file. It's often an integral part of application behavior. It decides when a request should be cached, when it should be passed to the backend, when a cookie should cause a bypass, when a header should be removed, when a URL should be normalized, when a response should have differentiated TTLs, or when an error should be transformed into a synthetic response.
Mistranslating even one of these logics can lead to serious problems: caching of private content, missed bypasses, backend overload, API errors, lost application headers, unexpected changes in TTLs, or hard-to-diagnose differences in client-side behavior.
For this reason, in this project, porting to VCL 4 or later was deliberately excluded from the operating system migration. It would have been technically possible, but it wasn't reasonable within the same project window.
Why old RPM EL7s couldn't be reused on EL9s
The natural first step when maintaining a legacy version is to check whether existing packages can be installed on the new operating system. In our case, the historical package was varnish-3.0.7-1.el7.centos.
On EL8, some dependencies were still available through compatibility packages. On EL9, however, the old binary broke for structural reasons. Runtime dependencies required by the EL7 package were no longer present or could no longer be resolved in the same way.
| EL7 Runtime Dependency | Situation on EL8 | Situation on EL9 |
|---|---|---|
libncurses.so.5 / libtinfo.so.5 | Available via ncurses-compat-libs | Removed |
libnsl.so.1 | Available via libnsl | Removed, available only libnsl.so.2 |
/sbin/chkconfig e /sbin/service | Dependencies still resolvable | Unresolvable dependencies due to UsrMove |
This ruled out a superficial adaptation. Creating symlinks, forcing installation, or ignoring dependencies wasn't enough. The problem was deeper: the binary had been compiled for a different runtime ecosystem.
The conclusion was clear: needed to recompile Varnish 3.0.7 from source, generating RPMs consistent with RHEL 9, AlmaLinux 9 and Rocky Linux 9.
Retrieving the official Source RPM
The work started from the original Source RPM declared in the metadata of historical packages: varnish-3.0.7-1.el7.centos.src.rpm.
The old historical repository repo.varnish-cache.org it was no longer available, but the Source RPM was still available in the repository varnishcache/varnish30 on packagecloud. Once downloaded and extracted, the package contained the two essential elements needed to proceed:
varnish-3.0.7.tar.gz, that is, the upstream source tarball;varnish.spec, that is, the original RPM spec file for EL7.
From there, the adaptation work began. The goal wasn't to alter Varnish's behavior, but to apply the minimum set of changes necessary to obtain a package that would compile and, most importantly, actually run on EL9.
Release 3.0.7-2: First RPM spec adaptation to EL9
The first release of the rebuild, 3.0.7 - 2, involved the initial adaptation of the spec and source to the EL9 build environment.
The first problem was in the source code included in the tarball. The historical version of jemalloc incorporated by Varnish 3 included the header <sys/sysctl.h>, removed in modern versions of glibc. On EL9, which is based on modern glibc, that reference prevented compilation.
A minimal patch was then applied to remove the no longer available include. This isn't a functional change, but a necessary fix to allow the 2015 source code to be compiled in a 2020+ environment.
The spec file then required several adjustments:
pythonit has been replaced withpython3inBuildRequires;groffit has been replaced withgroff-base;libedit-develit was managed via CRB repository;- Scriptlet dependencies have been expressed by package name, no longer by path
/sbin/*; - LTO has been disabled;
- the test suite was made opt-in via
--with check; - rpmdevtools rpath check has been disabled, because RPATH to private libraries in
/usr/lib64/varnishit is necessary.
An important point concerns scriptlets. Old packages relied on dependency files like /sbin/chkconfig e /sbin/serviceOn EL9, with UsrMove, this model is no longer reliable. The correct solution was to declare dependencies on packages like chkconfig, initscripts e initscripts-service.
SysV scripts have been retained for compatibility with EL7. This has preserved existing operational behavior, including the use of /etc/sysconfig/varnish.
SysV init on EL9: legacy choice, but useful for compatibility
Varnish 3 was born at a time when systemd integration wasn't the dominant model. To maintain maximum compatibility with the previous environment, the SysV scripts were left largely unchanged.
On EL9 the service can still be managed via systemd thanks to systemd-sysv-generator, which dynamically generates systemd units from legacy SysV scripts.
systemctl enable --now varnish systemctl status varnish
This allows you to maintain familiar service management without immediately rewriting the entire init layer. This solution is consistent with the project's goal: avoiding unnecessary changes while migrating the operating system.
However, there remains a known limitation. This choice is suitable for EL9, but it shouldn't be considered an indefinite future strategy. In future scenarios, especially considering RHEL 10 and beyond, native systemd units will be necessary.
Release 3.0.7-3: SIGSEGV Crash Caused by jemalloc
After the first rebuild, the package was built and installed. The next problem only emerged at runtime: varnishd crashed immediately on startup with a SIGSEGV.
This was one of the most interesting parts of the troubleshooting process, because the build was formally correct. The problem didn't appear during compilation, but only when the daemon was running on EL9.
Debugging with gdb, supported by packages -debuginfo produced during the build, showed the real chain:
varnishd
-> getgrnam("varnish")
-> nss-systemd
-> malloc_usable_size(NULL)
-> jemalloc legacy
-> SIGSEGV
In the startup phase, varnishd solves the group varnish means getgrnam("varnish"). On EL9 comes into play nss-systemd, which can be called malloc_usable_size(NULL). This behavior is legal for glibc.
The problem is that Varnish 3 incorporated a very old version of jemalloc, dating back to 2008, which intercepted process-level allocation functions. Its implementation of malloc_usable_size() did not handle the pointer correctly NULLThe result was an immediate segfault, even before varnishd could print a useful message.
The fix applied in the release 3.0.7 - 3 was compiling Varnish with:
--without-jemalloc
In this way varnishd Uses glibc's modern malloc, bypassing the old embedded jemalloc. This fixed the startup crash on EL9.
Release 3.0.7-4: VCL not compiling on clean EL9 hosts
Once the crash was fixed, a second, even more subtle runtime issue emerged. The service would start correctly on the build machine, but would fail on a minimal, clean EL9 host when compiling the VCL.
The error was as follows:
gcc: fatal error: cannot read spec file '/usr/lib/rpm/redhat/redhat-hardened-cc1'
To understand the problem, we need to remember an architectural detail of Varnish: Varnish compiles VCLs at runtime into C code. This means that gcc It's not just a build dependency, but also a runtime dependency. Each VCL is transformed into C code, compiled, and loaded by the daemon.
During the RPM build phase on EL9, the command cc registered by Varnish incorporated flags derived from the RPM environment, including references to:
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
That file is present on the build machine, where it is installed redhat-rpm-configBut it's not normally present on a minimal production server. So the package seemed to work on the build machine, but failed when deployed to a clean system.
The fix applied in the release 3.0.7 - 4 it was forcing a command VCC_CC clean and portable when configuring:
export VCC_CC='exec gcc -std=gnu99 -O2 -g -fpic -shared -Wl,-x -o %o %s'
In this way varnishd It registers a runtime compilation command that is independent of the files present only on the build machine. The VCL can therefore also be compiled on minimal EL9 hosts, as long as it is installed. gcc, required for design.
Installing RPM packages
Once the RPMs are produced or downloaded from the Releases page of the repository, the installation is done via dnf, leaving the package manager to automatically resolve dependencies:
cd /path/to/downloaded/rpms/ dnf install ./varnish*.rpm
After installation, the service can be enabled and started normally:
systemctl enable --now varnish curl -I http://localhost:6081/
The configuration remains in the historical paths used on EL7:
/etc/varnish/default.vcl /etc/sysconfig/varnish
This is a very important practical aspect in a 43-system migration. Maintaining the same paths reduces operational differences, simplifies rollout, and allows for the reuse of existing procedures, templates, and automations.
Build from source on AlmaLinux 9 or Rocky Linux 9
The repository also allows you to rebuild packages locally. On an AlmaLinux 9 or Rocky Linux 9 machine, the flow is the classic RPM flow:
dnf install -y rpm-build rpmdevtools gcc make ncurses-devel libxslt groff-base pcre-devel pkgconf-pkg-config python3 dnf config-manager --set-enabled crb dnf install -y libedit-devel rpmdev-setuptree cp SOURCES/* ~/rpmbuild/SOURCES/ cp SPECS/varnish.spec ~/rpmbuild/SPECS/ rpmbuild -ba ~/rpmbuild/SPECS/varnish.spec
RPMs are generated in:
~/rpmbuild/RPMS/x86_64/
The rebuilt Source RPM is instead produced in:
~/rpmbuild/SRPMS/
This allows for a repeatable, versioned, and verifiable process, rather than relying on manual installations or undocumented workarounds.
Testing on EL9: it's not enough that the package compiles
One of the key lessons from this intervention is that, in a legacy rebuild, filling out the form is just the first stepAn RPM can be formally correct, install without errors, and still not be usable in production.
In our case the most important problems emerged only after:
- the reference to
<sys/sysctl.h>it emerged during the compilation phase; - the jemalloc bug only surfaced when starting
varnishdon EL9; - the problem of
VCC_CCIt only surfaced on a clean host other than the build machine.
For this reason, the testing was performed on minimal EL9 systems, verifying not only the start of the service but also the loading of the VCLs, the HTTP headers, the HIT caches, the behavior under load and the visibility through tools such as varnishstat.
A basic check after startup can be performed with:
curl -I http://localhost:6081/
Expected headers include elements such as:
Via: 1.1 varnish X-Varnish: ...
In a real environment, of course, this isn't enough. You need to validate application VCLs, backends, bypass rules, cookie management, TTLs, purges, error paths, and all the other specific cases that have been encoded within VCL 3.x logic over the years.
Known limitations of the solution
This solution doesn't transform Varnish 3 into a modern software. It is and remains a legacy component, retained here only for compatibility reasons.
The main limitations are clear:
- Varnish 3 is End of Life and does not receive upstream security updates;
- should not be used for new deployments;
- does not support native TLS;
- does not support HTTP/2;
- still uses SysV scripts, managed on EL9 via
systemd-sysv-generator; - requires
gccat runtime, because the VCL is compiled on the fly to C code.
In production, TLS should be terminated in front of Varnish, for example via nginx, HAProxy, or hitch. For future environments, it will also be worth considering migrating to modern versions of Varnish and completely rewriting the VCLs.
Because this was the right choice for the customer
Theoretically, the simplest answer would have been: "Let's also update Varnish." Operationally, however, it would have been the riskiest choice.
The client didn't just need a new operating system. They needed to maintain application continuity across 43 systems without simultaneously introducing four critical changes:
- switching from CentOS 7 to RHEL 9;
- change of runtime and system libraries;
- Varnish major version change;
- Complete rewrite of VCLs from version 3.x to version 4 or later.
Separating these issues was the most prudent choice. The operating system was updated first, maintaining the existing cache behavior. Only at a later stage, with a dedicated project, could we evaluate porting the VCLs to Varnish 6 LTS or later versions.
This is an approach we at Managed Server Srl often adopt in complex contexts: avoiding "big bang" migrations when the risks are too high and instead building progressive, verifiable, and reversible paths.
Conclusions
The rebuild of Varnish Cache 3.0.7 for RHEL 9, AlmaLinux 9 and Rocky Linux 9 It was born from a concrete need: to migrate 43 CentOS 7 systems to RHEL 9 without immediately rewriting complex and already validated VCL 3.x in production.
The work was not limited to relaunching a rpmbuild. It was necessary to recover the official Source RPM, adapt the spec to EL9, correct the problem of <sys/sysctl.h>, fix the jemalloc-induced crash, make the VCL runtime compilation command portable, and test the behavior on clean hosts.
The easy part was compiling. The hard part was getting a package that installed, ran, compiled VCLs, and actually ran on EL9, not just on the build machine.
The public repository is available here:
https://github.com/MarcoMarcoaldi/VARNISH3-EL9
It's a compatibility solution, not a recommendation for new projects. But in a real-world context, with legacy systems, critical VCLs, and an infrastructure migration to complete without disrupting production, it was exactly the kind of intervention needed.
Because in real Linux systems, sometimes the most important job isn't installing the latest available version. It's building the technical bridge that allows you to get there without breaking what's currently holding the service up.
