Web Server Logs in accordance with the law for judicial authorities and the GDPR

Introduction

Over the past few decades, the Internet has become an essential part of the daily lives of billions of people. Surfing the web is a common activity for both personal and professional purposes, involving individuals of all ages and backgrounds. However, this growing dependence on the Internet has also led to an increase in illegal behavior, ranging from simple insults on social media to more serious crimes such as computer fraud, online scams, and child pornography crimes.

Behind every digital action, traces remain, often recorded in web server logs , which are files that collect information on users' browsing activities. This data can reveal who visited a particular page, uploaded a file, or left a comment.

For this reason, webmasters, hosting providers, and system administrators sometimes find themselves responding to requests from judicial authorities, who require access to web server logs to identify those responsible for unlawful acts. However, this information must be managed in compliance with the law, particularly Italian law and the General Data Protection Regulation (GDPR).

This article looks at how to keep web server logs in a legally compliant manner, both in compliance with the requirements of the authorities and those of personal data protection, offering practical guidance to IT professionals.

What is GDPR?

The General Data Protection Regulation (GDPR) , introduced by the European Union and effective as of May 25, 2018, is one of the most important regulatory frameworks for personal data protection. Its primary objective is to ensure that data collected and processed by organizations is managed transparently, securely, and respectful of individuals' rights.

The GDPR applies to any entity that processes personal data of EU citizens, whether operating within the Union or outside. Personal data means any information that can identify a natural person, directly or indirectly, such as a name, IP address, email or location information.

What is a web server log?

A web server is software designed to receive and respond to HTTP and HTTPS requests from clients, typically browsers or other applications. In other words, it's the component that allows websites and web applications to be displayed on the internet. Among the most widely used and well-known web servers are:

• Apache HTTP Server: One of the longest-running and most popular web servers, very versatile and supported by a large open source community.
• Nginx: Famous for its efficiency in handling simultaneous connections, often used as both a web server and a reverse proxy.
• LiteSpeed: Valued for its high performance, especially in shared hosting environments and with CMS like WordPress.

Every web server generates files called logs , which record the activities performed on the server itself. Web server logs contain a detailed history of what happens when the server handles user requests. Typically recorded information includes:

• User's IP address: Identifies the device from which the request comes.
• Date and time of the request: Indicates when the user interacted with the server.
• URLs requested: Specifies which resources were requested (pages, files, images, etc.).
• HTTP method used: Reports the type of operation performed, such as GET, POST, or DELETE.
• Server response: Reports the status code returned by the server, such as 200 (success), 404 (page not found), or 500 (server error).
• Useragent: Contains information about the browser, operating system and device used by the user.

This data is essential for the operation, security, and optimization of websites, allowing us to diagnose problems, monitor traffic, and detect suspicious activity. However, much of this information, such as IP addresses, may be considered personal data under the GDPR, requiring careful and compliant management.

Italian law and log retention

In Italy, the management and storage of web server logs are governed by a complex set of regulations that are intertwined with the General Data Protection Regulation (GDPR) . These regulations often add specific details and obligations compared to the European framework, especially regarding security requirements, judicial investigations, and the prevention of crime.

The Electronic Communications Code

A key reference is the Electronic Communications Code (Legislative Decree 259/2003), which establishes specific obligations for providers of telecommunications and internet access services. Specifically, the Code requires traffic data, collected for the purposes of detecting and suppressing crime, to be retained for a minimum period of 12 months . This data includes:

• Source and destination of the communication (IP addresses and port numbers).
• Date, time and duration of the connection.
• Protocol used (for example, HTTP, HTTPS, or FTP).

The requirement to retain for at least one year is designed to ensure that information needed for an investigation can be retrieved by the competent authorities.

Obligations for service providers

Internet service providers, hosting and telecommunications are subject to more stringent rules than other operators. In Italy, these entities must retain logs not only for technical or administrative purposes, but also to respond to any requests from judicial authorities. Failure to comply with these obligations may result in administrative or criminal sanctions, depending on the severity of the violation.

Regarding hosting providers , while not expressly equated with telecommunications providers, there are indirect obligations to cooperate with authorities. For example, web server logs may be requested to identify those who have uploaded illegal content to a hosted platform or to track down the perpetrator of fraud.

Prolonged storage in specific cases

In special situations, such as investigations into serious crimes (e.g. terrorism or child pornography), the retention of logs may be extended beyond the standard terms. In such cases, upon request of judicial authorities, data may be retained for a longer period, provided that the request is justified and occurs in compliance with the legislation on the protection of personal data.

Differences between GDPR and Italian law

Unlike the GDPR, which leaves room for more general interpretations regarding the duration of retention, the Italian regulation is often more detailed and prescriptive. This creates a complex framework, where IT professionals must balance:

1. Minimum duration: In Italy, the retention of traffic data for at least 12 months is mandatory for public safety purposes and for the detection of illegal activities.
2. Maximum duration: The GDPR imposes the principle of “storage limitation”, whereby data must not be retained longer than is necessary for the stated purposes.
3. Compatibility between regulations: Italian law can expand the requirements of the GDPR, but it cannot contradict them. For example, it cannot require the retention of logs for an indefinite period without a solid legal basis.

Other regulatory references

Other legislative instruments that influence log retention in Italy include:

• Personal data protection code (Legislative Decree 196/2003): Integrated with the GDPR, it specifies the criteria for the conservation of personal data at national level.
• Law against terrorism (Legislative Decree 155/2005): Introduces specific cooperation obligations for the monitoring and retention of data related to potential terrorist threats.
• AGCOM Regulation: For digital content providers, it imposes indirect responsibilities in the management of data relating to users and online activities.

Mistakes to avoid in managing logs and IPs when using CDN and Reverse Proxy

When using a CDN (Content Delivery Network) or reverse proxy technology like Varnish , NGINX , Cloudflare , or Caddy , you introduce an intermediate layer between the user and the originating server that, if not configured correctly, can alter the data collected in the logs. A common mistake is to record the IP address of the reverse proxy in the log files instead of the actual IP of the user making the request. This happens because, in the default configuration, the originating server sees the source address as the proxy or CDN node that forwarded the request, not that of the end client.

This issue can have significant implications. Recording the proxy's IP address compromises the validity of logs for several purposes. From a security perspective , it makes it difficult to identify the actual source of any attacks, such as intrusion attempts or suspicious activity. From a legal compliance perspective , this error can impair the ability to satisfy requests from law enforcement authorities, who often require logs to identify the user responsible for illicit online activity. Furthermore, in the context of traffic analysis , failure to record users' real IP addresses can lead to inaccurate data, reducing the reliability of key metrics for website optimization and management.

This challenge becomes even more relevant in complex scenarios, where traffic is distributed across multiple CDN nodes or through multiple proxy levels. To avoid these errors, it is essential to configure the system to preserve the user's original IP, using specific headers provided by the adopted technology, such as X-Forwarded-For o CF-Connecting-IP. Without such a configuration, the logs not only become unusable for investigative purposes, but can also expose the service provider to potential regulatory violations, especially in regulated environments such as GDPR or of Italian legislation.

Why does this error occur?

Reverse proxy technologies and CDNs act as intermediaries between the user and the originating server. When a request passes through a proxy like Varnish or a CDN like Cloudflare, the originating web server only sees the IP address of the proxy and not the actual client. This is because the connection between the proxy and the originating server does not preserve the user's IP unless it is configured correctly.

How to avoid logging proxy IP

With Varnish

Varnish, being a reverse proxy caching technology, doesn't handle HTTPS traffic directly, but relies on a server like NGINX or Caddy for SSL termination. To ensure the user's real IP address is logged, you need to configure logging during HTTPS termination. For example:

• Configuration Nginx o Caddy to add the header X-Forwarded-For on the incoming request. This header includes the client's original IP address.
• On Varnish, configure logs to read and log the IP from the header X-Forwarded-For, rather than the IP address of the direct connection.

Example configuration in Varnish VCL:

sub vcl_recv { set req.http.X-Real-IP = client.ip; }

With Cloudflare and other CDNs

Cloudflare and similar services (Akamai, Imperva's Incapsula, Sucuri, Fastly) provide specific guides to restore the user's real IP. You need to:

1. Configure the native web server (e.g. NGINX or Apache) to read the custom headers added by the CDN, such as CF-Connecting-IP (Cloudflare) or True-Client-IP (Akamai).
2. Enable the Real IP Module in NGINX or the equivalent module in Apache, specifying the CDN IP ranges as trusted proxies.

Example configuration for NGINX with Cloudflare:

set_real_ip_from 173.245.48.0/20; # Cloudflare IP Ranges set_real_ip_from 103.21.244.0/22; real_ip_header CF-Connecting-IP;

With other CDNs

This approach also applies to other CDNs that operate as reverse proxies. For example:

• fast: Use the header Fastly-Client-IP.
• Sucuri: Use the header X-Forwarded-For.
• Encapsulate (Imperva): Use the header True-Client-IP.

Implications and risks

Failure to properly configure real IP address logging can have significant consequences:

• GDPR Compliance and Italian Law: If the logs only contain proxy IPs, they may not be considered valid for reconstructing a user's activities in case of legal requests.
• SafetyIntrusion detection or anomaly analysis tools can be ineffective if they don't work with users' real IPs.
• Debugging and traffic analysisVisibility into real user behavior is compromised.

To avoid log management errors when using reverse proxies and CDNs, it's essential to configure servers to log users' real IP addresses, leveraging the appropriate headers provided by each technology. Whether you use Varnish , NGINX , Cloudflare , Akamai , Incapsula , or other solutions, following IP restoration best practices is essential to ensure reliable and compliant logs.

Conclusion

Web server log management is crucial for ensuring legal compliance, security, and transparency. In Italy, regulations require careful log retention, adhering to both the stringent rules of the Electronic Communications Code and the general principles of the GDPR , such as data minimization and data retention limitations. This balance must be carefully maintained, especially when using advanced technologies such as reverse proxies and CDNs.

Using services like Cloudflare , Akamai , Fastly , or Varnish introduces complexity in managing real IP addresses, which can be replaced by proxy IPs if not configured correctly. This error can invalidate logs for legal or security purposes and reduce their usefulness for technical analysis. It is essential to implement the necessary configurations to correctly log real IPs, for example, by using the specific headers of reverse proxy technologies.

Additionally, it is essential to establish clear communication with customers, especially when they manage their own domains. Hosting providers should inform customers about the potential impacts of CDNs or reverse proxies on logging in the preliminary stages of the contract. If a customer decides to use technologies such as Cloudflare, it is important to notify the provider to ensure the system is configured correctly.