Anonymizing logs for GDPR using Logback pattern layout

Logs are usually super useful, but they can also be a trap in a GDPR world. Effective ways of anonymizing them might become in handy.

In the new GDPR-present world, of many things, special attention must be given to logging. It stems from the fact that not only the brand new rulebook obliges companies to respect users’ right to be forgotten but also to make every effort needed to protect their privacy.

Thus far, a common practice in many applications could have been described as “extensive logging”. In fact, it might have to do with being pragmatic – disk space is cheaper than programmers’ time. If by logging more, it is easier and quicker for the developer to debug potential problems then it seems like a no-brainer. At least it had until May 25th.

GDPR reminded many of us how important users’ privacy is and that it is the responsibility of every application to respect that. Therefore it is essential for the logs to be GDPR-compliant. Here are some recommendations to achieve this:

Do not log user’s sensitive (private) data unless it is necessary

This states the obvious, but in real world scenarios it is not always easy to achieve. Nevertheless, you should strive for building your applications in a GDPR-driven fashion and one way of this is by paying special attention to your logs.

Set up reasonable log retention

In most cases you do not need logs after a specific period of time (depending on your business). Ideally you should set up the retention policy for all your logs in one place. If you use any log management service, it should be fairly easy.

Structure your logs

If, for some reason, it is necessary for you to log private data do it in a structured way. JSON, XML or any other machine-friendly format is a good recommendation. Try not to log sensitive information in a “random” fashion, think about how easy would it be to find that piece of data using regular expression – structured logs are easier to anonymize.

Anonymize/mask sensitive data

If you find yourself in a situation in which you already log something that may contain private data, you should consider implementing anonymization mechanisms. Solutions may differ depending on your needs – encryption, masking or complete removal are some of the possible options.

Masking Logs using Logback pattern layout

Logging request/response payloads is generally considered a good practice, but in the light of GDPR it is advised to prepare for that they may contain private data that you are supposed to anonymize. Let’s assume your application received the following request that you logged:

As you can see, structuring your logs lets you identify where the private data may be – at first glance ssn (Social Security Number) and the address are the fields that you should mask. Luckily enough, you can easily parse the log using regular expression to find them. Moreover, the Cookie header is something you should not log. You know the drill already. But wait! Client that uses your API put some other personal information in two text-free fields: additional_info_1 and additional_info_2. Fortunately, IP address and e-mail are also quite easy to catch by the regex. Those are two good examples of sensitive information you should be looking out for in your logs to make your  masking mechanism more bullet-proof.

How to make sure all this information is always masked in your logs?

There are, of course, many ways. One solution is to implement a custom servlet request/response filter. This is something we tried in our team by implementing JAX-RS custom ClientRequestFilter and ClientResponseFilter, but the biggest downside of the approach was that it only filters request/response payloads and not all logs.

Thus we needed to dig “deeper” and decided to mask the logs centrally, by configuring masking rules for all log entries produced by Logback. In order to do that we had to implement custom ch.qos.logback.classic.PatternLayout.

Configuration:

The idea behind the configuration is to extend every logback appender you need with custom layout, in our case it is: PatternMaskingLayout which is an implementation of ch.qos.logback.classic.PatternLayout Each mask pattern represents regular expression to fetch one piece of sensitive data.

PatternMaskingLayout:

Implementation of doLayout method from PatternLayout is responsible for masking matched data in each log message of your application if it matches one of the configured patterns. A multiline pattern is constructed from maskPatterns list taken from logback.xml. Unfortunately, logback engine does not support constructor injection (Or does it? I couldn’t find better solution) if it comes to list of properties, therefore addMaskPattern is invoked for every config entry (if there was a single property, setter method would be invoked) so we have to compile the pattern every time we add new regex to the list.

After that has been applied, when we run the application our example log entry looks like this:

Useful tips

Naturally, you always have to play carefully with the matchers. If the regex is broken, you may end up masking too much. Notice that I used reluctant quantifiers in my regular expressions. It wasn’t a coincidence – let’s change the ssn regex to use greedy quantifier to illustrate that:

\"ssn\"\s*:\s*\"(.*)\"

Output log message this time looks like this:

Bummer! Not exactly what we wanted to accomplish.

Final thoughts

To wrap up – even if you are very careful with logging, it’s inevitable that some day you will discover a piece of information there that should have never been logged. Whether it’s a new teammate who hasn’t fully got into common practices of the project, misbehaving 3rd party library or unwary API consumer – you should be prepared for the worst and build as many protection layers you can. One of them could be masking sensitive data in your application logs.

Helpful links

Subscribe to our newsletter
Menu