Sensitive data anonymisation on Airflow
Created: May 2025 Updated: Sept. 29, 2026
from work
Python, Apache Airflow, AWS S3, Pandas, Docker, GitLab CI/CD, pytest
A pipeline that takes raw files containing personal data, cleans them, anonymises them and checks them before anyone starts working with them.
CSV files with personal data arrive in S3, and before anyone starts working with them, personal data has to be removed or masked, formats unified and every file checked for whether it makes sense at all. The files vary in quality, some are incomplete, and under GDPR you have to be able to show later what happened to the data.
Where to do it
The first idea is to load everything into the database and mask the personal data with views or procedures, except the database is exactly where it wasn't supposed to be. The second, one script run from cron, works until the first time it fails halfway and nobody knows what has already gone through. We settled on Airflow, because retries and run history are built in, and that history doubles as the audit trail GDPR asks for.
How it's put together
A set of DAGs, each made of small steps: fetching the file from S3, cleaning, unifying formats, anonymising, validating the schema and quality rules. When something fails, Airflow retries only that step, and for each data type you can decide what happens to a file that fails validation.
from datetime import datetime from airflow.decorators import dag, task @dag(schedule=None, start_date=datetime(2025, 1, 1), catchup=False) def anonymise_file(): @task(retries=3) def fetch() -> str: ... # the S3 file given when the run is triggered @task def clean(path: str) -> str: ... @task def anonymise(path: str) -> str: ... # this is where personal data is removed or masked @task def validate(path: str) -> str: ... validate(anonymise(clean(fetch()))) anonymise_file()
Everything runs in Docker, pytest tests run in GitLab CI on every change and use synthetic data, so real data never ends up in the repository or in CI.
Machine-translated from Polish (original).