Senior Software Engineer – Data Quality Framework (PySpark / Databricks)

Location: Heerlen or Amsterdam, Netherlands (Onsite: 2 days per week after onboarding)
Hours per week: 40
Contract Type: Freelance / Interim Assignment (Not suitable for ZZP freelancers)
Start Date: ASAP
Duration: Initial contract with extension option
Maximum Hourly Rate: €42
Education Level: Bachelor's degree in Computer Science, Software Engineering, Data Engineering, or related field
Experience Required: Minimum 5 years
Travel Requirement: 25% – 50%
Language: English (Dutch is a plus depending on team setup)

Role Overview

We are seeking a Senior Software Engineer – Data Quality Framework to design, build, and maintain a scalable data quality framework within Databricks using PySpark.

You will play a key role in developing reusable framework components that support data validation, rule execution, and contract-driven data quality checks across enterprise data environments. The role combines strong software engineering practices with data platform engineering in a modern cloud-based ecosystem (Azure Databricks).

You will work in a multidisciplinary Agile team responsible for delivering reliable and scalable data capabilities that support critical business and data products.

Key Responsibilities Data Quality Framework Development

• Design, develop, and maintain a PySpark-based data quality framework in Databricks
• Build reusable components for rule execution, validation logic, and result handling
• Implement contract-driven data validation mechanisms across datasets
• Develop scalable and maintainable framework architecture

Engineering & Platform Ownership

• Ensure high engineering standards through:
  • Code refactoring and modular design
  • Unit and integration testing
  • CI/CD pipeline implementation and maintenance
  • Packaging and versioning of framework components
• Support multi-environment deployment strategies (dev/test/prod)

Databricks Platform Operations

• Develop and manage Databricks jobs, workflows, alerts, and notifications
• Support operational stability of data pipelines and framework execution
• Contribute to monitoring and reliability improvements

Stakeholder Collaboration

• Work closely with data engineers, analysts, and platform teams
• Translate data quality requirements into scalable technical solutions
• Contribute to documentation, standards, and best practices
• Support coaching and knowledge sharing within the team

Assignment Deliverables

• A scalable and reusable data quality framework in Databricks
• Robust PySpark-based validation and rule execution components
• Stable CI/CD and deployment processes for framework components
• Improved reliability and governance of data quality processes
• Documented standards, guidelines, and reusable engineering patterns
• Operational monitoring setup (alerts, triggers, notifications)

Required Qualifications

• Minimum 5 years of experience in software engineering, data engineering, or platform engineering
• Strong proficiency in Python and PySpark
• Experience working with Apache Spark in production environments
• Hands-on experience with Databricks (Azure preferred)
• Strong understanding of:
  • Software design principles (OOP, modular design, maintainability)
  • CI/CD pipelines and testing strategies
  • Multi-environment cloud deployments
• Experience with version control, packaging, and release processes
• Strong communication skills in English
• Ability to work independently and take ownership of tasks

Preferred Experience

• Experience building or maintaining data quality frameworks or rule engines
• Knowledge of metadata-driven validation approaches
• Experience with data contracts and governance concepts
• Familiarity with Databricks alerts, workflows, and operational tooling
• Experience with Azure DevOps and Artifactory
• Experience developing reusable platform components or shared libraries
• Exposure to enterprise-scale cloud data platforms

Key Competencies

• Strong software engineering mindset
• High attention to code quality and maintainability
• Analytical and structured problem-solving ability
• Ownership and proactive delivery attitude
• Ability to work across technical and business stakeholders
• Strong collaboration in agile teams
• Platform thinking and scalability focus