Lead Data Platform Engineer

Role Overview – Lead Data Platform Engineer

We are embarking on a full-scale digital transformation where data will be a foundational element. As part of this journey, we’re creating a decentralised, self-service data platform that allows engineering teams to independently publish and access high-quality datasets. Drawing on data mesh concepts, the platform will promote autonomy while upholding common standards to ensure seamless integration and interoperability across teams.

To accelerate this initiative, we are looking for a technically strong and proactive Lead Data Platform Engineer. In this role, you will be instrumental in designing and building the platform’s key features, selecting suitable technologies, and collaborating across engineering, platform, and infrastructure teams. Your focus will be on creating a scalable and developer-friendly environment that enables teams to efficiently build and share data products.

Key Responsibilities:

• Architect and implement core elements of the data platform, including dataset cataloguing, storage standards, publishing SDKs, metadata management, and orchestration
• Evaluate and integrate open-source tools such as Dagster, Flyte, DataHub, or Amundsen
• Define how Databricks will integrate within the broader data platform and optimise its interoperability
• Establish best practices and tooling for data formats (e.g., Parquet), access control, versioning, and schema evolution
• Develop robust pathways for data production and consumption across the organisation
• Ensure platform extensibility, observability, and ease of use through thoughtful design

What We’re Looking For:

• 7+ years of experience in software engineering, with a focus on data-driven systems or platform development
• Familiarity with orchestration tools like Airflow, Dagster, or Flyte
• Proficient in working with major cloud providers (preferably Azure; GCP or AWS also relevant)
• Hands-on experience with Databricks is highly desirable
• Solid understanding of data ownership, metadata, and lineage principles
• Previous involvement in internal platform or developer tooling projects
• Strong skills in Python, command-line tooling, and ideally SQL
• Ability to balance rapid delivery with scalable architecture and iterate efficiently
• Strong communicator who thrives in cross-functional collaboration and enjoys enabling others

Bonus Qualifications:

• Experience integrating with tools like Power BI, Backstage, or metadata solutions
• Knowledge of Delta Lake and lakehouse architecture
• Background in data engineering, operations, or developer experience domains

About Rasulson Consulting

• Rasulson Consulting is a specialized staffing and recruitment firm focused on the IT sector. We collaborate with leading tech companies and innovative startups to provide exciting career opportunities for individuals passionate about digital development. With our deep technical expertise and extensive network, we efficiently match the right talents with the right assignments. At Rasulson Consulting, you’ll receive personalized guidance, regular feedback, and the chance to take the next step in your IT career.