Data Pipeline Architect

The Opportunity:

Consulting as a Data Pipeline Architect on a major central government project to provide the required services to develop robust data pipelines, analytics and visualisation tools that will support the integration of multiple infrastructure datasets across various sectors.

These services will contribute to the creation of solutions that align with the principles of digital twins, which are virtual models connected to their real-world counterparts by a two-way flow of real-time data where the primary challenge is to consolidate these sources into one cohesive framework.

N.B. Candidates will have or be willing to undergo a basic level of security clearance prior to starting the role.

Skills and Experience:

• Architectural Design: proven experience in implementing robust and scalable software architectures that meet both functional and non-functional requirements.
• Data Pipeline Architectures: proven experience in implementing large-scale data warehousing solutions and data streaming technologies (such as Apache Kafka).
• Data Pipeline and Application Scalability: experience in designing hosting and runtime architectures to scale ETL data pipelines.
• Security Expertise: strong background in application and data security, including overseeing penetration tests and implementing security remediations.
• Identity and Access Management: hands-on experience with federated authentication solutions using open standards (OAuth 2.0, OIDC, SAML).
• RBAC/ABAC: experience in implementing RBAC/ABAC access solutions in a national-scale service based on user profiles and data attributes.
• Cloud Technologies: extensive experience with AWS cloud services and best practices for cost optimisation and performance.
• Containerisation: proficient in using Kubernetes (AWS EKS preferred) for deploying and managing applications and ETL components in a containerised environment.
• Stakeholder Management: excellent ability to engage with stakeholders, facilitate discussions and achieve consensus on technical solutions.
• Technical Communication: strong verbal and written communication skills, with the ability to convey complex technical concepts to both technical and non-technical audiences.
• Infrastructure as Code (IaC): experience in deploying cloud environments using tools like Terraform.
• CI/CD Solutions: proven ability to design and implement continuous integration and continuous deployment pipelines using GitHub Actions.

Please call Edward here at ISR to learn more about our client leading the way in developing the next-generation of digital-twin solutions through innovation and transformational technology?