Log In

Don't have an account? Sign up now

Lost Password?

Sign Up

Technology and IT

Data Engineering Technical Lead Full Time

Summary:

  • The Data Engineering Technical Lead will be part of iiDENTIFii’s central Data Team, helping to elevate their data practices.
  • This position focuses on refining the current platform and contributing to the development of a next-generation data platform, utilizing industry best practices such as the Medallion Architecture and cloud-based technologies.
  • The Technical Lead will collaborate with leaders, senior engineers, and architects to drive the data platform architecture, ensuring alignment with the company’s broader technology roadmap.
  • The role involves leading Data Engineers and Analysts to implement data platform architecture and build data use-cases to meet end-user requirements.
  • The Data Team at iiDENTIFii emphasizes quick, sustainable value delivery and leverages automation for efficiency and collaboration.

Responsibilities:

  • Lead the design, orchestration, and implementation of robust data pipelines for batch and real-time processing.
  • Ensure best practices in data engineering, monitoring, and observability are followed, using tools like Databricks, Azure Data Factory, Synapse, or Airflow.
  • Mentor junior team members and help foster their growth into experts.
  • Collaborate with executives and stakeholders to scale data platform capabilities and integrate them with other digital products.

Requirements:

  • Leadership Experience: 2+ years of experience leading teams, mentoring, and coaching, with a passion for leadership and team growth.
  • Data Engineering Expertise: 5+ years of experience building modern, cloud-native data platforms with deep hands-on expertise in:
    • Designing scalable and resilient pipelines for big data processing and real-time streaming using Apache Spark and Databricks.
    • Orchestrating data workflows using tools like Airflow, Databricks Jobs, Azure Synapse, or Azure Data Factory.
    • Working with streaming platforms like Apache Kafka, RabbitMQ, or Azure Event Hubs for real-time data processing.
    • Developing metadata-driven pipelines for flexible and dynamic data workflows.
    • Strong understanding of data security, governance, metadata management, and data architecture.
    • Proficiency with Linux, Python programming (object-oriented), and scripting for data automation.
    • Experience with DevOps, DataOps, and CI/CD pipelines.
  • Cloud Technology: 3-5 years of experience with Microsoft Azure (Azure, Azure Data Factory, Databricks) with exposure to other cloud platforms (AWS/GCP) being a plus.
  • Real-Time Processing: Hands-on experience building and maintaining streaming architectures using Kafka, Event Hubs, or similar technologies.
  • BI & Visualization: 2+ years of experience with tools like Power BI, Grafana, or other visualization platforms, focusing on secure and optimized deployments.
  • Monitoring & Observability: Familiarity with tools like Application Insights, Azure Monitor, and Grafana for tracking pipeline health and optimizing resource usage.
  • Leadership Skills: Proven ability to lead teams, drive performance, resolve conflicts, and align the team.

Benefits:

  • Competitive salary offered.
  • Remote working conditions, with a preference for Cape Town.
  • Wellness perks.
  • Opportunities for career growth and development within the company.
  • Work-life balance is prioritized.
  • Diverse tech stack.

Tagged as: , , , ,

Job Overview