Senior Data Engineer (immediate role) Full Time

Summary:

• The Senior Data Engineer role involves participating in business discussions and gathering data requirements.
• The position requires proficiency in writing complex SQL queries for data extraction, transformation, and analysis.
• The engineer will be responsible for navigating source systems with minimal guidance and using data profiling to better understand data relationships.
• Hands-on experience with PySQL/PySpark and creating/managing data pipelines using Azure Data Factory is required.
• Knowledge of data integration, transformation, and workflow orchestration in Azure environments is essential.
• The role involves expertise in Databricks workflows and best practices, including the use of Unity Catalog.
• The Senior Data Engineer must be proficient in Git for version control and be able to work effectively in agile or collaborative settings.
• Clear communication skills to document and share findings with team members are required.
• The candidate should be adaptable and willing to learn new tools and techniques as the field of data analytics evolves.

Responsibilities:

• Participate in business discussions and assist in gathering data requirements.
• Apply analytical and problem-solving skills to address data challenges.
• Write complex SQL queries for data extraction, transformation, and analysis.
• Navigate source systems to understand how data is related and use data profiling for insights.
• Hands-on experience with PySQL/PySpark for data processing and analysis.
• Create and manage data pipelines using Azure Data Factory.
• Understand data integration, transformation, and workflow orchestration in Azure environments.
• Apply best practices for data engineering workflows, particularly in Databricks.
• Work with existing templates and patterns for development in Databricks workflows.
• Use Git for version control and collaborate effectively in data projects.
• Document processes, workflows, and data analysis results effectively.
• Communicate findings and recommendations clearly to other team members.
• Adapt to changing project requirements and priorities.
• Willingness to learn new tools, technologies, and techniques in data analytics.

Requirements:

• 7+ years of overall experience, with 5+ years of expertise in Azure technologies.
• Azure Databricks, Data Lakehouse architectures, and Azure Data Factory proficiency.
• Expertise in optimizing data workflows and predictive modeling.
• Experience designing and implementing data pipelines using Databricks and Spark.
• Expertise in batch and streaming data solutions and automating workflows with CI/CD tools like Jenkins and Azure DevOps.
• Knowledge of data governance with Delta Lake.
• Proficiency in Spark, PySpark, Delta Lake, Azure DevOps, and Python.
• Certification in Azure technologies is mandatory.