Senior Data Engineer Full Time

Responsibilities

• Design and build robust ingestion pipelines using ELT and schema on read in Databricks Delta Lake.
• Develop transformations within ELT frameworks to modernize ingestion pipelines and scale data transformations.
• Provide technical expertise in the design and implementation of Ratings Data Ingestion pipelines utilizing modern AWS cloud technologies such as S3, Hive, Databricks, Scala, Python, and Spark.
• Maintain a high-performance data environment focused on speed, accuracy, consistency, and uptime.
• Collaborate with data and Data Science teams to develop efficient ingestion pipelines.
• Implement data governance, quality checks, and data lineage throughout the data handling processes.
• Stay abreast of emerging trends in big data and cloud technologies, evaluating new technologies for potential adoption.
• Ensure compliance with enterprise standards and best practices aligned with organizational standards.

Requirements

• Minimum of 8 years of significant experience in application development with a strong background in system architecture, object-oriented design, and design patterns.
• Expertise in streaming pipelines, particularly with Kafka and Spark structured streaming.
• Proficient in SDLC methodologies such as Agile and test-driven development.
• Experience in handling high-volume data and computationally intensive systems.
• Knowledge of tuning Java garbage collection and performance optimization.
• Proficiency in development environments including IDE, web & application server, GIT, CI tools, unit-testing, and defect management tools.
• Domain knowledge in the Financial Industry and Capital Markets is highly desirable.
• Excellent communication skills with strong verbal and writing proficiencies.

Benefits

• An unmatched experience in handling massive volumes of data and advanced analytics.
• The opportunity to work on converting innovative ideas into revenue-generating streams.
• A dynamic work environment where one can mentor teams, innovate, and experiment.
• The chance to give a face to business ideas and present to key stakeholders.