Excellent Opportunity with our client forLead Data Engineer (offshore) (Job Code : J47768)Experience:4 - 8 Years Location:Mumbai Key Skills:SQL AND Python AND Spark AND (Dagster OR Airflow OR Air flow OR Air-flow)Please pass on or Share this message to your friends who are looking for similar roleshttps://www.sampoorna.com/JobPostingDisplay.aspJobID=30977&JRID=0.122599Client is looking out for a technical delivery leader on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions.The role will drive the development of data processing frameworks and architectural approaches that scale to handle the business's growing demands.This hands-on role requires technical expertise to deliver data and analytical services, including data ingestion, transformation, storage, and reporting and strives to continuously develop new and improved data engineering capabilities.The key accountabilitiesDesign data solutions to meet business, technical and user requirements. This includes building modern data pipelines that meet functional/non-functional business requirements and provide end-to-end data solutionsParticipate in proofs-of-concept on platform innovation, and effectively transition and scale those concepts into production at scale through, engineering, deployment and commercialisationContribute to ongoing management of the platform, including platform enhancements, capacity management, performance monitoring, troubleshooting and resolution of technical issuesIdentify, design, and implement process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalabilityTechnical ownership for data and data pipelines to ensure compliance with data standards, architectural standards, and achievement of business requirementsBuild a strong understanding of platform architecture and the best way to leverage it to achieve effective outcomes for a given projectSupport compliance with data, information and security management requirementsTechnical qualifications & skills:Experience in building and maintaining real-time data pipelines.Expected experience using the following tech stackdbtSQLSQLSparkPythonDagster/AirflowPreferred but not mandatoryKubernetes and TerraformCI/CD - Github actions, CircleCIDatadog / Cloudwatch - LoggingPrior work with stream-processing systems (e.g., Kafka, Kinesis).Exposure to Trino, Dagster, Cubejs, metabase/looker will be advantageo
