Software Engineer II (Data Engineering)

Role Description

Job: Software Engineer II (Data Engineering)

Location: Bangalore

Responsibilities

• Design, develop, and maintain scalable data processing pipelines using the Spark ecosystem, including
  
  

PySpark, Databricks, and Synapse Notebooks.

• Develop and manage data pipelines using Azure Data Factory, Azure Synapse, and Azure Data Lake to
  
  

ingest, process, and integrate data from Azure and on-premises sources.

• Implement full and incremental data load strategies, ensuring efficient, reliable, and timely movement of
  
  

data across source systems and target platforms.

• Perform data cleansing, transformation, aggregation, and modeling to prepare high-quality datasets for
  
  

analytics, reporting, and downstream consumption.

• Apply Azure security, access control, and data governance practices to ensure secure and compliant data
  
  

processing and storage.

• Optimize Spark jobs, data pipelines, and SQL workloads for performance, scalability, reliability, and cost
  
  

efficiency.

• Collaborate with BI and analytics teams to support Power BI reports and dashboards, including
  
  

datasets/models hosted on Azure Analysis Services (AAS) and Power BI.

• Troubleshoot data pipeline and processing issues, perform root-cause analysis, and implement appropriate
  
  

fixes to maintain data availability and quality.

• Develop reusable frameworks, processes, and best practices for data ingestion, transformation, validation,
  
  

and monitoring.

• Work closely with business stakeholders, data analysts, and engineering teams to translate data
  
  

requirements into scalable and maintainable data solutions.

Key Skills Required

Data engineering & Analytics:

• 5+ years of experience in statistics and analytics on large-scale datasets
• Strong SQL and data analysis skills, including SQL DW concepts
• Experience with Spark ecosystem: PySpark, Databricks, Synapse Notebooks (1+ year hands-on)
• Proficiency in Azure Data Services:
• Azure Synapse
• Azure Data Factory
• Azure Data Lake
• Azure security model
• Experience building pipelines for full and incremental data loads from Azure and on-prem sources
• Strong knowledge of data management, processing, cleaning, transformation, aggregation, and modeling
• Working knowledge of Power BI reports and dashboards (AAS / PBI)
  
  

Systems, Cloud & DevOps

• Strong programming skills in Python (Java is a plus)
• Experience with FastAPI, RESTful services, and microservices architecture
• Proficiency with databases such as MySQL, PostgreSQL, and MongoDB
• Experience deploying applications using Docker and Kubernetes
• Experience with streaming and messaging systems: Kafka, Kinesis, RabbitMQ
• Familiarity with Azure Git and DevOps practices
  
  

Leadership & Soft Skills

• Proven experience leading projects end-to-end
• Strong mentoring and team leadership experience
• Excellent written and verbal communication skills
• Ability to work effectively in distributed teams
• Strong understanding of data structures and design patterns
• Exposure to the retail industry is a plus
  
  

Good To Have Skills

• AI /ML and Agentic AI
  
  

Education

• Bachelor’s degree in computer science or a related field (required)
• Master’s degree is a plus