Data Engineer- RAG

About the Role

We are seeking a Senior Data Engineer to join the RBQM Production Pod. In this role, you will design, build, and maintain data pipelines that power AI/GenAI-driven applications for Risk-Based Quality Management (RBQM) in clinical trials. You will be responsible for RAG document ingestion, vector database indexing, and developing scalable data APIs to support AI applications. Strong expertise in modern AI data architectures is essential, as traditional ETL and data warehouse experience alone will not meet the requirements of this role.

Key Responsibilities

• Design and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality data
• Build and manage vector databases (AWS OpenSearch) for RAG-powered AI workflows
• Develop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX, clinical reports)
• Build and expose data APIs for AI application consumption
• Optimize chunking strategies, embedding generation, and retrieval performance for RAG architectures
• Manage data quality, lineage, and governance for AI/ML data pipelines
• Deploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
• Collaborate with Data Scientists and Backend Developers in an integrated pod team

Requirements

• 5–7+ years hands-on data engineering at scale
• RAG document ingestion pipelines (chunking, embedding, vector indexing) — traditional ETL/DWH experience alone is not sufficient
• AWS OpenSearch (vector database for RAG workflows)
• Python (advanced) including SQL/Spark SQL
• Unstructured data transformation (PDF, DOCX) for RAG/LLM applications
• AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
• Containerization: Docker
• Must demonstrate building custom pipelines from scratch — not just configuring out-of-the-box services