Gen AI
- AIML
Role Objective
You will build a suite of AI-driven Accelerators and Autonomous Agents designed to automate the entire lifecycle of our Data Platform—from legacy migration and pipeline construction to incident response and ML lifecycle management. Your goal is to turn the Data Platform into a self-service, self-healing ecosystem.
Key Focus Areas & Deliverables
1. Migration & Construction Accelerators
- Legacy to dbt Engine: Build agents to parse legacy SQL, Stored Procedures, and ETL tool exports into modular, optimized dbt models.
- Copilot for Pipelines: Develop an internal "Pipeline Assistant" that generates boilerplate Spark/Python code and dbt transformations based on metadata and business requirements.
2. Autonomous Incident & Problem Solver
- Self-Healing Ops: Build agents that monitor platform logs (Spark, dbt, Cloud-native) to identify failures, explain the root cause in plain language, and automatically propose (or commit) a fix.
- On-call Assistant: An AI agent that triages data quality alerts, identifies upstream dependencies, and notifies affected downstream stakeholders.
3. Machine Learning (MML) Construction
- Auto-MML: Develop agents to accelerate the "MML" (Machine Learning Modeling) lifecycle—from automated feature engineering suggestions to skeleton code for training and deployment.
Technical Stack & Requirements
AI & Agentic Frameworks
- Agent Orchestration: Expertise in LangChain, CrewAI, or AutoGPT to build multi-agent workflows.
- Code Generation: Proven experience using LLMs for Code-to-Code translation (specifically SQL to dbt/Python).
- RAG Mastery: Building advanced Retrieval-Augmented Generation systems over technical documentation, data catalogs, and codebases.
Data Engineering & Platform
- Expert dbt: Deep knowledge of dbt Jinja, Macros, and Mesh—essential for teaching an AI how to write "good" code.
- Spark & Lakehouse: Understanding of Spark internals to build agents that can optimize job performance and resolve OOM (Out of Memory) errors.
- Cloud Infrastructure: Experience with GCP (Vertex AI) or equivalent cloud AI suites
contact@qtsolv.com
