Your responsibilities
- Data Ingestion: Own and optimize our ingestion layer using Airbyte (or similar tools), build custom connectors where required, and ensure reliable scheduling, retry mechanisms, and observability. Drive the transition from full data syncs to incremental (delta) loading wherever it adds value.
- Bronze & Silver Data Layers: Design, maintain, and monitor clean, reliable, and well-defined Bronze and Silver layers in Databricks, providing the trusted foundation for the Analytics Engineer's dbt models.
- Schema Contracts: Establish and maintain robust schema contracts between the backend platform and the data platform to ensure compatibility and prevent production incidents.
- Data Quality & Reliability: Build and operate tooling for data quality, including automated testing, monitoring, alerting, schema change notifications, and data lineage.
- Compliance Enablement: Implement the technical capabilities required for data deletion workflows, Data Subject Access Requests (DSARs), and Records of Processing Activities (RoPA). The legal framework is owned by the DPO and VP; your responsibility is its technical implementation.
- AI Data Infrastructure: Build and maintain the data infrastructure required for AI use cases by making evaluation datasets, training corpora, and inference logs available in a secure, anonymized, reproducible, and production-ready manner.
