Data Platform Lead / Cloud Systems Architect

Summary / Primary Role:

We are seeking a Data Platform Lead / Cloud Systems Architect to own Odyssey’s enterprise analytics ecosystem. You will manage everything from terraform deployments to our emerging GenAI infrastructure. Beyond technical execution, you will serve as a strategic guardian against engineering inertia, enforcing a discipline of rational adaptability to ensure investments are driven by objective future value rather than past expenditures.

Principal Duties and Responsibilities:

• Architect and maintain a multi-region Databricks on AWS environment, enforcing "secure by default" networking via customer-managed VPCs, PrivateLink, Transit Gateway, and strict IAM cross-account roles.
• Design the underlying infrastructure for Generative AI, including configuring MLflow for model governance, Databricks Vector Search for RAG applications, and GPU-optimized compute clusters for inference.
• Enforce reproducibility by managing all platform resources (workspaces, clusters, jobs) via Terraform and Databricks Asset Bundles (DAB), ensuring no manual changes exist in Production.
• Systematically audit legacy technical choices with a "clean slate" perspective. You are expected to seek disconfirming evidence that contradicts the status quo to prevent confirmation bias and the escalation of commitment to outdated strategies.
• Lead monthly FinOps reviews using AWS Cost Explorer and Databricks system tables to differentiate between necessary investment and "sunk costs," ensuring we do not irrationally commit resources to failing projects simply because they are already underway.
• Own strategic vendor relationships (Databricks, AWS, Fivetran, Sigma), holding partners accountable for successful outcomes and resolving support blockers aggressively rather than passively accepting roadmap delays.
• Drive a Data Mesh culture by treating data assets as "Data Products." Define clear contracts, SLOs (Service Level Objectives), and publication standards to decouple producers from consumers.
• Design and operationalize data privacy frameworks to satisfy GDPR, CCPA, and SOC2 requirements, including automated workflows for "Right to Be Forgotten" (RTBF) requests and PII masking within the Lakehouse architecture.
• Administer the Sigma Computing environment, overseeing workspace architecture, version tagging strategies, and the promotion path of analytics assets from Dev to Production.
• Design and enforce comprehensive Role-Based Access Control (RBAC) policies across Unity Catalog (Catalogs, Schemas, Tables) and AWS IAM, ensuring "least privilege" access while maintaining operational velocity.
• Manage high-volume data replication pipelines using Fivetran and AWS DMS, ensuring data fidelity, efficient schema drift handling, and cost-optimized sync frequencies across heterogeneous sources.
• Perform rigorous code reviews on high-impact pipelines, enforcing best practices such as Z-ordering, Liquid Clustering, and Schema Evolution, with the authority to reject sub-standard code.
• Partner with analytics, data science, and business teams to design scalable data solutions that align to strategic priorities and unlock advanced use cases.
• Drive innovation by introducing emerging technologies, frameworks, and best practices into engineering workflows to improve scalability, automation, and productivity.

Qualifications and Key Skills:

• 8+ years in Data Engineering/Architecture, with 3+ years of daily production mastery in Databricks on AWS.
• Deep familiarity with global data compliance standards (GDPR, CCPA, HIPAA) and experience implementing technical controls for PII protection (e.g., dynamic views, column-level encryption) in a distributed data environment.
• Expert-level proficiency with modern ELT and BI tools, specifically Fivetran (connector configuration, transformation), AWS DMS, and Sigma Computing (administration, version control, security).
• Deep understanding of Role-Based Access Control (RBAC) models within distributed data systems, specifically regarding Unity Catalog grants and AWS IAM Identity Center.
• Deep expertise in AWS networking (Transit Gateway, VPC Endpoints, Private Subnets) and security (SCPs, Encryption, IAM identity center).
• Proven production experience deploying CI/CD pipelines via GitHub Actions and Databricks Asset Bundles (DAB).
• Demonstrated experience managing enterprise software vendors, including contract utilization, technical escalation, and roadmap alignment.
• A track record of making high-stakes technical decisions, including examples of when you recommended stopping a project to save resources (managing Sunk Cost bias).
• Databricks Certified Data Engineer Professional (strongly preferred) or AWS Solutions Architect Professional.
• Familiarity with Generative AI concepts and modern ML techniques (LLMs, transformers, ensemble methods, deep learning frameworks) and how they integrate into data engineering workflows.
• Working knowledge of agile/scrum methodologies and use of agile lifecycle management tools (e.g., Jira, Confluence).
• English level: advanced.

Show more Show less