AI Architect - Databricks

AI Architect – Databricks

Primary Skills: Databricks, Apache Spark, Python, SQL, MLOps, Machine Learning, Cloud Architecture

Skill Level: Master

────────────────────────────────────────────────────────

Role Overview

AI Architect responsible for designing and implementing enterprise AI, machine learning, and modern data platform solutions using the Databricks Lakehouse Platform. The role focuses on building scalable AI architectures, enabling advanced analytics, generative AI, and MLOps capabilities while ensuring secure, high-performing, and governed AI solutions. The ideal candidate combines deep expertise in Databricks, distributed computing, cloud platforms, and AI technologies with strong leadership and stakeholder management skills.

────────────────────────────────────────────────────────

Key Responsibilities

• Design and implement enterprise AI and machine learning architectures using the Databricks Lakehouse Platform
• Lead the development of scalable machine learning, generative AI, and advanced analytics solutions
• Architect and optimize data pipelines, feature stores, model lifecycle management, and MLOps frameworks
• Design scalable Lakehouse architectures using Delta Lake and modern data engineering practices
• Define best practices for data governance, security, monitoring, compliance, and AI model deployment
• Collaborate with business stakeholders, data scientists, data engineers, and platform teams to translate business requirements into scalable AI solutions
• Integrate Databricks with cloud platforms including Microsoft Azure, AWS, and Google Cloud Platform
• Drive adoption of Databricks technologies including MLflow, Delta Lake, Unity Catalog, Mosaic AI, and Apache Spark
• Design and support generative AI solutions leveraging Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG)
• Support AI production deployment, performance optimization, scalability, and operational excellence
• Lead architecture workshops, mentor engineering and data science teams, and promote AI architecture best practices
• Produce architecture documentation, technical roadmaps, and solution blueprints
• Evaluate emerging AI and Databricks technologies and recommend strategic improvements

────────────────────────────────────────────────────────

Required Skills

• Strong experience designing enterprise AI, machine learning, and cloud architectures
• Hands-on experience with the Databricks Lakehouse Platform
• Strong expertise in:
• • Apache Spark
  • Python
  • SQL
  • Distributed computing
• Experience designing, deploying, and operating machine learning models in production environments
• Strong understanding of:
• • MLOps
  • CI/CD pipelines
  • Model governance
  • Model monitoring
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform
• Strong knowledge of Lakehouse architectures and modern data platforms
• Experience with AI and machine learning frameworks including:
• • TensorFlow
  • PyTorch
  • Scikit-learn
  • LangChain
• Experience designing REST APIs, microservices, and distributed applications
• Experience with containerization and orchestration technologies:
• • Docker
  • Kubernetes
• Strong communication, stakeholder management, and problem-solving skills
• Ability to lead technical architecture discussions and mentor engineering teams

────────────────────────────────────────────────────────

Nice to Have

• Databricks certifications, including:
• • Databricks Certified Data Engineer
  • Databricks Machine Learning Professional
  • Databricks Solution Architect
• Experience with Generative AI, Large Language Models (LLMs), vector databases, embeddings, and Retrieval-Augmented Generation (RAG)
• Knowledge of governance and security frameworks in regulated environments
• Experience working in industries such as finance, healthcare, retail, or manufacturing
• Experience with Infrastructure as Code (IaC) tools such as Terraform
• Familiarity with DevOps practices and Agile delivery methodologies
• Experience implementing enterprise-scale AI platforms and data governance solutions

────────────────────────────────────────────────────────

Candidate Profile

Experienced AI Architect with deep expertise in the Databricks Lakehouse Platform, enterprise AI, and modern data architectures. Proven ability to design scalable, secure, and production-ready AI and machine learning solutions while leading technical strategy across complex cloud environments. Strong background in Apache Spark, MLOps, cloud platforms, and generative AI technologies, with the ability to collaborate across business and engineering teams, mentor technical specialists, and drive innovation through modern AI and data platform architectures.