Design, develop, and optimize scalable Azure Databricks data pipelines, lakehouse solutions, and cloud data platforms. Build batch and streaming ETL/ELT processes using Spark, PySpark, Delta Lake, Auto Loader, and Structured Streaming. Implement data modeling, SCD, CDC, governance, Unity Catalog, integrations, CI/CD, monitoring, testing, and performance tuning while collaborating with architects and stakeholders.
We are seeking an experienced Databricks developer with
strong expertise in Azure Databricks, Apache Spark, and modern data engineering
practices. The ideal candidate will be responsible for designing, developing,
and optimising scalable data pipelines, data lakehouse solutions, and
cloud-based data platforms. The role requires hands-on experience with Spark
processing, Delta Lake, Unity Catalogue, data modelling, and enterprise-grade data
integration solutions.
Key
Responsibilities
Data
Engineering & Databricks Development
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks,
Apache Spark, and SQL.
- Build and optimize batch and streaming data pipelines using PySpark,
Spark Structured Streaming, and Auto Loader.
- Develop and support enterprise data lakehouse solutions using Delta
Lake and Databricks technologies.
- Implement data ingestion, transformation, cleansing, and
aggregation processes for large-scale datasets.
- Develop reusable frameworks and best practices for data engineering
solutions.
Data
Modeling & Performance Optimization
- Design and implement data models to support reporting, analytics,
and business requirements.
- Build and maintain Slowly Changing Dimensions (SCD Type 1 &
Type 2) for data warehousing solutions.
- Develop and optimize Change Data Capture (CDC) pipelines.
- Optimize Spark workloads through partitioning, clustering, caching,
and performance tuning techniques.
- Ensure efficient query performance and scalability across large
datasets.
Unity
Catalog & Data Governance
- Configure and manage Databricks Unity Catalog environments.
- Create and manage catalogs, schemas, tables, materialized views,
functions, and volumes.
- Implement enterprise data governance, security, access control, and
compliance standards.
- Support metadata management and data lineage initiatives across the
data platform.
Cloud &
Integration
- Develop cloud-native data solutions on Microsoft Azure and
related cloud services.
- Integrate data from multiple internal and external data sources.
- Implement Lakehouse Federation and foreign catalogs to
access external data platforms.
- Collaborate with architects and stakeholders to design scalable
cloud data solutions.
DevOps
& Operational Excellence
- Support CI/CD implementation and automated deployment processes.
- Participate in code reviews, testing, and release activities.
- Monitor and troubleshoot data pipeline failures and performance
issues.
- Ensure adherence to development standards, security policies, and
operational best practices.
Required
Qualifications
- Strong hands-on experience with Databricks and Apache
Spark (PySpark and/or Scala).
- Extensive experience with SQL and complex data
transformation techniques.
- Experience in ETL/ELT development and enterprise data pipeline
implementation.
- Strong experience with Microsoft Azure and cloud-based data
platforms.
- Hands-on experience with Azure Databricks and Delta Lake.
- Experience building batch processing pipelines using Auto Loader and real-time pipelines using Spark Structured Streaming.
- Strong understanding of data warehousing concepts and dimensional
modeling.
- Experience implementing SCD Type 1, SCD Type 2, and CDC processes.
- Strong knowledge of Spark performance tuning, partitioning, and
optimization techniques.
- Experience with CI/CD pipelines and DevOps practices.
- Strong analytical, troubleshooting, and problem-solving skills.
Kumaran Systems Toronto, Ontario, CAN Office
703 Evans Avenue, Suite 605 , Toronto, ON , Canada, M9C 5E9
Similar Jobs
Fintech • Payments • Financial Services
Develop and maintain enterprise-scale full-stack applications for CIBC’s Regulatory Technology Client Data Services team. Responsibilities include designing technical infrastructure, building .NET and database solutions, integrating backend systems, developing technology roadmaps, leading complex projects, conducting code reviews, and mentoring peers. The role uses C#, SQL Server, Azure Databricks, React, microservices, REST APIs, and test-driven development in a hybrid Toronto environment.
Top Skills:
.Net.Net CoreAsp.NetC#DatabricksElasticsearchGraphQLJavaScriptJSONMicroservicesAzureMvcPowershellReactRestful ApiSQL ServerTest Driven DevelopmentUnit TestingXML
Fintech • Payments • Financial Services
Design, develop, and optimize Azure SQL databases, data warehouses, ETL pipelines, and lakehouse solutions for datasets exceeding 5TB. Write complex SQL, stored procedures, functions, and triggers; perform performance tuning and troubleshooting; and develop data models and cloud infrastructure solutions. Lead technical design, roadmaps, code reviews, and complex backend projects while mentoring developers. Work with Azure Data Factory, Databricks, PySpark, Delta tables, and Agentic AI tools.
Top Skills:
Agentic AiAzure Data FactoryAzure DatabricksAzure Sql DatabaseCopilotDatabricks Unity CatalogDelta LakeLakehouse ArchitectureAzurePysparkSQL
Information Technology
Own and evolve a production Azure platform supporting global, compliant cloud-native applications. Responsibilities include Bicep-based infrastructure, AKS operations, identity and access management, networking, compliance audits, multi-region resilience, observability, security, cost optimization, incident response, and engineering consultation. The role also develops reusable platform modules and templates, leads technical reviews and audit responses, mentors engineers, and participates in on-call and disaster recovery activities.
Top Skills:
AksApp ServiceApplication GatewayApplication InsightsArgo CdArmAzure AdAzure CliAzure DevopsAzure Front DoorAzure Key VaultAzure OpenaiAzure PolicyAzure RbacBicepC#Container InsightsEntra IdFluxGithub ActionsGitopsGoKqlKubernetesLog AnalyticsAzureMicrosoft Defender For CloudMicrosoft Defender For ContainersNsgsPimPowershellPrivate DnsPrivate EndpointsPythonTerraformVnetsWaf
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
