Build and maintain batch and streaming ETL pipelines using Databricks, Apache Spark, Azure, and Delta Lake. Develop CDC and SCD1/SCD2 solutions, data models, federated lakehouse connections, and governed Unity Catalog environments. Configure catalogs, schemas, tables, views, functions, and volumes while optimizing Spark performance through partitioning and liquid clustering. Apply data warehousing, security, CI/CD, and DevOps practices to deliver reliable cloud data platforms.
Required Skills.
• Strong hands-on experience with Data-bricks and Apache Spark (PySpark/Scala).
• Experience in SQL and data transformation techniques.
• Knowledge of ETL tools and data pipeline development.
• Experience working with cloud platforms (Azure/AWS/GCP)
• Strong Azure cloud background
• Understanding of data warehousing concepts.
• Strong problem-solving and analytical skills.
• Hands-on experience with Azure Data bricks or Delta Lake, in building ETL pipelines : batch (autoloader) and Spark structured streaming
• Knowledge of data modelling and performance tuning in Spark.
• Exposure to CI/CD pipelines and DevOps practices.
• Familiarity with data governance and security practices.
• Strong hands on working experience of Unity catalog:
• Hands on exposure to Creating end to end environments : creating catalogs, schemas, tables . materialized views, functions, volumes
• Experience in building SCD 1 and SCD2 (slowly changing dimensions ) on dimension tables
• Experience in building CDC (change data capture pipelines)
• Strong hands on experience with Lakehouse federation , creating foreign catalogs to get data from external sources
• Strong understanding of databricks partitioning , Liquid clustering
Kumaran Systems Toronto, Ontario, CAN Office
703 Evans Avenue, Suite 605 , Toronto, ON , Canada, M9C 5E9
Similar Jobs
Artificial Intelligence • Marketing Tech • Software
Build and operate high-volume event streaming pipelines and analytical databases. Write production Elixir/Python services, integrate legacy pipelines, enforce event schemas and data quality, and provide monitoring, alerting, and SLA-driven observability for event data products.
Top Skills:
Apache FlinkAWSCloudFormationColumnar/Olap DatabasesDatabricksElixirGCPKafkaKinesisPulsarPythonSQLTerraform
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Design, build, and operate production AI systems and marketing data infrastructure. Develop retrieval pipelines, agent orchestration, CDP integrations, ETL/ELT pipelines, and analytics. Automate workflows with AI, ensure data quality and reliability, partner with stakeholders, ship Python and SQL code, mentor engineers, and drive requirements to production.
Top Skills:
Agent OrchestrationBigQueryClaude CodeCursorCustomer Data Platform (Cdp)Data WarehouseDatabricksDbtEtl/EltHightouchLlmsMarketing AutomationPythonSalesforceSegmentSnowflakeSQLWeb Analytics
Fintech • Machine Learning • Payments • Software • Financial Services
Lead the architecture and development of scalable data platforms and distributed processing systems. Establish enterprise data governance, quality, privacy, compliance, schema management, and lifecycle standards. Drive resilient, automated, and observable data pipeline strategies; evaluate technologies; review complex architectures; collaborate with business and technology stakeholders; and mentor data engineers in stream processing, data modeling, and MLOps.
Top Skills:
AWSContainersConversational AiDistributed Processing SystemsGCPMachine LearningMicroservicesAzureMlopsRestful ApisStream Processing
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
