Build and maintain batch and streaming ETL pipelines using Databricks, Apache Spark, Azure, and Delta Lake. Develop CDC and SCD1/SCD2 solutions, data models, federated lakehouse connections, and governed Unity Catalog environments. Configure catalogs, schemas, tables, views, functions, and volumes while optimizing Spark performance through partitioning and liquid clustering. Apply data warehousing, security, CI/CD, and DevOps practices to deliver reliable cloud data platforms.
Required Skills.
• Strong hands-on experience with Data-bricks and Apache Spark (PySpark/Scala).
• Experience in SQL and data transformation techniques.
• Knowledge of ETL tools and data pipeline development.
• Experience working with cloud platforms (Azure/AWS/GCP)
• Strong Azure cloud background
• Understanding of data warehousing concepts.
• Strong problem-solving and analytical skills.
• Hands-on experience with Azure Data bricks or Delta Lake, in building ETL pipelines : batch (autoloader) and Spark structured streaming
• Knowledge of data modelling and performance tuning in Spark.
• Exposure to CI/CD pipelines and DevOps practices.
• Familiarity with data governance and security practices.
• Strong hands on working experience of Unity catalog:
• Hands on exposure to Creating end to end environments : creating catalogs, schemas, tables . materialized views, functions, volumes
• Experience in building SCD 1 and SCD2 (slowly changing dimensions ) on dimension tables
• Experience in building CDC (change data capture pipelines)
• Strong hands on experience with Lakehouse federation , creating foreign catalogs to get data from external sources
• Strong understanding of databricks partitioning , Liquid clustering
Kumaran Systems Toronto, Ontario, CAN Office
703 Evans Avenue, Suite 605 , Toronto, ON , Canada, M9C 5E9
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Build Python-based frameworks to validate, reconcile, cleanse, and transform large datasets. Use Pandas and NumPy to identify data quality issues, discrepancies, duplicates, and anomalies. Automate validation workflows, investigate root causes, produce data quality reports, optimize processing, and collaborate with business, analytics, and engineering teams on data migration, onboarding, and integration initiatives.
Top Skills:
AWSAzureAzure Data FactoryCi/CdDatabricksGCPGitNumpyPandasPower BIPysparkPythonSQL
AdTech • Digital Media • eCommerce • Marketing Tech
Build and operate Databricks-based data products and pipelines that create, model, and deliver syndicated and custom consumer audience segments. Develop tables, views, jobs, workflows, and real-time processing solutions using Python, SQL, Spark, and Delta Lake. Support marketplace delivery, reporting, observability, metadata, testing, CI/CD, privacy compliance, and AI-assisted development. Partner with data science and commercial teams to translate business requirements into scalable engineering solutions, while promoting architecture and engineering best practices.
Top Skills:
SparkCi/CdClaude CodeDatabricksDatabricks Jobs & WorkflowsDatabricks NotebooksDelta LakeGithub CopilotLakeflow Declarative PipelinesLiverampLotamePythonSpark Structured StreamingSQLTruaudienceUnity Catalog
.jpg)
AdTech • Cloud • Marketing Tech • Productivity • Software • Analytics • Automation
Build and ship production AI systems across retrieval, knowledge graphs, inference, agentic workflows, and data platforms. Develop customer-context retrieval, scored signals, workflow automation, evaluation, and observability capabilities. Work hands-on with Python, SQL, dbt, lakehouse data, cloud infrastructure, model selection, and inference-cost optimization. Establish reusable engineering patterns and mentor teams through high-quality code and system design.
Top Skills:
Amazon S3Apache IcebergAWSClaudeCursorDbtDistributed Query EnginesDockerEmbeddingsGithub CopilotHybrid RetrievalKnowledge GraphsLanggraphMcpOpentelemetryPythonSQLTemporalVector Search
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
