The Data Engineer designs and maintains enterprise data warehouse systems, develops data pipelines, ensures efficient data flow and storage, and collaborates with stakeholders on data-related solutions.
Purpose of Job:
The Data Engineer will have the responsibility for the design and maintenance of enterprise data warehouse for various projects, as well as providing on-going support for activities impacting all enterprise databases. The ideal candidate will have hands-on experience with Azure Data Factory, Azure Data Lake, Azure Synapse, Azure Function App, Logic App, REST APIs, mapping dataflow, and Event Hub. As a key member of our team, you will collaborate with various stakeholders to ensure efficient data flow, storage, and integration, while employing best practices in data engineering.
Main Activities:
- 1. Developing of Data Pipelines (50%)
- 2. Requirements gathering and Technical Documentation (10%)
- 3. Data Strategy and Planning (10%)
- 4. Support and Maintenance of Enterprise Data Warehouse and BAU Tasks (30%)
· Design, develop, and maintain scalable Synapse pipelines for data ingestion, processing, and transformation in Azure data platform.
· Develop and optimize T-SQL and Python-based data processing scripts and integrate them with Synapse pipelines for ETL tasks and data transformations.
· Design and develop mapping dataflow solutions for complex data transformations.
· Develop and maintain data storage solutions, ensuring efficient data retrieval and storage on Azure Data Lake and databases.
· Implement and maintain API integrations, including REST APIs, for seamless data flow between systems.
· Design and test the end-to-end process for data ingestion, data transformation, data cleansing, data delivery, data quality and data timeliness.
· Implement and ensure data security and privacy measures, complying with industry standards and regulations.
· Monitor and optimize the performance of Azure data services, Azure Synapse pipelines, Function Apps, and Logic Apps.
· Improve the scalability, efficiency, and cost-effectiveness of data pipelines.
· Participate in Joint Application Development sessions with business units to gather and understand their data and reporting requirements.
· Create and maintain technical documentations for tools, development, and procedures.
· Provide inputs and assist in enablement of Cloud based data solutions.
· Develop and recommend innovative approaches to solve business and technical problems.
· Work and collaborate with the larger team to exchange knowledge, solutions, and practices to build a more consistent, robust approach to development.
· Continuously improve performance and proactively identify and resolve bottlenecks that will reduce time to build and deliver our products.
· Troubleshoot and resolve data-related issues in a timely manner for production and other environments (DEV,UAT,QA)
· Perform business as usual tasks (BAU) that assigned
Skills/Knowledge Requirements:
- Bachelor's degree or equivalent in the field of Computer Science or Engineering
- 3+ years of Proven experience as a Data Engineer, specifically working with Microsoft Azure technologies.
- Proficient in T-SQL, Pyspark, Mapping Dataflow and Python programming language
- Hands-on experience with Azure data services, including Azure Data Factory, Azure Data Lake, Azure Function App, Azure Logic App, Azure SQL Database, Azure Synapse Analytics, mapping dataflow, and Event Hub.
- Understanding of data warehousing concepts and best practices.
- Experience with Data modeling.
- Experience in working with structured, semi-structured and unstructured data.
- Excellent written and verbal communication skills
- Excellent multi-tasking and organizational skills
EQ Bank | Equitable Bank Toronto, Ontario, CAN Office
30 St Clair Avenue West, Suite 700, Toronto, Ontario, Canada, M4V 3A1
Similar Jobs
eCommerce • Fintech • Real Estate • Software • PropTech
The Senior Finance Data Engineer will lead finance data architecture, automate workflows, and build AI-driven insights systems to enhance financial operations and reporting.
Top Skills:
BigQueryClaudeGumloopLangchainLlamaindexPythonSnowflakeSQL
Real Estate • Financial Services
As a Data Developer, you will create and optimize data pipelines, collaborate with teams to meet business data needs, and apply advanced technical skills in data engineering and automation.
Top Skills:
AirflowSparkAWSAzureAzure Data FactoryCubejsDatabricksDelta LakeGCPPysparkPythonSQL
Information Technology • Software
As a Data Developer, you will create enterprise data solutions, build data pipelines, and collaborate with teams to meet business needs using intermediate to advanced technical skills.
Top Skills:
AIAirflowSparkAutomationAWSAzureAzure FunctionsCi/CdDatabricksDelta LakeGCPPythonSQLSsis
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
