Data Engineer

The Ontario Securities Commission (OSC) is the statutory body responsible for regulating Ontario’s capital markets in accordance with the mandate established in the provincial Securities Act and the Commodity Futures Act. The mandate of the OSC is to provide protection to investors from unfair, improper or fraudulent practices, to foster fair, efficient and competitive capital markets and confidence in the capital markets, to foster capital formation, and to contribute to the stability of the financial system and the reduction of systemic risk. This mandate is performed through policy, operational, and enforcement activities. The OSC also contributes to national and global securities regulation development.

We offer a diverse, fair, and flexible work environment and take pride in our challenging and rewarding work.

Summary

Under the guidance of the Technical Manager, Data & Analytics, this position will be responsible for partnering with internal stakeholders to understand and assess current and future business needs to inform design and development of data solutions, build  data pipelines, and perform data management and optimization. The candidate will have good knowledge of capital markets data sets. This role will be required to ensure the capturing and translation of complex business logic into technical data deliverables. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up and support business units, data analysts and data scientists on data initiatives and ensure optimal data delivery. They will need to work to continuously improve and redeploy data solutions to meet evolving regulatory needs.

Key Duties and Responsibilities

• Partners with internal stakeholders across the organization to understand current and future business needs, to translate and develop overall strategy for data platform architecture. As familiarity on projects increases, work to proactively address needs with the business, and validate assumptions with business partners
• Independently provides recommendations to internal stakeholders on data solutions and assesses impact and risks for branch and broader organization implications, in collaboration with Business Data Architect and the Data Management and Reporting Lead
• Partner with business units to translate and assess business requirements into data ingestion and standardization scripts. Determine appropriate solution, assess impact and risk of solutions, develop and maintains optimal yet scalable data pipelines that supports efficient extraction, transformation, and loading of data from a wide variety of high volume and complex data sources
• Work with Security Specialists to ensure compliance with security and data management requirements
• As subject matter expert, provide advice in the build of analytics tools that utilize the data pipeline to provide actionable insights and inform of risks and impact to ensure successful implementation.
• As subject matter expert, advise analytics and business teams to develop data models and constantly strive for excellence in our data capabilities
• Recommend and implement processes and systems to continuously monitor data quality to ensure that production data is always accurate and available for authorized stakeholders
• Lead data analysis to troubleshoot data related issues and provide expertise in the resolution of data issues
• As subject matter expert in data engineering, partner with Security, Architecture and Technical Services to align solution designs with requirements
• Act as subject matter expert in agile pods (multi-disciplinary teams) to complete complex projects, representing the data engineering function.
• Ensure quality of data is upheld throughout the transformation process and meets the desired state within the target data architecture
• Contribute towards data integrations and data quality framework to ensure that OSC data engineering practices are following standards set by our Data Governance teams, and limit the access and processing of data as per regulations and internal controls
• In partnership with business, Digital Solutions and IT branches increase data accessibility and fostering a data-driven decision-making culture across the organization
• As the subject matter expert, provides regular guidance and advice in management and use of data to business units in the Spokes and across the Digital Solutions branch.
• Collaborate with the data strategy chapter as data engineering expert in - (a) identifying and acquiring data from internal and external data sources, (b) Prioritizing business and information needs
• Support interpretation and analysis of data and trends using programming or statistical techniques and generate insights and reports on on-going basis

Qualifications

• 10+ years of experience working with data in roles such as Data Engineer, ETL Engineer, or similar
• Graduate degree in Computer Science, Statistics, Informatics, Information Systems, or another quantitative field

Cloud & Data Platforms

• Strong hands-on experience with Databricks (mandatory)
• Experience with Azure Data Factory for orchestration and data pipelines
• Experience with Azure Synapse Analytics for data warehousing and analytics
• Strong understanding of Azure cloud-based data engineering ecosystems

Data Engineering & Architecture

• Experience building and optimizing large-scale data pipelines and datasets
• Experience designing and managing batch and streaming data pipelines
• Experience managing metadata, dependencies, workload orchestration, and data structures
• Proven ability to work with large, disconnected datasets and extract meaningful business value

Data Ingestion & Processing

• Strong hands-on experience with Databricks Auto Loader for scalable incremental ingestion
• Experience building production-grade pipelines using Delta Live Tables (DLT) / declarative pipelines
• Experience with schema evolution, data ingestion frameworks, and pipeline optimization

Data Governance, Contracts & Frameworks

• Strong experience designing and implementing YAML-based data contracts
• Hands-on experience enforcing schema consistency, validation rules, and data quality standards
• Experience building metadata-driven frameworks where pipeline behavior is configuration-driven (no hardcoding)
• Strong experience with data governance, lineage, and validation frameworks

Programming & Data Skills

• Advanced working knowledge of SQL and relational databases, including query authoring and optimization
• Strong programming skills in Python and PySpark (mandatory)
• Exposure to R, Java, and scripting languages such as PowerShell
• Strong analytical skills working with both structured and unstructured datasets
• Strong ability to collect, organize, analyze, and interpret large volumes of data with high accuracy and attention to detail

Reusable Engineering & Packaging

• Strong experience building and deploying reusable Python packages (Wheel files) in Databricks environments
• Experience building modular frameworks for:
  • ETL utilities
  • Data validation
  • Logging and auditing
  • Shared reusable pipeline components

Visualization & Reporting

• Strong experience with Databricks Dashboards (mandatory)
• Experience with Power BI for enterprise reporting and visualization

CI/CD, DevOps & Testing

• Strong experience with Azure DevOps (CI/CD pipelines mandatory)
• Experience with Databricks Asset Bundles (DAB) for deployment automation
• Experience validating data quality, schema evolution, and regression testing
• Experience supporting production releases and workflow automation
• Exposure to unit test automation for data pipelines

Analytical & Problem-Solving Skills

• Extensive experience performing root cause analysis (RCA) on data and process issues
• Experience designing processes supporting:
  • Data transformation
  • Metadata management
  • Dependency management
  • Workload orchestration
• Strong ability to identify opportunities for process and data improvement

Soft Skills & Mindset

• Experience working with cross-functional teams in dynamic environments
• Strong curiosity and interest in emerging technologies
• Ability to adapt to change and support teams through transformation journeys
• Strong communication, collaboration, and stakeholder engagement skills

Grow your career and make a difference working at the OSC.

* OSC Employees: please apply in Workday using the Browse Jobs feature within your Jobs Hub *

We thank all applicants for their interest in the Ontario Securities Commission. We will contact those selected for an interview.

The OSC is committed to diversity and providing an inclusive workplace and providing accommodation in accordance with the Accessibility for Ontarians with Disabilities Act and the Human Rights Code. It is our priority to ensure employment opportunities are visible and barrier-free to all under-represented groups including but not limited to, Indigenous, Black and racialized groups, people with disabilities, women and people from the 2SLGBTQI+ community, to achieve an employee demographic profile reflective of the demographic profile of Ontarians.

The OSC is a proud partner with the following organizations: Ascend Canada, BlackNorth Initiative, Canadian Centre for Diversity and Inclusion, and Pride at Work Canada

If you require an accommodation during the recruitment process, please let us know by contacting our confidential inbox [email protected].  
 

Visit Accessibility at the OSC to review the OSC’s policies on accessibility and accommodation in the workplace.