Design, develop, and maintain data engineering solutions across the SDLC using Python (primary) and Scala (secondary). Build distributed data pipelines on Hadoop/Spark ecosystems, extend data processing platforms (Spark SQL, Hive, Starburst), manage CI/CD, and deploy on Kubernetes/OpenShift. Collaborate with stakeholders in Agile teams and ensure maintainable, reusable code.
Qualifications:
- Strong communication skills
- Experience of Agile development and scrums
- Banking and securities domain knowledge would be an added advantage
Skills Required:
- Strong experience working across the entire SDLC lifecycle
- Programming experience in one or more application or systems languages, Python - Primary, Scala - Secondary (basic knowledge)
- Strong experience working with Python concepts and libraries such as Jupyterhub, Airflow, Pandas, NumPy etc.
- Good experience with classes based OOP and design patterns
- Distributed Systems Design Experience - including understanding of distributed systems concepts and principles
- Knowledge and understanding of Kerberos and authentication
- Hadoop Ecosystem of Tools (Spark, Hive, Impala, etc).
- Experience extending and implementing core functionality and libraries in data processing platforms (Spark / Spark SQL, Hive, Starburst, etc)
- Strong experience working with the CI/CD pipeline and tools like Jenkins , Harness/Tekton, Udeploy/Ansible,Bitbucket, Jira
- Strong experience in cloud platforms like Kubernetes, OpenShift4
- Ability to deal with multiple stakeholders and follow through on open issues.
- A commitment to writing understandable, maintainable, and reusable software.
- Willingness to learn new languages and methodologies.
- Experience working with business partners and engineers to gather, understand, and bridge definitions and requirements.
- An innate desire to deliver and a strong sense of accountability for your work.
Education:
- Bachelor’s degree/University degree or equivalent experience
Similar Jobs
Agency • Information Technology
Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Top Skills:
SparkCi/CdCloudera (Cdh)FlumeGitGitlabHbaseHdfsHiveImpalaJenkinsKafkaNifiOoziePostgresScalaSpark SqlSqoop
.png)
Healthtech • Social Impact • Telehealth
The Medical Claims Billing Specialist manages medical claims, resolves denials, and ensures timely payment for services, impacting financial health.
Top Skills:
Billing SystemsClearinghousesEhrsInsurance Portals
Fintech • Financial Services
Lead end-to-end data science initiatives: design, deploy, and monitor ML/statistical models for credit risk, pricing, collections and fraud. Collaborate cross-functionally to integrate models into production, define model development standards, mentor team members, and maintain scalable ML pipelines and monitoring systems to drive measurable business impact.
Top Skills:
ArizeAWSDatabricksGitMetaflowPythonSagemakerSnowflakeSQLTaktileTecton
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
