Design, develop, and maintain data engineering solutions across the SDLC using Python (primary) and Scala (secondary). Build distributed data pipelines on Hadoop/Spark ecosystems, extend data processing platforms (Spark SQL, Hive, Starburst), manage CI/CD, and deploy on Kubernetes/OpenShift. Collaborate with stakeholders in Agile teams and ensure maintainable, reusable code.
Qualifications:
- Strong communication skills
- Experience of Agile development and scrums
- Banking and securities domain knowledge would be an added advantage
Skills Required:
- Strong experience working across the entire SDLC lifecycle
- Programming experience in one or more application or systems languages, Python - Primary, Scala - Secondary (basic knowledge)
- Strong experience working with Python concepts and libraries such as Jupyterhub, Airflow, Pandas, NumPy etc.
- Good experience with classes based OOP and design patterns
- Distributed Systems Design Experience - including understanding of distributed systems concepts and principles
- Knowledge and understanding of Kerberos and authentication
- Hadoop Ecosystem of Tools (Spark, Hive, Impala, etc).
- Experience extending and implementing core functionality and libraries in data processing platforms (Spark / Spark SQL, Hive, Starburst, etc)
- Strong experience working with the CI/CD pipeline and tools like Jenkins , Harness/Tekton, Udeploy/Ansible,Bitbucket, Jira
- Strong experience in cloud platforms like Kubernetes, OpenShift4
- Ability to deal with multiple stakeholders and follow through on open issues.
- A commitment to writing understandable, maintainable, and reusable software.
- Willingness to learn new languages and methodologies.
- Experience working with business partners and engineers to gather, understand, and bridge definitions and requirements.
- An innate desire to deliver and a strong sense of accountability for your work.
Education:
- Bachelor’s degree/University degree or equivalent experience
Similar Jobs
Agency • Information Technology
Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Top Skills:
SparkCi/CdCloudera (Cdh)FlumeGitGitlabHbaseHdfsHiveImpalaJenkinsKafkaNifiOoziePostgresScalaSpark SqlSqoop
eCommerce • Fashion • Retail • Sales • Wearables • Design
Lead regional retail training and customer experience initiatives, deliver and implement sales, service, and clienteling programs, monitor KPIs, coach store teams and managers, support onboarding and digital tool adoption (Coach Journey, Client Compass), and drive consistent brand service standards across the market.
Top Skills:
Client CompassCoach JourneyExcelMS OfficePowerPointWord
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Build and operate ingestion, reconciliation, and reporting systems that reconcile card-network and partner settlement files against internal transactions. Deliver end-to-end features for traceability, accounting journals, tax and regulatory reporting, and compliance tooling while ensuring reliability, scalability, and data privacy.
Top Skills:
Ai ToolsAirflowAWSBigQueryCi/CdDelta LakeGoHadoopIso-8583JavaKafkaKubernetesObservabilityPysparkPythonSnowflakeSparkSQLTemporalTerraform
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
