Senior Data Engineer responsible for designing and validating data solutions, consolidating disparate data sources, documenting functional specifications, reviewing data models, supporting production deployments, investigating data quality issues, and collaborating with stakeholders and technology leads to ensure data completeness and lineage.
Responsibilities
- Big Data - Spark, Hive, Java, CDP, 6 Years + with 3 Years of Dev experience on Big Data
- Analyze data requirements and identify disparate data sources required for consolidation and distribution.
- Document functional specifications and coordinate delivery of the same with technology team.
- Review logical and conceptual data models in alignment with business requirements.
- Work with the stakeholders to understand and gather requirements and produce business specifications
- Validate solution implementations and ensure they meet business and functional requirements.
- Provide production deployment support and investigate data quality issues.
- Work with various technology leads to ensure the gaps in the data completeness or accuracy are bridged.
Qualifications
- Subject matter expertise in financial industry - wholesale loans /lending business OR Capital Markets or Finance or Risk Reporting
- Strong hands on experience with database and SQL is required.
- Excellent documentation, analytical skills to produce process flow diagrams, business modelling, and functional design.
- Proficiency in MS Office (Word, Excel, Visio, PowerPoint) with extensive experience using Excel for data analysis.
- Experience with Data tracing/ Lineage efforts
- Knowledge of logical and physical data model
Similar Jobs
Agency • Information Technology
Senior Java Big Data Developer with 8+ years building applications on Cloudera/Hadoop ecosystems using Spark and MapReduce. Responsible for designing performant Java (1.8+) solutions, data analysis of large datasets, unit testing (JUnit), and collaborating with/managing global technology teams. Preferred skills include Unix/Python scripting and familiarity with Hive, Impala, YARN, and Kafka.
Top Skills:
SparkClouderaHadoopHiveImpalaJava 1.8JunitKafkaMapreduceNoSQLPythonUnix ShellYarn
Agency • Information Technology
Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Top Skills:
SparkCi/CdCloudera (Cdh)FlumeGitGitlabHbaseHdfsHiveImpalaJenkinsKafkaNifiOoziePostgresScalaSpark SqlSqoop
Agency • Information Technology
Design, develop, and maintain data engineering solutions across the SDLC using Python (primary) and Scala (secondary). Build distributed data pipelines on Hadoop/Spark ecosystems, extend data processing platforms (Spark SQL, Hive, Starburst), manage CI/CD, and deploy on Kubernetes/OpenShift. Collaborate with stakeholders in Agile teams and ensure maintainable, reusable code.
Top Skills:
AirflowAnsibleBitbucketHadoopHarnessHiveImpalaJenkinsJIRAJupyterhubKerberosKubernetesNumpyOpenshift4PandasPythonScalaSparkSpark SqlStarburstTektonUdeploy
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
