Design, orchestrate, and optimize large-scale LLM pre-training across 1,000+ GPUs. Implement 3D parallelism, manage GPU clusters (SLURM/Kubernetes), optimize InfiniBand/RDMA networking and memory, and automate checkpointing and failure recovery for long training runs.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
Artificial Intelligence • Consumer Web • Digital Media • Information Technology • Social Impact • Software
Build end-to-end full-stack AI features on a Ruby on Rails backend and React frontend, design and run experiments (A/B tests and AI evaluations), implement AI infrastructure for reliable LLM inference, and iterate quickly to scale production-ready LLM-powered products.
Top Skills:
Ai AgentsLlmsMySQLPostgresReactRetrieval-Augmented Generation (Rag)Ruby On Rails
Gaming • Information Technology • Mobile • Software • Esports
Lead design and build high-quality combat arenas in-engine, define arena standards and combat pacing, iterate with Game Designers, prototype enemy/mechanic synergies, own level creation in production, and document best practices to raise overall game quality.
Top Skills:
Unreal Engine 4Unreal Engine 5
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Drive new SaaS license revenue through account and territory planning, prospect research, and field sales. Build C-suite relationships, orchestrate cross-functional account teams, act as trusted advisor on IT and AI integration, identify and involve specialists at the right time, and close deals while meeting sales targets.
Top Skills:
AIArmisSaaSServicenowVeza
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
