Design, orchestrate, and optimize large-scale LLM pre-training across 1,000+ GPUs. Implement 3D parallelism, manage GPU clusters (SLURM/Kubernetes), optimize InfiniBand/RDMA networking and memory, and automate checkpointing and failure recovery for long training runs.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
Cloud • Security • Software • Cybersecurity • Automation
Designs, operates, and improves production Kubernetes platforms across cloud environments. Manages ArgoCD GitOps workflows, Terraform infrastructure, CI/CD integrations, platform reliability, observability, security controls, and developer enablement. Collaborates with application teams to onboard services, create self-service tooling and documentation, respond to incidents, and improve platform availability and performance.
Top Skills:
ArgocdAWSAzureBashCiliumCrossplaneDnsExternal Secrets OperatorGCPGithub ActionsGitlab CiGitopsGoGrafanaHelmIngressIstioKubernetesKyvernoLinkerdLoad BalancingLokiOpa/GatekeeperOpentelemetryPrometheusPythonSealed SecretsService MeshTektonTerraformVault
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Build and operate end-to-end full-stack features for Loom billing, subscriptions, integrations, and platform migrations. Develop responsive React interfaces, scalable backend APIs and microservices, payment and identity integrations, and reliable cloud-based systems. Responsibilities include observability, monitoring, testing, code reviews, documentation, security, performance optimization, and participation in Agile ceremonies and technical design reviews.
Top Skills:
AWSBitbucketCi/CdConfluenceCypressDockerGitJavaJavaScriptJestJIRAMicroservicesMochaMySQLNode.jsPostgresPythonReactRest ApisTypescript
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Own and grow the Toronto sales territory through extensive in-person outreach, business visits, live demos, partnerships, referrals, and full-cycle deal closing. Build pipeline, sell Square’s software and hardware to restaurants, retailers, and service businesses, maintain Salesforce activity and forecasts, support onboarding, and consistently exceed quota. The role requires field sales experience, reliable transportation, and residence in the assigned market.
Top Skills:
Payment Processing TechnologySalesforce
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
