Staff Data Engineer

See yourself at Twilio

Join the team as Twilio’s next Staff Data Engineer - Profiles.

Who we are & why we’re hiring

Twilio powers real-time business communications and data solutions that help companies and developers worldwide build better applications and customer experiences.

Although we're headquartered in San Francisco, we have presence throughout South America, Europe, Asia and Australia. We're on a journey to becoming a global company that actively opposes racism and all forms of oppression and bias. At Twilio, we support diversity, equity & inclusion wherever we do business.

About the job

This position is needed to create high-quality, modern solutions to complex engineering problems and help define the future of Modern Data Warehouse and Customer Data Platforms. Our mission is to help businesses unlock customer insights and make better decisions faster and cheaper by creating a suite of industry-leading storage and compute services inside the Segment customer data platform. 

We are seeking a skilled and experienced Staff Software Engineer to join our data engineering team. You will be responsible for designing and building a suite of storage and compute building blocks that form the foundation of a strong Customer Data Platform capable of processing billions of events per hour and drives the best digital interactions Twilio can enable our customers to deliver. In this role, you will be responsible for building and maintaining our data  lake house architecture, enabling seamless data integration and analytics. Your expertise in Spark and Scala will be essential in developing scalable solutions that meet the demands of our growing data landscape.

Responsibilities

In this role, you’ll:

• Design and develop robust, scalable data pipelines using Databricks, Apache Spark, and Delta Lake
• Collaborate with cross-functional teams to gather requirements and design data solutions that meet business needs.
• Optimize performance and scalability of data pipelines, ensuring high availability and reliability.
• Ensure data quality, consistency, and security across the data platforms
• Monitor and troubleshoot data workflows and systems, providing timely resolutions to issues.
• Mentor and guide junior engineers, promoting best practices in software development and data engineering.
• Stay current with industry trends and technologies to continuously improve our data architecture.
• Build globally available and highly scalable distributed systems
• Participate in an on-call rotation to support our business-critical infrastructure

Qualifications 

Not all applicants will have skills that match a job description exactly. Twilio values diverse experiences in other industries, and we encourage everyone who meets the required qualifications to apply. While having “desired” qualifications make for a strong candidate, we encourage applicants with alternative experiences to also apply. If your career is just starting or hasn't followed a traditional path, don't let that stop you from considering Twilio. We are always looking for people who will bring something new to the table!

Required:

• 7+ years of experience in software engineering, with a focus on data engineering and lake house architecture.
• Strong proficiency in Spark and Scala; experience with Databricks technologies is highly desirable.
• Strong proficiency with cloud platforms (e.g., AWS, GCP) and data storage solutions (e.g., Delta Lake, Iceberg).
• Solid understanding of ETL processes and data pipeline architecture.
• Experience with SQL and data modeling; knowledge of NoSQL databases is a plus.
• Fundamental understanding and experience with asynchronous architectures, their benefits and challenges including various broker implementations (Kafka, SQS, MQ)
• Experience operating large-scale, distributed systems on top of cloud infrastructure such as Amazon Web Services (AWS) or Google Compute Platform (GCP)
• Have hands-on experience with container orchestration frameworks (e.g. Kubernetes, EKS, ECS)
• Experience shipping services (products) following CI/CD development paradigm.
• Deep understanding of architectural patterns of high-scale web application (e.g., well-designed APIs, high volume data pipelines, efficient algorithms).
• Have led teams, large projects, or owned and built an important, complex system end to end.
• Excellent written and verbal communication skills.

Location

 This position offers the flexibility of remote work, with the requirement that the candidate is based in one of the following locations: Alberta, British Columbia, or Ontario, Canada.

Travel

We prioritize connection and opportunities to build relationships with our customers and each other. For this role, you may be required to travel occasionally to participate in project or team in-person meetings.

What We Offer

There are many benefits to working at Twilio, including, in addition to competitive pay, things like generous time-off, ample parental and wellness leave, healthcare, a retirement savings program, and much more. Offerings vary by location.

Twilio thinks big. Do you?

We like to solve problems, take initiative, pitch in when needed, and are always up for trying new things. That's why we seek out colleagues who embody our values — something we call Twilio Magic. Additionally, we empower employees to build positive change in their communities by supporting their volunteering and donation efforts.

So, if you're ready to unleash your full potential, do your best work, and be the best version of yourself, apply now!

If this role isn't what you're looking for, please consider other open positions.

Twilio is proud to be an equal opportunity employer. Twilio is proud to be an Equal Employment Opportunity and Affirmative Action employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or other applicable legally protected characteristics. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law. Additionally, Twilio participates in the E-Verify program in certain locations, as required by law.

Twilio is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, please contact us at [email protected].