Senior Software Engineer -Kafka / Event Streaming

Description

Shape the Future of AI Infrastructure with VAST Data

This is a rare opportunity to join one of the fastest-growing infrastructure companies in tech history, right at the epicenter of the AI revolution.

Recognized by Forbes as "the future of the market," VAST Data is building the foundational enterprise software powering the AI era. We engineer platforms that capture, catalog, refine, and protect massive datasets across data centers, edge, and cloud, making real-time analytics, AI training, and inference simpler and faster than ever before.

Our explosive growth is fueled by relentless engineering innovation, a customer-first mindset, and a team of fearless VASTronauts who thrive on solving computing's hardest problems. Join us at this pivotal moment in technology history and make a lasting impact on how the world processes data.

We are seeking a talented and experienced Software Engineer to design, build, and optimize high-performance distributed systems, core data engines, and backend infrastructure. This role requires deep system-level architecture understanding, the ability to handle large-scale clusters processing petabytes of data, and mastery of modern C/C++ and Linux environment internals.

What You'll Do:

• Design and develop core components of a next-generation distributed event-streaming and messaging platform.
• Implement and maintain a high-scale, high-performance Kafka-protocol-compatible broker.
• Build scalable and resilient distributed services handling petabytes of data across large clusters.
• Design solutions around correctness semantics (ordering guarantees, transactions, exactly-once / idempotence, repartitioning), load balancing, replication, fault tolerance, and scalability
• Investigate and resolve complex production issues across the full produce-to-consume path - from the Kafka client that produced it, through the broker and all the layers in between, all the way to the consumer.
• Collaborate closely with architects, researchers, and engineering teams to deliver high-performance data infrastructure powering AI, analytics, and enterprise workloads.

Requirements

Requirements:

• Strong hands-on experience in C/C++ software development, including system design, implementation, debugging, performance optimization, and integration of large-scale production systems.
• B.Sc. or M.Sc. in Computer Science, Software Engineering, Electrical Engineering, or equivalent practical experience.
• Deep understanding of Linux internals, including processes, threads, memory management, synchronization primitives, I/O subsystems, and performance troubleshooting.
• Experience developing distributed systems with a focus on scalability, reliability, fault tolerance, concurrency, and high availability.
• Solid understanding of networking fundamentals, including TCP/IP, routing, transport protocols, distributed communication patterns, and OSI networking layers.
• Experience designing and implementing high-performance backend services operating in multi-node environments.
• Strong debugging and root-cause analysis skills across complex production environments.
• Experience with message brokers or event-streaming systems - advantage.
• Apache Kafka internals or Kafka wire protocol experience - advantage.