Splunk Staff Software Engineer - Performance Optimization & Innovation (PerfOpt)

Cisco is a technology company innovating in data and infrastructure to provide security, visibility, and insights across digital environments. The Performance Optimization and Innovation (PerfOpt) team at Splunk, part of Cisco, focuses on deep performance work for petabyte-scale data in a large distributed architecture, improving customer experience and setting performance standards.

• Advanced production experience with C++, including memory management, allocators, move semantics, cache-aware structures, and performance trade-offs.
• Deep performance engineering expertise in CPU/memory profiling, flamegraph analysis (perf, eBPF, VTune), lock contention resolution, latency optimization in distributed architectures, benchmarking, and performance modelling.
• Strong skills in architecture and system design for performance, scalability, and operability.
• Experience with Docker and Kubernetes performance characterization.
• Proficient in Python for tooling, benchmark harnesses, telemetry pipelines, and performance data analysis.
• Knowledge of Linux performance internals including scheduler, memory subsystem, page cache, filesystems, block I/O, and network stack.
• Experience with Git and CI systems like GitLab CI.
• Leadership skills including informing technical decisions, distilling complex problems into comparable data, clear communication, AI tool usage for delivery speed, and mentoring engineers.

Preferred Qualifications:

• Experience with search or storage engines and large-scale data platforms.
• Familiarity with Splunk internals.
• Hardware-level experience with SIMD, NUMA, cache/TLB behavior, PGO/LTO, compiler optimization.
• Knowledge of Go, Rust, or another systems language alongside C++.
• Experience with cloud platforms such as AWS, Azure, or GCP.
• Familiarity with infrastructure tools like Terraform, Puppet, or Ansible.

• Lead performance work from hypothesis through instrumentation and design to measured customer impact, coordinating across teams as needed.
• Design petabyte-scale components for memory, concurrency, I/O, and tail latency, guiding architecture for scalability and reliability.
• Identify and prevent performance issues using profiling, flame graphs, lock contention analysis, benchmarks, and CI regression gates.
• Instrument production with telemetry, eBPF, and continuous profiling to measure performance in real workloads.
• Compare performance options, quantify costs and risks, and present evidence-based recommendations.
• Develop reusable AI workflows to shorten performance analysis.
• Plan performance work, estimate effort, and lead postmortems to identify root causes and prevent regressions.
• Mentor peers and less experienced engineers on the team.