AI Inference / Systems Software Architect
Reports To: VP, Engineering
Company Overview:
Netlist, Inc. (OTCQB: NLST) is a leading provider of high-performance modular memory subsystems and next-generation storage-class memory solutions. We specialize in cutting-edge technologies that enable innovation in AI, cloud computing, and enterprise infrastructure. Our product portfolio includes DDR4/DDR5 DIMMs, NAND-based SSDs, and advanced interconnect technologies such as CXL and PCIe.
Join our world-class engineering team and be part of shaping the future of memory and storage architecture.
Position Summary:
This is a highly technical role for an individual who deeply understands AI Data Center and Edge AI Inference workloads, model execution, systems software, memory management, and hardware/software co-design.
The successful candidate will work closely with the Chief AI Systems Architect, AI Product Strategist, AI Hardware / Memory Systems Architect, and R&D engineering teams to understand how AI workloads execute on current infrastructure, identify software bottlenecks, and architect software that enables new hardware and memory technologies to deliver measurable customer value.
We are particularly interested in investigating opportunities involving AI workload, software stack, and inference runtime usage of the hardware to deliver the required performance and efficiency.
Key Responsibilities:
AI Inference Architecture
Develop a deep understanding of modern AI inference workloads and their system-level behavior
Analyze the execution characteristics of Large Language Models (LLMs), Generative AI, Agentic AI, Long-context inference, Mixture-of-Experts (MoE), Retrieval-Augmented Generation (RAG), Multimodal AI and Edge inference
Analyze the differences between prefill and decode workloads
Characterize the impact of batch size, sequence length, concurrency, model size, attention mechanisms, and memory access patterns
Identify whether workload bottlenecks are caused by compute, memory bandwidth, memory capacity, latency, interconnect, software overhead, or scheduling
AI Runtime and Systems Software
Architect software infrastructure that allows AI workloads to efficiently utilize heterogeneous memory and compute resources
Areas may include AI inference runtimes, model execution engines, scheduling, resource management, memory management, KV-cache management, data movement, prefetching, caching, model partitioning, load balancing, multi-device execution, distributed inference, accelerator orchestration and telemetry and performance monitoring
Evaluate and contribute to open-source and commercial AI software stacks such as PyTorch, CUDA, ROCm, Triton, vLLM and other emerging inference runtimes and compiler frameworks
AI Memory Management
Work closely with the AI Hardware / Memory Systems Architect on next-generation memory architectures to make the entire memory hierarchy appear as an efficient, application-aware resource rather than a collection of independent storage tiers
Explore how software can intelligently manage HBM, DDR, CXL memory, memory pools, disaggregated memory, NVMe, persistent storage, local versus remote memory, model weights, context state and KV cache
Develop policies and mechanisms for data placement, tiering, migration, caching, prefetching, eviction, compression, decompression, memory sharing, QoS, isolation and resource allocation
Hardware / Software Co-Design
Work closely with the hardware architects to ensure that architectural features and hardware capabilities like CXL memory, memory pooling, smart memory controllers, near-memory compute, PIM, inference engines, AI accelerators, compression engines and specialized cache architectures can be effectively exposed and utilized by the software stack
Collaborate on AI runtime architecture, memory management, KV-cache management, scheduling, data placement, prefetching, compression, workload orchestration, APIs and software interfaces and hardware telemetry and monitoring
Ensure that hardware innovations translate into measurable application-level benefits rather than isolated component-level benchmarks
Develop and lead performance optimization methodologies based on real AI workloads. Measure and optimize tokens/sec, time-to-first-token (TTFT), time-per-output-token (TPOT), latency, throughput, memory bandwidth utilization, memory capacity utilization, accelerator utilization, power efficiency, cost per inference, cost per million tokens and system level TCO
Inference Performance Optimization
Identify opportunities to improve performance through kernel optimization, quantization, sparsity, compression, operator fusion, memory placement, scheduling, batching, speculative decoding, KV-cache optimization, prefill/decode optimization and communication reduction
Software Architecture for CXL and Disaggregated Memory
Help define the software architecture necessary to make CXL-based memory expansion, pooling, and disaggregation practical for AI workloads
Investigate host memory versus CXL memory placement, shared versus dedicated memory, memory allocation policies, NUMA-aware scheduling, remote-memory latency, bandwidth allocation, memory QoS, memory migration, application-level
Develop software mechanisms that allow applications to benefit from additional capacity without unacceptable latency or performance penalties
Develop software architecture for AI context memory systems. Potential capabilities include KV-cache tiering, KV-cache compression, KV-cache quantization, prefix sharing, context reuse, cache eviction, context migration, session-aware caching, distributed context management, GPU-to-remote-memory transfers and context prefetching
versus system-level control and multi-host memory orchestration
AI Context / KV-Cache Software
Architecture Modeling and Prototyping
Develop software prototypes to validate new hardware architectures
Build workload models and emulation environments before silicon availability
Create benchmark suites representative of customer workloads
Evaluate architecture concepts using existing CPUs, GPUs, FPGAs, or emulation platforms
Quantify software overhead and expected system-level benefits
Establish performance baselines against existing GPU, CPU, CXL, and inference solutions
Required Qualifications:
Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field; PhD is a plus
10+ years of experience in systems software, AI infrastructure, computer architecture, inference systems, compilers, runtimes, or related areas
Deep understanding of modern AI/ML inference workloads
Strong systems programming skills, preferably in C/C++, Python, or comparable languages
Strong understanding of computer architecture, memory systems, operating systems, and data movement
Experience architecting or optimizing large-scale AI inference systems
Demonstrated experience translating application/workload requirements into systems architecture
Strong understanding of performance modeling and benchmarking
Excellent communication skills and ability to work across hardware and software organizations
Preferred Qualifications:
Experience with several of the following:
LLM inference
Transformer architecture
KV cache
vLLM
PyTorch
CUDA
Triton
TensorRT
ROCm
XLA
MLIR / LLVM
FlashAttention
MoE
Quantization
Speculative decoding
Distributed inference
GPU systems
AI accelerators
HBM
DDR
CXL
NUMA
Memory pooling / disaggregation
PIM / near-memory computing
AI networking
AI storage
Edge AI
Hardware/software co-design
Salary Range: $275,000