OpenAI appears to have used its present spate of fashions that run on NVIDIA GPUs to design the Jalapeno chip, after which paired it with its personal Gluon kernel programming language, thereby daring to threaten NVIDIA’s legendary CUDA moat. How the tables have turned!
The structure of OpenAI’s Jalapeno chip
SemiAnalysis has simply supplied a detailed breakdown of OpenAI’s Jalapeno ASIC, describing a considerably novel structure that delivers substantial economies of scale.
As such, the Jalapeno pairs a single reticle-sized compute die – constructed on TSMC’s N3P node – with an N3E I/O chiplet and HBM4 reminiscence (possible from Samsung), delivering 15.4 TB/s of reminiscence bandwidth per bundle.
The cores and reminiscence are divided into matching “slices,” giving every core slice a low-latency native view of its personal assigned HBM slice, linked via a high-bandwidth collective community.
Now, AI fashions don’t usually course of information line-by-line. As a substitute, they course of billions of numbers organized in grids or matrices. As such, the matrix engine is the “heavy lifter” constructed solely to speed up these grid-based math issues.
In fact, conventional AI math makes use of normal 16-bit or 8-bit numbers. Jalapeno, nevertheless, makes use of MXFP numerical codecs. For the good thing about those that may not bear in mind, MX codecs additional compress AI math information – all the way down to 4-bit in MXFP4. This compression is finished by grouping numbers collectively and making them share a single scaling issue (exponent), thereby slashing the quantity of reminiscence wanted to retailer and transfer information.
The Jalapeno’s matrix engines function a weight-stationary systolic array. In a typical CPU, the chip has to consistently learn information from reminiscence, carry out a calculation, and write the end result again to reminiscence. This creates a bottleneck. A systolic array, however, works like a human coronary heart that pumps blood: information flows right into a grid of tightly related processing cells, passing immediately from one cell to the subsequent one with out consistently stopping to learn/write to exterior reminiscence.
Additionally, AI calculations usually contain multiplying incoming information (like your immediate) by mounted inner values (the mannequin’s weights). Nonetheless, the Jalapeno’s matrix engines are weight-stationary, which implies that the chip hundreds the mannequin weights into the processing grid as soon as after which locks them in place (stationary).
So, by freezing the heavy AI weights in place (weight stationary), pumping information easily via the grid (systolic array), and compressing the numbers (MXFP), Jalapeño eliminates data-travel bottlenecks.
Other than matrix engines, OpenAI’s Jalapeno chip additionally sports activities 64-bit scalar cores to run the management code, handle reminiscence, and orchestrate the entire operation. These scalar cores bear Out-of-Order (OoO) execution pipelines and are available paired with an L1 cache. This prevents these cores from stalling whereas ready for information. As such, the out-of-order scalar cores work forward of the maths engines: they stream information into specialised {hardware} queues in order that when the inflexible, in-order matrix and vector cores are prepared, the numbers are already lined up ready for them.
Lastly, the chip boasts of FP32/INT32 vector cores, which might apply one single instruction to a whole row of numbers concurrently, stepping in for situations that require high-precision math, and the place compression from matrix engines would smash the outcomes. That is helpful for finalizing the chances of the subsequent phrase (softmax layers) or normalizing information layers.
In fact, OpenAI isn’t just constructing the Jalapeno chip but in addition a whole rack-scale answer that consists of:
- The CPU Host Rack (Katsu): This rack homes 16 distinct host CPU trays. Every particular person Katsu tray is provided with two Turin-class AMD EPYC processors, 1.5TB of ordinary DRAM system reminiscence, native NVMe SSD storage, and 400G frontend networking hyperlinks.
- The ASIC Accelerator Rack (Vindaloo): Positioned on to the correct of the host rack, this comprises 16 corresponding accelerator trays. Every Vindaloo tray homes 8 Jalapeno ASICs, totaling 128 Jalapeno chips per single rack deployment.
- To attach the 2 racks collectively, 8 exterior, high-speed PCIe Direct Connect Copper (DAC) cables run horizontally throughout the entrance of the server cupboards, bridging every Katsu host tray on to its matching Vindaloo ASIC tray.
- Every 128-chip ASIC rack delivers as much as 1.7 ExaFLOPs of 4-bit (MXFP4) processing energy, and packs 27.5 TB of next-generation HBM4 reminiscence per rack, with every particular person Jalapeno bundle using six 12-high HBM4 stacks to attain an astonishing 15.4 TB/s of reminiscence bandwidth.
OpenAI’s Jalapeno chip blows away competitors
As per the outcomes shared by SemiAnalysis, the Jalapeno is by far probably the most environment friendly chip for inference workloads. On the InferenceX benchmark suite, it delivered 1.5x to 1.9x more AI work per watt at peak throughput and 1.7x to three.6x decrease end-to-end latency than the perfect NVIDIA GB200/GB300 outcomes which might be obtainable proper now, based mostly on the testing of OpenAI’s GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T fashions. Critically, the chip is just not particularly optimized for one single household of fashions, and may seemingly run any mannequin with ease.
Based on SemiAnalysis, the Jalapeno “delivers 13.4 PFLOPs of MXFP4 on a single reticle-sized compute die that’s manufactured on TSMC’s N3P. This compares to 17.5 PFLOPs of dense Rubin NVFP4 for a single Rubin compute die that’s comparable measurement and on the identical node.”
What’s extra, whereas OpenAI has rated Jalapeno at a TDP of 700W, it is sustained energy consumption usually stays at or beneath 550W throughout energetic AI workloads.
Critically, the Jalapeno achieves this feat whereas utilizing a Single-Token Prediction (STP) structure, the place the mannequin takes a immediate, processes it, and predicts precisely one token (a phrase or phrase piece) at a time. Because of this the ASIC didn’t depend on architectural tips like multi-token prediction (MTP) or speculative decoding to inflate its efficiency numbers. Do word that speculative decoding provides round a 3x enchancment on value per token.
As a remaining caveat, do word that the Jalapeno sports activities HBM4 whereas the GB200 and GB300 techniques sport HBM3E. This does stack the deck barely in favor of OpenAI’s ASIC.
Concerning the author: Writing is my one incontrovertible ardour. Over the previous six years, he has authored over 2,200 distinct articles on monetary and tech-related subjects, spanning almost 1 million phrases. And he has been a member of Wcctech mobile workforce since 2025.
As an alumnus of the College of Toronto, Rotman Commerce Program, I deliver nuance, in-depth information, and a novel perspective to each matter that I cowl.
After I’m not writing, I am touring the world, exploring hidden confectionaries and eating places as an aspiring meals connoisseur.
Observe Wccftech on Google to get extra of our information protection in your feeds.
