AVX-512 explained: VBMI and the CPU flags behind Java4Shell, and how to check a server
AVX-512 is a family of separate extensions, and only some CPUs have the VBMI one that Java4Shell needs; here is how to check hosts and JVMs.
· 6 min read
Written by AI from the sources linked below. How we use AI
On this page
In short
AVX-512 is a set of 512-bit vector instructions for x86-64 processors, split into many separate extensions that each have their own CPU flag. "Does this server have AVX-512?" has no single answer: you have to name the extension. The Java4Shell vulnerability (CVE-2026-114938) only matters on hosts with the one called VBMI, and on Linux you can check in seconds by looking at the flags in /proc/cpuinfo.
What AVX-512 is
AVX-512 consists of 512-bit extensions to the 256-bit Advanced Vector Extensions instructions for x86. Intel proposed it in July 2013 and first shipped it in the 2016 Xeon Phi x200 (Knights Landing); it later appeared in a number of Intel and AMD CPUs, according to Wikipedia's reference article. A vector instruction applies one operation to many pieces of data at once, which suits workloads such as compression, cryptography, machine-learning inference and, as in HotSpot, text decoding.
Besides wider registers, AVX-512 raises the number of vector registers from 16 to 32 and adds eight "mask registers". A mask lets one instruction work on only some of the lanes, for example writing only the bytes whose mask bit is 1. That detail is exactly what went wrong in Java4Shell: Calif explains that the decoder built its store mask from the output length, and a wrong length produced a mask that was all ones.
Why there are so many flags
Only the core extension, AVX-512F (Foundation), is required by every AVX-512 implementation. The rest, such as CD, DQ, BW, IFMA and VBMI, are optional and can be implemented independently. Wikipedia notes this was a departure from the historical rule that a whole instruction block must be implemented together. The practical result is that two CPUs can both be described as "AVX-512" and still differ in which instructions they run.
The extensions that matter here:
- AVX-512F: the foundation. Introduced with Xeon Phi x200 and Skylake-SP.
- AVX-512BW: extends AVX-512 to 8-bit and 16-bit integer operations.
- AVX-512VBMI: adds vector byte permutation instructions that AVX-512BW lacks. Wikipedia lists it as introduced with Cannon Lake.
- BMI2: a separate, older bit-manipulation extension; Calif's check includes it.
Per Wikipedia's CPU table, the mainstream server families with VBMI include Intel Sapphire Rapids and later P-core Xeons and AMD Zen 4, Zen 5 and Zen 6. Skylake-SP (2017) and Cascade Lake (2019) have AVX-512 but, per the same table, not VBMI. Endor Labs states the same split for Java4Shell: Skylake-SP, Cascade Lake and AMD before Zen 4 are not affected, while Ice Lake and later Xeons and Zen 4 and later are. One more wrinkle: Wikipedia notes Alder Lake desktop chips never officially supported AVX-512.
How to check a Linux host
Calif gives this command, which prints a flag only if the CPU reports it:
grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(avx512f|avx512bw|avx512vbmi|bmi2)$'
For Java4Shell, Calif says the output must include all four flags: avx512f, avx512bw, avx512vbmi and bmi2. Endor Labs uses a narrower test for the key one:
grep -o -w -m1 'avx512vbmi' /proc/cpuinfo
If that prints avx512vbmi, the CPU has the instructions. If it prints nothing, the vulnerable decoder cannot run on that host.
Two cautions. First, the CPU flags describe the machine, not the JVM. Virtual machines and containers can hide or expose flags depending on how the hypervisor presents the CPU, so run the check inside the environment where your Java process actually runs. Second, mixed fleets are common: an autoscaling group may include both AVX-512 and older instance types, so a single passing host does not clear the cluster.
How to check the JVM
Whether HotSpot will use AVX-512 is a separate question from whether the CPU has it. Endor Labs recommends running this with the same JDK and options as the service:
java -XX:+PrintFlagsFinal -version | grep -E ' (UseAVX|UseBASE64Intrinsics) '
UseAVX reports the vector level HotSpot decided to use; a value above 2 means the AVX-512 path is allowed. UseBASE64Intrinsics is true when HotSpot is allowed to replace the Java Base64 loop with its optimised version. Endor Labs' rule: when the CPU prints avx512vbmi, UseAVX is above 2 and UseBASE64Intrinsics is true, the vulnerable path is live on that JVM. Horizon3 adds that a running process can be checked with jcmd <PID> VM.flags -all.
These two flags are also the workarounds Calif names for Java4Shell: -XX:-UseBASE64Intrinsics disables hardware-accelerated Base64, and -XX:UseAVX=2 keeps the JVM off AVX-512. The second is safe only on x86-64; Calif warns that OpenJDK on ARM does not recognise it and fails to start. Horizon3 notes Oracle does not publish either as a supported workaround, so treat them as a stop-gap until you install a patched build.
Why this matters beyond one CVE
HotSpot contains hand-written assembly "intrinsics" for hot library routines, and they are selected at runtime based on the CPU. Endor Labs' timeline says OpenJDK 17.0.6 shipped in January 2023 with the AVX-512 Base64 decoder backported from JDK 18, which is why the problem reaches back beyond the latest releases. The same hardware dependence explains why ARM64 hosts, including AWS Graviton and Apple silicon, are not affected: Endor Labs says the x86 stub is never generated there.
The lesson for operators is that hardware inventory is now part of vulnerability triage. For this bug, three facts decide exposure: the Java version (17.0.6 or later), the CPU flags (VBMI) and whether the application decodes attacker-controlled Base64. Our news report on Java4Shell lists the affected distributions and the patch status.
A triage checklist for a fleet
Use the same order as the facts decide exposure, cheapest check first:
- Inventory by CPU, not by Java. On each host class (instance type, bare-metal model, hypervisor cluster), run the
avx512vbmicheck. Hosts that print nothing, and all ARM64 hosts, drop out of scope for Java4Shell immediately. - Then by Java version. On the remaining hosts, list every JVM, including those bundled inside products, containers and appliances. Endor Labs says 17.0.0 to 17.0.5, Java 8 and Java 11 lack the x86 decode stub, with the Oracle Java 8 performance build named by Oracle as an exception (Calif, Horizon3), so confirm that one directly.
- Then by exposure. Does the application call
java.util.Base64on data an attacker can supply? Horizon3 says the flaw needs attacker-controlled input to reach the basic or URL decoder injava.util.Base64, and that the MIME decoder takes the scalar path (Endor Labs). Authentication tokens, uploads and network messages are the usual routes, per Calif. - Patch or mitigate in that order of urgency. Internet-facing hosts running the products Calif names (Keycloak, Jira, Confluence, Artifactory, Jenkins, PingFederate) go first. Where no fixed build exists yet, apply the JVM flag and note the performance trade-off.
- Record the result. Because instance types change, keep the CPU-flag result with the host class so the next autoscaling or hardware refresh does not reintroduce an exposed pool.
The same method applies whenever a vulnerability is tied to an instruction-set extension: work out which extension, find which hosts expose it, and only then ask whether your software uses it.
Common questions
Does an old Xeon with "AVX-512" in its spec sheet count? Not necessarily. Skylake-SP and Cascade Lake have the foundation instructions but not VBMI, so Java4Shell's path cannot run on them (Wikipedia's table, Endor Labs).
Is ARM safe? For this bug, yes: Calif and Endor Labs both say ARM64 is unaffected. That does not make ARM Java safe from other bugs.
Does turning off AVX-512 slow everything down? Calif says the -XX:UseAVX=2 flag keeps the JVM off AVX-512; the cost depends on how much your workload benefits. Horizon3 notes disabling the Base64 intrinsic removes its performance benefit. Measure before and after if you rely on heavy Base64 decoding.
Why does AVX-512 sometimes make a CPU slower? Wikipedia notes that on some processors, mostly pre-Ice Lake Intel, 512-bit instructions can trigger frequency throttling greater than earlier vector sets, which can penalise mixed workloads. The downclocking is triggered by the 512-bit width and not by the 128 or 256-bit parts of AVX-512.
What is AVX10? Announced in July 2023 as AVX-512's successor, it replaces the many flags with a version number where each version includes the previous one, per Wikipedia. That is meant to make detection simpler.
Sources
Written by
The official account of the people who run audiotale.net: announcements, conversation starters and the newsroom's drafts. Posts written with AI say so. Every other account here is a real person.
Read next
How Java decides which certificates to trust: cacerts, keytool and revocation
A step-by-step guide to the Java trust store, the keytool commands to inspect and edit it, why revocation is off by default, and a routine for the next CA compromise.
audiotale.net team7 min read
CVE-2026-129448: Apache Roller XML-RPC RCE exploited in attacks, fixed in 6.1.6
Apache says unauthenticated attackers are already exploiting a critical deserialization flaw in Roller 4.0.0 through 6.1.4; here is how to patch or mitigate.
audiotale.net team8 min read
Java4Shell CVE-2026-114938: OpenJDK Base64 RCE on AVX-512 CPUs, what to patch
A heap overflow in HotSpot's AVX-512 Base64 decoder affects OpenJDK 17.0.6 and later on x86-64; here is who is exposed, how to check and the flag that turns off the bad code.
audiotale.net team8 min read