What I find odd is that after the initial Spectre attacks, there have been a long string of these attacks discovered by outside researchers and then patched by the chipmakers.
In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips.
Outside researches need to reverse all this by probing a black box (plus a few much-worse-than-insider sources like patents).
Yet years after the initial disclosures it's still random individuals or groups who are discovering these? Perhaps pre-Spectre this attack vector wasn't even considered, but once the general mechanism was obvious did the chip-makers not simply set their biggest brains down and say "go through this with a fine-toothed comb looking for other Spectre attacks"?
Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public?
I considered this, but we have pretty good evidence that the chipmakers have not been busily secretly patching Spectre attacks:
1) Microcode updates are visible and Spectre fixes are hard to hide: most have performance impacts and most require coordination from the kernel to enable or make effective (which are visible for Linux). There have been a large number of microcode changes tied to published attacks and corresponding fixes, but no corresponding "mystery" updates and hidden kernel fixes to my knowledge which have a similar shape to Spectre fixes.
It's possible they could wait to try to bundle these fixes into a microcode update that arrives for another reason, but the performance impacts and kernel-side changes are harder to hide.
2) If this were the case, we'd expect independent researches to be at least in part re-discovering these attacks, rather than finding completely new ones. This would lead to a case where an attack was already resolved in a release microcode version. To my knowledge this hasn't really happened.
That's true, but it leads the odd assumption that the vendor managed to fix N side-channel attacks before release but 0 thereafter, while random individuals fixed M thereafter over a period of years with N >> M.
This seems to be much less likely than the conclusion that vendors are not in fact fixing many prior to the release and then stopping "cold turkey" after that. Especially since these attacks seem to cross chip versions, in many cases 6+ generations of chips: if vendors had substantial and increasing efforts on new chip versions they'd also be catching issues that applied to old released chips as well. We don't see that happening.
This. Downfall affects CPUs released almost a decade ago. The argument of 'Intel caught a ton of these and downfall is the one that got away' doesn't add up. Surely Intel would find at least one issue during development of a new chip that has the property that it _also_ affects older chips. They can hardware-fix it before ever releasing their new chip, but unless they just decided to ignore the vulnerability in their older chips, they'd have starte work on a patch or at least on contacting OS vendors.
None of which is easy to do without the public at large figuring out what happend.
Conclusion: That is highly unlikely to have ever happened. Therefore, this isn't survivor bias and it really is bizarre that chip vendors (or, at least, Intel) doesn't look for this stuff or at least didn't find this.
FWIW. Spectre/meltdown really caught the hardware development world off guard. Speculative execution was well-trodden ground and we thought it was fine. After these attacks, we had legacy designs that we needed to patch in a hurry, and the performance costs of not speculating were unfathomable. A lot of work went into mitigations like new kinds of barriers. But hardware designs are enormous and there is state lurking absolutely everywhere. It’s just a really hard problem.
dan bernstein had pointed out the risk that speculative execution would lead to side-channel attacks years before, but there's a common pattern where security people point out a risk and then vendors dismiss it as impractical until it's demonstrated in practice
In computer engineering grad school, I mentioned to my PI that there was an upcoming hardware security flaw that I had heard about from … sources… and he quickly guessed it was speculative execution related without knowing anything more. I think people knew it was possibly dangerous, but the performance gains from speculative execution were huge enough nobody in Intel/AMD red-teamed the design basically.
If a bug is not known, most of the incentives to the vendor are to not bother investigating, I suspect.
You could spend arbitrary amounts of time looking for these bugs and find nothing. Simpler and easier to offer a bounty or something and fix it then. If no one publicly finds the bug it doesn't matter to the mfg (and it wouldn't surprise me if there's truth to your supposition that they know about the bug but wait to fix until someone reports it - no public backlash so long as the bug is unknown)
Fixing bugs prior to release seems easy and free, though, especially since many more eyes would also be on the "new" in progress architecture, and proper hardware mitigations that don't cost a lot of performance can be made.
What about the incentive to release "the most secure chips on the market", are you discounting that a bit too much?
Granted that human nature tends to mean these factors don't have a high enough weight, e.g. it's not the safest airplanes that sell the most, it's the cheapest ones that meet the regulations, and the regulations drive safety improvements, for the most part
I guess there's probably some margin in it - if both parties seem about equally vulnerable, there's not much lost. You could expend a lot of effort into security, but the nature of these bugs is still that they are fairly rare, often require pretty significant hurdles, etc. The mfg. could probably spend a lot more money and find a few extra bugs, but who knows if they would have turned into "real" exploits?
Remember that this particular bug isn't actually present on the newest chips either - and 12th/13th gen were shipping before Intel was informed of this bug - so it was fixed eventually, probably incidentally as a result of design changes.
The unknown factor is how much additional money you'd have to invest to gain additional security, given how esoteric many of these bugs are.
Or the problem could be with methodology, and the wrong people are in charge of the right people left, and so the mindset for testing is just wrong.
Also you’re dealing with a company that has been running to stand still for a long time. There’s been a lot of pressure to meet numbers that they simply cannot keep up with. At some point people cheat, unconsciously or consciously, to achieve the impossible.
There's probably thousands if not more. The way I always imagined this working was. agency and company work together to leave gaps in our hardware and software under the ruse of "it must be secure enough for our use." Agencies get a preemptive six months or so to find enough zero days to do what they want. The engineers at the company screaming about the issues are ignored.
Eventually a foreign adversary or domestic hacker finds one that can cause a lot of harm. As soon as they find one a DOD funded student simultaneously discovers it. Alternatively, if documents leak showing how these exploits could happen, same scenario.
Not to say all bugs are known, but I'd imagine a fair deal of them certainly are.
The logic in this comment rubs me the wrong way. You could use the same train of thought to postulate programmers that have made 2 memory safety errors are nefarious instead of simply human.
When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why.
I think the comparison between CPU and software exploits holds at a very high level, but in the case of software the gap between internal and external researches seems lower. Much software is open-source, in which case the play field is almost level and even closed source software is available in assembly which exposes the entire attack surface in a reasonably consumable form.
Software reverse-engineering is a hugely popular, fairly accessible field with good tools. Hardware not so much.
> When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why.
To be very clear none of these errors have been found by billions of random users but by a few interested third parties: many of them working as students with microscopic funding levels and no apparent inside information.
I'm not actually suggesting that the nefarious explanation holds: I'm genuinely curious.
I'm not sure I agree significant gaps in the playing fields are really there. By significant I mean something that explains it should be e.g. 10x harder or something to the point it's supposed to be suspicious how the ratio is indicative of something off. Sure, you don't get to see how they laid out the transistors of the CPU but that's not how these attacks work it's by some oversight in memory handling not that different from software. They analyze how individual assembly instructions behave in certain scenarios vs how they are designed to behave. Compare this to the process of attacking closed source software like Windows, sift through a bunch of assembly in a debugger and see what gets left behind or compared incorrectly, and it's not glaringly different just because hardware is involved. Difficult, sure, but far from anything to suggest it should be uncommon. More importantly, by definition you don't really get to see all of the things they do catch. Maybe they are getting 90% of other related vulnerabilities with the patches but you only hear about the 10% that weren't covered by it because that's the only thing someone is going to publish/get a cve for/make the front page of HN.
The point isn't that billions of users all actively try to exploit software it's that if you have billions of users then even if 0.01% try to then that's still a hell of a lot more external bug finders than internal bug finders.
Yeah nothing against you or genuine curiosity it's just when a comment sets up a series of logic and concludes with a leading question then the conversation is damned to largely revolve around the leading question instead of genuine answers.
I dabble in this space (hardware reverse-engineering) and write software for a living and in my opinion the gaps are huge.
I should disclose have been paid by a chip-maker for a blog post that I wrote which "disclosed" an optimization which could be uses for a side channel attack (though I did not even suggest that aspect) and which was subsequently patched away via a microcode update. The whole process was very surprising to me in that there must have been several people inside the chip-maker who knew about the optimization I described in much deeper detail ... after all they conceived and implemented it.
So by what path does a blog post mentioning it get treated as the disclosure that results it it being removed when they knew about it all along?
> that's not how these attacks work it's by some oversight in memory handling not that different from software.
I think it is very different. Assembly is merely a somewhat less convenient form of the original semantics that embeds all the relevant semantics related to the attack surface since the original source has been "erased". Many analysis tools such as fuzzers operate directly on assembly with little loss in functionality.
These attacks are against completely unspecified aspects of the instruction execution and lean heavily on the actual hardware implementation (almost at the level of "how the transistors are laid out") such as what hidden buffers are used, when they are filled, how they are shared with sibling threads, etc.
In my experience there are very few people interested in these details outside of the vendors themselves and these folks and the ones creating the exploits would fit in a modestly sized lecture hall. The scope has increased a bit lately (see Tavis's fuzzer work) but it was originally a small group with little or no funding.
Do you have a link for the blog post? I'd love to read more about that.
Since we disagree in how big the gap is, and neither of us is going to get a satisfactory answer out of a chip maker any time soon, perhaps a different argument: there are plenty or microcode updates all the time, doing more than fix just security bugs. There are also security bugs like M1racles which have nothing to do with performance incentives. If these can all be explained by a lecture halls worth of people finding things most wouldn't post release of the chip then why does the same situation on security issues require unique explanation?
This reminds me of lock manufactures vs The Lockpicking Lawyer. He is able to pick nearly every damn lock out there, yet they have all the design resources and money to hire people like him to make better locks.
I had a failed startup where we warned our customers they were using suppliers without sufficient qualifications, insurance, etc. Nobody wanted the product, even for free. At a system level, they chose to use cowboy tradespeople and cover the risk with "plausible deniability", because the market wouldn't pay them to only use quality.
It's like a Gresham's law - eventually the lowest quality dominates the market, because that's what maximises profit.
One possibility nobody mentioned yet: the chip vendors don't invest a ton of time looking for them because they don't actually matter that much.
Bear in mind, security researchers are incentivized to find things to build their reputation. It's very often the case that they claim something is a world-shaking security vulnerability when in reality it doesn't matter much for real world attackers. Has anyone ever found a speculation attack in the wild? I think the answer might be no. In which case, why would chip vendors invest tons of money into this? Real customers aren't being hurt by it except in the sense that when an external researcher forces action, they're made to release new microcode that slows things down. Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter. All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack.
> Has anyone ever found a speculation attack in the wild? I think the answer might be no.
this is known as the Y2K paradox.
The Y2K bug had the potential to be very dangerous, but due to a wide-reaching campaign and tonnes of investment in prevention, when the millennium came, it did so with very few issues (though there were still some); leading many to speculate that the issues were overblown
Yeah, but we have practically usable variants of meltdown and spectre.
the only reason its not useful to deploy them is due to a massive amount of herd immunity (and the fact you consume a lot of CPU when trying the attack making it clear - bad combo for an attacker)
> Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter.
They have off switches because they can have severe performance costs, which most security fixes don't have.
> All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack.
Former is simply not true. For example, EC2 avoids mixing tenants on the same core, but even the AWS serverless stuff doesn't do that, for cost reasons. For most end user computers, this is blatantly not true.
Operating system level sandboxing is largely irrelevant if the attacker can simply read secrets from other processes running on the same core at the CPU level. Most processes will have a way to smuggle the stolen goods out.
By "sandboxed so heavily" I meant browsers, which don't allow shared memory multi-threading, tight control over CPU instructions or high resolution timers, so it's very hard to mount specex attacks there. I've seen claims it can be done but very few demos, and the only demo I remember trying didn't actually work.
By "running on the same physical core" I meant simultaneously. You can usually wipe uarch state when switching between tenants pretty well. If AWS aren't doing this then that's something for them to solve but I'm pretty sure major cloud users don't have to worry about these attacks as so many only affect hyperthreads that are running concurrently and it's not cost prohibitive to avoid mixing tenants that way.
Maybe they're simply victims of Kernighan's Law of Debugging: "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"
There is no doubt that Intel make chips "as clever as they can". Hence, by definition, they can't fully debug them.
As a CPU engineer, I can say that spectre highlighted channels of information escape that weren't previously considered as vulnerable. That's why it kicked off a new batch of exploits. There was a new idea at the heart of it and others built on that idea.
It's also important to say that these are not bugs. The design is behaving as intentioned. That the performance differs based off the previous code that the CPU has executed was understood and deameed acceptable because the cost of the alternative was considered too high (either in power, performance or area) than the alternative. That's what it means to be an engineer. You weigh up alternatives and make a choice.
In this case, CPUs became fast enough that the fractional part of a bit per iteration became high enough bandwidth to be exploitable, but it needed someone to demonstrate it for it to be understood in the industry. That changed the engineering decision.
The problem is that nobody wants to admit that the old, stodgy mainframe guys were right 30 years ago and that sharing anything always results in an exfiltration surface.
Nobody wants to be the first to take proper steps because they have to either:
1) Partition hardware properly so that users are genuinely isolated. This costs silicon area that nobody wants to pay for.
2) Stop all the speculative bullshit. This throws performance into rewind and will put chip performance back a decade or two that nobody wants to pay for.
Until people are willing to value security enough to put real money behind it, this will continue ad nauseam.
I'd add that the status quo has done pretty well, and many of these exploits are fixed. It's also worth noting that a lot of the exploits in question may be known, but the people working on them couldn't theorize a practical exploit. How many web browser sandbox breaches have there been over the years? Far less than the CPU exploits in the past several years. The latter can have a much bigger impact though.
The biggest risk target seems to be shared servers, and you often don't know who you're sharing with, so is it worth trying? It seems to be usually, no... in a specific target, maybe.
No, 1993 is about right. The mainframe guys were screaming their heads off as these dumbass, insecure, non-ECC x86 processors were eating up more and more computing--before that the mainframe guys kind of dismissed x86 as simply toys.
A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot.
And in line with that and
>Maybe they _did_ and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public?
... maybe they have patched a number of issues and just never announced them.
> A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot.
Sure. If they had patched a bunch of Spectre vulnerabilities and independent researchers had discovered a few more that would be one thing, but as far as I can tell they have patched _zero_ while independent researches have found many and it has been years since the initial attack. Many of these follow very similar patterns and "in what cases is protected data exposed via speculative execution" is something that an architect or engineer could definitely assess.
Generally all these workarounds have a measurable slowdown associated with them. This mitigation apparently has an up to 50% cost. It's unlikely many of them have been silently fixed without people noticing.
It's the same as any product, the product team wants a faster, cheaper product, yesterday. Security and trust is secondary, because if you're lucky enough, that will fall on the next product team.
Beyond that, processors contain billions (or trillions?) of possible outcomes from a set of inputs. Testing for all of these just to verify reliability and stamp out logic bugs is really hard due to the combinatorial explosion. Putting security testing on top just complicates matter further. The best they can probably do is map out potential ways in which their general purpose processors could be used for specific nefarious uses.
The chipmakers don't have an incentive to look too hard for speculation security issues beyond a bit of PR. If they succeed, they lose money and marketshare, while their 'insecure' opponents gain and at most patches later. And in fairness, a lot of these bugs are rather theoretical. Until buyers take these bugs much more seriously, this isn't going to change.
Clouds take these vulns seriously, and have a lot to lose, and have deep wallets. I'd be surprised if this topic didn't come up when large purchases are discussed.
They have workarounds. If you prevent multi-tenant from sharing their threads on the same core, that eliminates the most desirable goal of an attacker.
However it does not eliminate the vulnerability within a single tenants own threads.
You also have to think about all the web shops that are out there that are just running proxmox or a cloud reseller who reintroduces the vulnerability in their multicore VPS setup
Big clouds have the exact same incentive issue - if a customer is really paranoid, the customer can pay the cloud extra and ensure their own exclusive infra. For regular users, the clouds can mitigate a bit at scale (but not care much about this in practice, it's not as important as cheaper faster processors).
>In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips.
I hope you're not in charge of hiring QA. Bugs are often found by people who AREN'T thinking like the developers whichstart wearing blinders on how they're stuff should work and stop trying stupid things.
I think this sort of excuses the initial blindness to Spectre style attacks in the first place, but once the basic pattern was clear it doesn't excuse the subsequent lack of discovering any of the subsequent issues.
It is as if someone found a bug by examining the assembly language of your process which was caused by unsafe inlined `strcpy` calls (though they could not see the source, so they had no idea strcpy was the problem), and then over a the subsequent 6 years other people slowly found more strcpy issues in your application using brute-force black-box engineering and meanwhile you never just grepped your (closed) source for strcpy uses and audited them or used many of the compiler or runtime mitigations against this issue.
That still seems like a job for QA and you do have a good point. If there's an attack using one technique, there should be an audit to make sure that technique can't be modified to be used in other nasty ways, then have those tests part of the test suite.
I'm in the camp that developers shouldn't be responsible for having a transcendental ability to predict future (example) security holes. If one appears, it's really the QA's job to document it and experiment with other similar vectors of attack. The developers are, you know: developing. Bring them back in when the security issues needs to be fixed, once it's found just how big the problem is.
Shouldn't the functions of "development" and "QA" both reside under the umbrella of the chip-maker though?
In fact, chip-makers famously invest an insane amount of money into "QA" (aka "validation") and many features or lack thereof are often put down to the cost of QA rather than the cost of development.
>Shouldn't the functions of "development" and "QA" both reside under the umbrella of the chip-maker though?
Sure, umbrella, but different teams. What I'm trying to get across is that it's a good thing that the developers are not wearing the QA hat, so that the QA people are thinking in ways that aren't exactly parallel to the developers. You get close to your work, you sometimes forget that you're not look at the larger picture.
Think of it like if I was building a big secure wall in front of my house, but I neglect that it's easy to just walk to the opposite side to get in. Oh, whoops, I was too focused on the solution in front of me.
Possibly. When I worked at a place that designed a "simple" chip that was just a more energy efficient and very parallel version of its FPGA, the fab that was contracted to make it insisted on validating it themselves. It consisted primarily of a lot of copy-paste of the primary logic to build as many paths as would fit on the die. They described it as "unusually dense" and, I heard they later said that they'd never seen a design that dense. The validation process was partly described as people manually driving a car through a 3D model, making sure there were no unexpected junctions or other divergences. This process took months longer than their initial estimate, allegedly due to the density.
I guess you could say this was "under the umbrella of the chip-maker", though we had little say in it aside from pressing them for progress as our final product's shipdates came and left. When we finally got the first samples, power consumption was, I think, an order of magnitude higher than expected. Our lead engineer struggled to get it down without going to a smaller process that we could barely afford (and given the delays already, could probably not have afforded to wait for). We thus thought we had working logic, but our case designs were scuttled. After enlarging the cases to accommodate extra cooling, our base unit was more than ten times taller, and our next size up, while the same height, was three times longer. Highest units had a water cooling system[1].
Our QA was able to find other sorts of flaws, like misprinted unpopulated circuit boards, software faults on the host, or when we received shoddy interlink cables that either melted under test[2] or other cables that scrambled communications[3].
All of which is to say, that many other pressing issues can interfere with doing what you feel you ought to be doing. At least at our scale. I can't speak for the likes of big guys like Intel or AMD, but it's possible that unfound faults or known unpatched flaws can ship because resources were committed elsewhere or fabrication leadtimes preclude waiting. This is not to say that shipping a security flaw is okay, but rather that sometimes you think you've done you're due diligence, or sometimes your choices seem to be "Ship, ship late, or never ship.". The answer you pick can be existential, so you hope you've picked the least bad option.
[1] Misbegotten, because "beauty of the promo images".
[2] Conductors much thinner than spec, not initially observed because both ends were fitted with moulded plugs.
[3] Longer than spec, initially recieved with enthusiasm by assembly staff, before an engineer investigating a difficulty saw them and exclaimed, "No-no-no! That's longer than I am tall! Stray capacitance alone will kill the communication.". (He uncharacteristic'ly exagerated here. While they were three times longer than expected, this was at most .40 times his height.)
Generalizing: QA/Test and dev people just think differently.
I served as QA manager for a while. I was fortunate to have worked with some really, really good QA/Test people. They're more rare than good devs.
Some individuals can do well in both worlds.
I've had great devs who were pretty good at test. It seems to me like these devs came from outside of software and CS. Like from aerospace or ballet or history.
I've mentored QA/Test people so they could better automate and manage their work. Then they could pick up tasks like CI/CD, testing scripts, scrub data, etc.
But, in my experience, devs are bad at testing, and just terrible at QA.
Getting enough QA/Test support on a team in the 90s was a tough sell, even though everyone gave lip service to quality.
It's been a long time since I've worked with an actual Test person, dedicated to that role. And that was just 1 person vs 8 devs. So ridiculous.
These days, any kind of "test" is done by "business analysts", whatever that means. And I can't recall the last QA person I've worked with (since my stint as QA Manager in the '90s).
FWIW, I wholly agree with Dan Luu's observations about today's QA/Test standard practice in software vs hardware.
And maybe just maybe when the Snowden revelations started to come out some people woke up and realised that the companies who design the processors used in the vast majority of computers are from the US.
Since Applied Cryptograhy and everyday since publication it's been well-known and well-understood that NSA's never-ending efforts to weaken systems to make it easier for them to do their work wrecks havoc and costs billions to everyone else. Luckily their attempt to get everyone to adopt a PRNG that was broken on-purpose was thwarted.
But who's to say that any chipmaker gets bribed to backdoor their design to allow the reading of any page of RAM from any protection level doesn't happen? It would make their job super easy.
Some people might say “don’t attribute to malice what can be attributed to incompetence” but introducing bugs via bribing an insider or even getting one of your people a job at Intel or AMD would be a very clever way to give yourself the keys to (nearly) all the castles.
Just don’t forget to patch the microcode on your own systems.
The intersection of probabilistic optimization and timing based side channels is a gift that will never ever fully go away.
Everything _really fast_ which is approximately any very mature systems gear has probabilistic optimization in it now, and that's where a great deal of modern performance comes from.
Even thermals and power draws produce side channels. Eradicating every side channel is untenable, research is required to understand what side channels are tenable, then we have to patch them up as best we can.
I'm waiting for the one where we find that binning is involved in an integrated way, and that some arbitrary sub-population of popular chips turn out to much more exploitable than others due to particular paths being disabled and leaking extra timing info. That'll be a really fun and awful day.
You are describing a creator bias, there are probably a better name, where you think the ones who created something knows it best. For example, you could create a programming language, a game, or anything, and you think that you know it better then someone who use it for several hours every day. The larger the user base the less likely you are better or know it better then all users.
I think you're last statement is half right. They probably don't bother looking for them that hard because if they look for them they might find them and then have to make their CPUs slower before launch.
> Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public?
Or they chose not to look for these types of bugs in the first place, for those reasons.
There's more money to be made in performance and efficiency than security. If chipmakers could design and build the perfect processor, they would. But like anything else complex, there are compromises everywhere.
Granted but the vendors accept these are serious problems given that they are immediately patched and the mitigation all enabled by default even at significant performance cost (most chip generations are down double digit perf % based versus "zero mitigations" at this int).
So they don't need to build the "perfect processor" but why aren't they discovering any of these issues themselves?
It's probably that they are aware of several vulnerabilities, such is life. But they are unable to prioritize them and assess which to solve first, they need external auditors using black box techniques to help them identify which are exploitable by external attackers, so they can fix that one, and not the other non issues
They know at release about many of these bugs but between incentivizing upgrades and selling the resultant bugs as backdoor to 'three letter agencies' there's simply too much money to be made by not disclosing/patching the problems before third parties release discovery of the problems.
My guess: There are a lot more researchers on the outside than inside. And also incentives - you would need a red team at Intel who tries to attack their own chips, and those people would be even fewer. And the best people may want to stay independent.
Having worked in the industry, my gut feeling is that chipmakers don't invest all that much in looking for and preventing these sorts of attacks.
When working on a new feature, you are desperately trying to deliver on time something that adds value in the sorts of scenarios that it was designed for. And that is already hard enough.
an easier explanation is that modern chipsets are incredibly complex, side channel attacks like these are hard to reason about and traditionally, processors themselves have not been the target of these kinds of attacks, so i don’t think the engineers working on them are accustomed to thinking about them as attack surfaces in this way.
You didn't have to think about this when you ran your code on your own chips; so it's your fault for backdooring the front end into the datacenter.
But now we have the 21st-century mainframe we like to call the cloud, where everything is shared. So I upload a container image with the vampire vuln intending to read all the activity on the host. Other customers jobs, the OS itself, even steal internal keys used at Amazon.
The motivation to do this kind of attack now is incalculable.
I kid. The entire Zen class of AMD processors are vuln to Inception so you're not safe anywhere.
There is the "just use OS/2" strategy of the 1990s (referring to the period from 1995-2002 where you could sit happy in your niche OS that researchers and BHs were not targeting): in this case s390x probably has the same problem but no one is paying attention to it (other than a 3 letter agency, of course), so the chances of a payload reaching that arch are very close to zero.
There are security teams and/or methodologies inside all major CPU designers today that look at speculation and other side channels. Although these might still not be up to quite the kind of rigor that you see in traditional verification. That is to say, their unit tests and fuzzers and formal analysis and proving methodologies are all well set up to verify that architectural results are correct, my guess is that they don't all verify intermediate results or side effects can't be observed by different privilege contexts.
In many ways it is a much more difficult problem too. Going back to first principles, what execution in one context could have an affect on a subsequent context? The answer is just about everything. If you really wanted to be sure you couldn't leak information between contexts, you could only ever run one privilege level on the machine at any time. Not just core, entire machine. When switching contexts, you would have to stop everything, stop all memory and IO traffic, flush all caches and queues, and idle all silicon until all temperatures had reached ambient, voltages, fans, frequency scaling had settled. Then you could start your next program. Even then there's probably things you've forgotten -- programs can access DRAM in certain ways to cause bitflips in adjacent cells for example, so you've probably got a bunch of persistent and unfixable problems there to if you're paranoid. That's not even starting on any of the OS, hypervisor or firmware state that the attacking program might have influenced. So the real answer is that you simply can't share any silicon, software, or wires whatsoever between different trust domains if you are totally paranoid.
All of these things are well known about, but at some point you make your best estimation of whether something could realistically be exploited and that's very hard to actually prove one way or another. Multiply by all possible channels and techniques.
That's probably why you see a side channel vulnerability discovered every month by outsiders, but very few architectural defects (Pentium FDIV type bugs).
That said, this issue looks like a clear miss by their security process. Supplying data to an untrusted context, even if it can only be used speculatively, is clearly outside what is acceptable, and it is one of the things that could be discovered by an analysis of the pipeline.
Contrast with the recent AMD branch prediction vulnerability, which could plausibly fall under teh category of it being a known risk but was not thought to be realistically exploitable.
As others have said though, everyone makes mistakes, every CPU and program and every engineering project has bugs and mistakes. I don't know if you can deduce much about a CPU design company's internal process from looking at things like this.
In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips.
Outside researches need to reverse all this by probing a black box (plus a few much-worse-than-insider sources like patents).
Yet years after the initial disclosures it's still random individuals or groups who are discovering these? Perhaps pre-Spectre this attack vector wasn't even considered, but once the general mechanism was obvious did the chip-makers not simply set their biggest brains down and say "go through this with a fine-toothed comb looking for other Spectre attacks"?
Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public?