The security of the supply chain is all Enchilada, but who is willing to pay the price? – Krebs on security

From time to time, cybersecurity stories of potential impact appear such that they make all the other tiny and insignificant security concerns appear in comparison. Yesterday was one of those moments. Bloomberg Businessweek On Thursday, a bomb investigation revealed that Chinese cyber-spies would have used a US-based technology company to secretly integrate tiny computer chips into electronic devices purchased and used by nearly 30 different companies. There are still no corroborative accounts of this scoop, but it is both fascinating and terrifying to explain why threats to the global supply chain can be so difficult to detect, verify and counter.

In the context of IT and Internet security, supply chain security refers to the challenge of validating that a given electronic component – and by extension the software that powers these IT components – does not include any foreign components or fraudulent beyond what was specified by the manufacturer. company that paid for the production of the said article.

In summary, the Bloomberg story claims that the technology giant based in San Jose, California Supermicro was sort of caught in a plan to quietly insert a rice-size computer chip on the circuit boards that are placed in a variety of servers and electronic components purchased by major suppliers, which would include apparently Amazon and Apple. The chips reportedly spied on device users and returned unspecified data to the Chinese army.

It is essential to note that Amazon, Apple and Supermicro have categorically denied most of the claims contained in Bloomberg's article. In other words, their positions refuting the essential elements of the story would seem to leave little room for a retreat on these statements. Amazon has also written a blog post that more categorically states its objections to the Bloomberg article.

Nevertheless, Bloomberg reporters write that "business refusals are being countered by six senior national security officials, both current and past, who – during conversations begun under the Obama administration and prosecuted under the Trump administration – detailed the discovery of the chips and the government investigation. "

The story continues:

Today, Supermicro sells more server motherboards than almost everyone else. It also dominates the $ 1 billion market for cards used in special computers, from MRI devices to weapons systems. Its motherboards include custom server configurations such as banks, hedge funds, cloud providers, and web hosting services. Supermicro has assembly sites in California, the Netherlands and Taiwan, but its motherboards, its main product, are almost all manufactured by subcontractors in China.

Many readers have asked for my opinion on this piece. At the beginning of the year, I heard similar claims about Supermicro and tried hard to verify them, but without success. In itself, this should be an indicator of the potential merit of the story. After all, I'm just a guy, but that's the type of scoop that usually takes entire portions of a newsroom to do research, report and perform veterinary checks. According to Bloomberg's own account, it took more than a year for the story to be told and written, and 17 anonymous sources confirmed the activity.

Most of what I have to share here is based on conversations with enlightened people over the years who would probably be confined to a tiny, windowless room for an extended period of time if their names or quotations appeared in a story like this. Here I am going to walk carefully on this subject.

The US government is not eager to admit this, but there has long been an unofficial inventory of technical components and suppliers who are prohibited from buying goods or services on behalf of the US government. Call it the "blacklist", "blacklist", "list of entities" or what you have, but it basically consists of an indelible index of companies on the permanent list of Shit of Uncle Sam for being caught in the act of cheating. .

More than ten years ago, when I was a journalist with The Washington PostAccording to an extremely well-placed source, a Chinese technology company was added to Uncle Sam's list of entities because it sold a custom hardware component for many printers equipped with the Internet, which secretly copied every document or image sent to the printer. and passed this on to a server allegedly controlled by hackers lined up with the Chinese government.

This example gives a whole new meaning to the phrase "supply chain", is not it? If Bloomberg's reports are accurate, it's kind of the same thing we're dealing with here in Supermicro.

But here's the bottom line: Even if you identify technology providers guilty of hacking the supply chain, it can be difficult to impose a ban on the supply chain. One of the reasons is that it is often difficult to distinguish the brand name of a given gadget that actually manufactures all the multiple components that go into an electronic device being sold today.

Take, for example, the problem right now with insecurity Internet of Things Peripherals (IoT) – cheapo security cameras, Internet routers and digital video recorders – sold in places such as Amazon and Walmart. Many of these IoT devices have become a major security problem because they are massively insecure by default and difficult or even impractical to secure after their sale and commissioning.

For every Chinese company that manufactures these IoT devices, there are dozens of "white label" companies that market and / or sell the essential electronic components as such. Thus, while security researchers could identify a set of security vulnerabilities in IoT products manufactured by a company whose products are labeled in white, informing consumers of third-party products containing these vulnerabilities can prove extremely difficult. In some cases, a technology provider responsible for some of this mess can simply shut down or shut down and be reborn under different names and managers.

Notice, nothing indicates that anyone is intentionally designing so many of these IoT products as insecure; A more plausible explanation is that increased security tends to make devices much more expensive and slower to be marketed. In many cases, their insecurity comes from a combination of factors: they come with all the imaginable features enabled by default; they include obsolete software and firmware components; and their default settings are difficult or impossible for users to change.

We do not often hear about intentional efforts to undermine the security of the technological supply chain, simply because these incidents tend to be quickly classified by the military when they are discovered. But the US Congress has held numerous hearings on supply chain security issues and the US government has repeatedly taken steps to prevent Chinese technology companies from doing business with the federal government and / or companies based in the United States.

More recently, the Pentagon has banned the sale of ZTE and Huawei Chinese phones made in China on military bases, according to a directive from the Department of Defense that cites the risks related to the safety of these devices. the US Department of Commerce also introduced a seven-year export restriction for ZTE, which had the effect of banning US manufacturers of components that sell products to ZTE.

However, the problem is not that we can not trust the technological products made in China. Indeed, there are many examples from other countries – including the United States and its allies – that have slipped their own "back doors" into hardware and software products.

Like it or not, the vast majority of electronic products are made in China, and that should not change any time soon. The central problem is that we have no choice at the moment.. The reason is that, according to almost all accounts, it would be extremely expensive to replicate this manufacturing process here in the United States.

Even if the US government and Silicon Valley had the necessary funding and political will, insisting that products sold to US consumers or the US government be made only with components manufactured here in the United States would significantly increase the cost of doing business. all forms of production. Technology. Consumers would almost certainly hesitate to buy these much more expensive devices. Years of experience prove that consumers are not interested in paying a huge premium for safety when a comparable product offering the features they are looking for is available at a much cheaper price.

Indeed, noted security expert Bruce Schneier calls security of the supply chain "an insurmountable problem".

"Our IT industry is inexorably international and anyone involved in the process can undermine the safety of the end product," wrote Schneier in an article published earlier this year in The Washington Post. "Nobody even wants to think of anything in the United States; prices would multiply several times. We can not trust anyone, but we have no choice but to trust everyone. No one is ready for the costs that this would entail. "

The Bloomberg piece also addresses this elephant in the room:

"The problem under discussion was not only technological. He referred to the decisions made decades ago to send advanced production work to Southeast Asia. In the meantime, China's low-cost manufacturing industries have been at the base of the business models of many of the largest US technology companies. For example, Apple early made many of its most sophisticated electronic devices in the country. Then, in 1992, the company closed a state-of-the-art motherboard and computer assembly plant in Fremont, California, and sent much of that work overseas.

Over the decades, the security of the supply chain has become an article of faith despite repeated warnings from Western officials. A belief was formed that it was unlikely that China would compromise its position as the world's workshop by letting its spies meddle in its factories. It remained to decide where to build commercial systems based largely on where the capacity was the largest and least expensive. "You end up with a classic Satan market," said a former US official. "You can have less supply than you want and guarantee its security, or you can have the supply you need, but there will be risks. Each organization accepted the second proposal. "

Securing the technology supply chain poses another major challenge: it is tedious and costly to detect when products may have been deliberately compromised during part of the manufacturing process. Your typical motherboard, produced by a company such as Supermicro, can hold hundreds of chips, but it only takes one chip to corrupt the security of the entire product.

In addition, most of the US government's efforts to control the global technology supply chain seem to be aimed at preventing counterfeits – not at secretly finding added spyware.

Finally, it is also unclear whether the private sector is up to the task. At least not yet.

"In the three years since McLean's briefing, no commercially viable means of detecting attacks like Supermicro's motherboards has emerged – or seems likely to be born", concludes the story. Bloomberg. "Few companies have the resources of Apple and Amazon, and it took them luck even to detect the problem. "This equipment is at the cutting edge of technology and there is no simple technological solution," says one of McLean members. "You have to invest in things the world wants. You can not invest in things that the world is not yet ready to accept. "

For my part, I try not to spin my wheels in worrying ways that I can not change, and the challenges of the supply chain definitely fall into this category. I will have some additional thoughts on the supply chain problem and what we can do about it in an interview that will be published next week.

But for the moment, there are some things that deserve reflection that can help mitigate the threat of stealthy hackers in the supply chain. Writing this week's newsletter published by the SANS Institute, security training company based in Bethesda, Maryland, member of the editorial board William Hugh Murray has some provocative thoughts:

1. Abandon the password for all applications, but trivial. Steve Jobs and the ubiquitous mobile computer have helped reduce costs and sufficiently enhance the convenience of strong authentication to overcome all the arguments.
2. Abandon the flat network. Secure and secure communication now takes precedence over the simplicity of communication.
3. Move traffic monitoring from encouraged to essential.
4. Establish and maintain end-to-end encryption for all applications. Think about TLS, VPN, VLAN and physically segmented networks. Software networks define this in the budget of most companies.
5. Abandon the "read / write / execute" default access control rule, which is convenient but dangerously permissive, in favor of the "read / execute only" restriction or, better still, the "minimal privilege". The least privilege is expensive to administer but it is effective. Our current strategy of "shipping early / late to low quality" is proving to be inefficient and costly in terms of maintenance and violations that we could never have imagined.