WASHINGTON (AP, KXAN) — Hackers broke into the networks of the Treasury and Commerce departments as part of a global cyberespionage campaign. They accessed those networks by slipping malware into a SolarWinds software update, according to the global cybersecurity firm FireEye, which was also compromised.

The first phases of this monthslong cyberespionage campaign started in the spring. The malware gave the hackers remote access to victims’ networks.

The FBI and the Department of Homeland Security’s cybersecurity arm are investigating what experts and former officials said appeared to be a large-scale penetration of U.S. government agencies. Industry experts said it bore the hallmarks of Russian tradecraft.

“This can turn into one of the most impactful espionage campaigns on record,” said cybersecurity expert Dmitri Alperovitch.

The hacks were revealed less than a week after FireEye disclosed that foreign government hackers had broken into its network and stolen the company’s own hacking tools. Many experts suspect Russia is responsible. FireEye’s customers include federal, state and local governments and top global corporations.

How and why hackers targeted SolarWinds

SolarWinds emailed its customers warning about the hack and recommending they upgrade immediately.

The apparent conduit for the Treasury and Commerce Department hacks — and the FireEye compromise — is a hugely popular piece of server software called SolarWinds. It is used by hundreds of thousands of organizations globally, including most Fortune 500 companies and multiple U.S. federal agencies who will now be scrambling to patch up their networks, said Alperovitch, the former chief technical officer of the cybersecurity firm CrowdStrike.

SolarWinds is headquarted in Austin with offices off of Southwest Parkway in Southwest Austin.

FireEye, without naming the breached agencies or other targets, said in a blog post that its investigation into the hack of its own network had identified “a global campaign” targeting governments and the private sector that, beginning in the spring, slipped malware into a SolarWinds software update.

The malware gave the hackers remote access to victims’ networks.

SolarWinds responds

SolarWinds said the “potential vulnerability” was related to updates released between March and June for it Orion software, which helps organizations monitor their online networks for problems.

“We believe that this vulnerability is the result of a highly-sophisticated, targeted and manual supply chain attack by a nation state,” said SolarWinds CEO Kevin Thompson in a statement.

The compromise is critical because SolarWinds would give a hacker “God-mode” access to the network, making everything visible, said Alperovitch.

SolarWinds notified around 33,000 Orion product customers on Sunday, although it’s only believed that fewer than 18,000 were possibly made vulnerable, the company said.

According to the U.S. Securities and Exchange Commission’s report for SolarWinds, the company has retained third-party cybersecurity experts to investigate, including whether any customer systems were possibly exposed.

FireEye said it had notified “multiple organizations” globally where it saw indications of compromise. It said that the hacks did not seed self-propagating malware — like the 2016 NotPetya malware blamed on Russia that caused more than $10 billion in damage globally — and that any actual infiltration of an infected organization required “meticulous planning and manual interaction.”

The U.S. government did not publicly identify Russia as the culprit behind the hacks, first reported by Reuters, and said little about who might be responsible. Cybersecurity experts said last week that they considered Russian state hackers to be the main suspect.

National Security Council spokesperson John Ullyot said in a statement that the government was “taking all necessary steps to identify and remedy any possible issues related to this situation.”

Who are the SolarWinds customers who might be affected?

FILE – This March 27, 2008 file photo shows the Pentagon in Washington. (AP Photo/Charles Dharapak, File)

On its website, SolarWinds says it has 300,000 customers worldwide, including all five branches of the U.S. military, the Pentagon, the State Department, NASA, the NSA, the Department of Justice and the White House. It says the 10 leading U.S. telecommunications companies and top five U.S. accounting firms are also among customers.

The government’s Cybersecurity and Infrastructure Security Agency said it was working with other agencies to help “identify and mitigate any potential compromises.”

President Donald Trump last month fired the director of CISA, Chris Krebs, after Krebs vouched for the integrity of the presidential election and disputed Trump’s claims of widespread electoral fraud.

In a tweet Sunday, Krebs said “hacks of this type take exceptional tradecraft and time,” adding that he believed that its impact was only beginning to be understood.

CISA warns companies to update networks immediately

Late Sunday, CISA released a rare emergency directive asking all federal civilian agencies to update their networks by 11 a.m. on Monday and then send in a “completion report.”

This is just the fifth time CISA has issued an emergency directive since 2015.

Federal government agencies have long been attractive targets for foreign hackers.

Hackers linked to Russia were able to break into the State Department’s email system in 2014, infecting it so thoroughly that it had to be cut off from the internet while experts worked to eliminate the infestation.

Reuters earlier reported that a group backed by a foreign government stole information from Treasury and a Commerce Department agency responsible for deciding internet and telecommunications policy.

The Treasury Department deferred comment to the National Security Council. A Commerce Department spokesperson confirmed a “breach in one of our bureaus” and said “we have asked CISA and the FBI to investigate.” The FBI had no immediate comment.

“I suspect that there’s a number of other (federal) agencies we’re going to hear from this week that have also been hit,” former NSA hacker Jake Williams said.

Asking an Expert

Dr. Akbar Namin, an associate professor of computer science at Texas Tech University, weighs in with his opinion of the SolarWinds hack.

Q: The hackers were able to infiltrate their software updates that started rolling out in March, but we’re just now finding out about this. So what does that kind of speak to in terms of kind of the magnitude of this espionage campaign?

A: You don’t know what they have done. What what means have been exchanged. They have they have to what we call it situation awareness, they have to do some analysis regarding the situation. So they have to shut down the entire network immediately. And then they need to actually do a what we call it reverse updates, or, or clean updates of the network in order to make sure that that malicious updating is not there. Those are the things that they can actually do. What the implication is, it is unknown.

Q: SolarWinds, like you mentioned before, has over 300,000 clients, and 18,000 have specifically been affected at least by this hack. But you take a look at the list of their clients, and that includes many federal agencies. What is the implication for the average American? Should Americans be worried, and what what kind of information is at risk of being leaked here?

A: I would say it is not going to affect an average person like you or me; it’s going to actually affect the overall integrity and infrastructure of the United States of America. These kinds of sophisticated attacks have been launched by another state or another country, and then they are backed up or supported by the government. Whenever they are attacking at that level, they are not looking at your credit card number or my credit card number. They are looking at something else, they’re looking at the technology, they’re looking at the sensitive information related to some precedent for instance. That information perhaps is not valuable for us for you or me, but it is very, very valuable for the enemies or for the other countries who don’t have or during the competition for some other person for some reasons.


Krisher reported from Detroit and Bajak reported from Boston. Associated Press writer Matt O’Brien contributed to this report from Providence, Rhode Island.

Top Stories on KXAN.com