S4x24 – Miami – Last year, energy giant Southern Company decided to inventory all the hardware, software and firmware of equipment operating at one of its Mississippi substations in an effort to create a software bill of materials (SBOM) for the OT site .
The SBOM experiment began with Southern Company’s cybersecurity team traveling to the Mississippi electrical substation to physically catalog the equipment there, take photos, and collect data from network sensors. Then came the most daunting – and sometimes frustrating – part: acquiring software supply chain details from the 17 vendors whose 38 devices the company identified at the substation during its reconnaissance mission.
Alex Waitkus, principal cybersecurity architect at Southern Company and lead on the SBOM project, said that collecting and correlating information about all the hardware, software, firmware and interdependencies in systems for SBOM provided the company energy a deeper understanding of all software components in the substation and their potential exploitable vulnerabilities. Prior to the project, Southern had visibility into its OT network assets via its Dragos platform, but the details of the software were an enigma.
“We had no idea about the different versions of the software we were using” before launching the SBOM project, he said in a presentation here this week at the S4x24 ICS/OT security conference. “We had different business partners managing different parts of the substation.”
This meant consulting with physical security and maintenance teams, for example, to help the company get a complete picture of the substation’s software supply chain. Southern cataloged a total of 18 control network system devices from two vendors; eight physical security devices from three vendors; four cybersecurity and telecommunications devices from four vendors; eight OT monitoring systems from six vendors; and a vendor’s OT failure system device.
The next step in the project was to collect SBOMs from each of the vendors represented in the substation. But there were obstacles, as nearly 60 percent of the suppliers Southern contacted for their products’ SBOM information refused to provide the information to the utility. On average, it took Southern 60 days and a dozen meetings to actually receive SBOMs from collaborating suppliers.
“So we were updating the firmware [in some cases] “when we receive” the SBOM, Waitkus said. That meant an outdated SBOM, said Waitkus, who co-presented Southern’s findings here at S4 with Matt Wyckhouse, CEO of software supply chain risk provider Finite State. OT SBOM roundtable organized by Wyckhouse at last year’s S4 conference.
Countering the software supply chain
SBOMs are a kind of list of ingredients of a software program – the components and their versions – aimed at providing transparency into the software and its composition to identify possible security weaknesses and vulnerabilities. Southern’s project was particularly challenging because creating an SBOM for an OT environment involves more hurdles than for an IT network, primarily because industrial networks often have older software in legacy equipment critical to supporting process uptime industrial. And getting information about that old code isn’t easy.
“Supply chain transparency is an interesting question that I never thought I would get into,” Waitkus told Dark Reading in an interview. But Southern’s SBOM experiment, he said, highlighted three key security benefits for the power company: North American Electric Reliability Corporation’s Critical Infrastructure Protection (NERC CIP) compliance management, vulnerability management and patching priorities of the software.
“When you have a lot of substation sites, a lot of people aren’t always comfortable rolling out firmware updates remotely… so you have a lot of people going to the sites to do work. With proper vulnerability management integration , you can prioritize patching vulnerability and exploitability analysis – and potentially reduce the overall costs” for such updates, he explained.
SBOMs can also provide visibility into a threat before it escalates, he noted. “Think about Log4j: It took us almost two years to get a complete read” on what software was vulnerable to the flaw, for example, she said. “SBOMs can help identify that and make it easier for them [software suppliers] to know if they will have a Log4j in the future.”
Waitkus said he envisions SBOMs as useful tools in the company’s procurement process as well, allowing Southern to gain deeper visibility into software products during the launch phase: “What’s your third-party code? your copyright license?”
In their presentation, Waitkus and Wyckhouse admitted that the project did not capture all the data they had hoped for, and due to a lack of information from vendors, the overall quality of the resulting SBOM was less than ideal.
Some vendors were eager to assist Southern in identifying its software, but most resisted. For example, one supplier told Southern in an email that SBOM information on its particular product was not available because it predates the SBOM requirements under the Biden Administration’s 2021 time frame Executive order for the security of critical infrastructures.
Trust but verify
However, Southern did not simply treat supplier SBOMs received at face value. Waitkus and his team created scripts to analyze the accuracy of the SBOMs and confirm that they had correct data on component and code dependencies. In one case, Southern found in its firmware analysis that the vendor’s SBOM was outdated and 72 components were missing. Several SBOMs from other vendors were also missing component and dependency data.
Software vulnerability information provided by Southern’s vendors also required vetting to ensure it was accurate and timely, so Waitkus and his team mapped SBOMs into vulnerability databases. Waitkus and his team ran vulnerability assessment and analysis tools to strengthen the testing process and brought in independent vulnerability testers to eliminate bugs.
The goal was to find exploitable vulnerabilities in substation systems. In some cases, this meant that a vendor’s vulnerability exploitability analysis included in SBOM did not show the true picture of threats to Southern.
“We started to understand that the resounding number of vulnerabilities [in some vendor reports] it wasn’t that big,” Waitkus told attendees. In one open source package, for example, there were 3,000 vulnerabilities listed in the vendor-provided SBOM, but Southern narrowed it down to just 7 that were actually exploitable, he said.
In another case, a vendor had classified a vulnerability as unexploitable, but Southern found otherwise in its own analysis. “There were public exploits on the interwebs that we could use” to prove it, she said in his presentation. “Trust but verify.”
SBOM: a work in progress
Southern Company plans to make the SBOM program operational, but it still has work to do before it can go into production. One challenge: Some vendor contracts will need to be restructured to include SBOM requirements, Waitkus told Dark Reading.
“Sharing SBOM is complicated right now,” Finite State’s Wyckhouse noted in the presentation. “If you’re just collecting SBOM and can’t do anything with the data, it’s just JSON documents in a folder.”
Meanwhile, a next phase of the OT SBOM project is about to get underway. Schneider Electric, MITER, Ameren, EPRI and Scythe will join Southern in an effort to automate certain elements of the Southern SBOM project, including inventory, SBOM collection, auditing, vulnerability and exploit analysis.