Cybersecurity's Double-Edged Sword
RL chatted with Malcolm Harkins, chief security and trust officer at HiddenLayer, to discuss why artificial intelligence (AI) is cybersecurity’s double-edged sword.
EPISODE TRANSCRIPT
Paul Roberts: Hey everybody, welcome back to another episode of ConversingLabs podcast. I'm your host, Paul Roberts. I'm the director of editorial and content here at ReversingLabs. And today I'm thrilled to say that we're being joined by Malcolm Harkins, who is the chief security and trust officer at HiddenLayer.
Malcolm and I are going to be talking about AI's double-edged sword in cybersecurity. HiddenLayer's a firm that specializes in AI security or AISec, if we wanna invent a new cybersecurity term. And we're gonna talk about that, about the origins of the company and about the latest AI driven threats to software supply chains, including the Chinese DeepSeek R1 that HiddenLayer did a really interesting report on.
So Malcolm, welcome to ConversingLabs Podcast. Really great to have you.
Malcolm Harkins: Thanks, Paul. Happy to be here.
Paul Roberts: So for our audience who may not have met you or be familiar with you, talk a little bit about first of all your role at HiddenLayer, chief security and trust officer, what that's all about.
And also your sort of journey to cyber if you will, or AI cyber and how that all played out.
Malcolm Harkins: Yeah. Thanks. As you mentioned, I'm chief security and trust officer at HiddenLayer, so I'm trying to oversee and shepherd all of our internal risk controls, compliance activities for the company.
I also dabble and spend a lot of time in the public policy space. So the legal and regulatory angles of technology, and then dabble in social responsibility and a few other things. I've been with the company now officially about a year and a half. The company is about three years old right now. Came out of stealth mode, late summer of 2022, I had been an advisor to the founders and the company upon its founding. And this also relates back to how I ended up here, the origins of the company and back into my history. So I'll roll back the clock a little bit further to discuss the company origins and our basically founding story.
We were born basically out of a breach of artificial intelligence. A lot of people don't know that. But it goes back into my days when I was chief security and trust officer at Cylance. So I was chief security and trust officer for four or five years, and then after the Blackberry acquisition in March of 2019, I left a few months later, and kind of summer-ish going into fall of 2019, an adversarial research organization team published a public blog, "Cylance, I Kill You," one of the first known public inference attacks against artificial intelligence.
Cylance's antivirus capability was all built around artificial intelligence and machine learning to preempt the execution of malicious code using artificial intelligence capabilities. So again, do that for a number of years. AI-first company, and it's in essence, breached and published to the world, a universal bypass because the threat actors were able to, from the interactions with the model, do an inference attack.
Once they were able to infer the weights, do some reverse engineering and a few other things, they were able to basically create a bypass. And then publish it to the world. Now, the founders of HiddenLayer were the threat research and data science team charged with basically patching and mitigating that risk to protect, BlackBerry's investment in Cylance, BlackBerry, you know, Cylance's revenue and then protect its customers.
So they spent months dealing with that and then once they get done with that, they started having this concept of and I was off on the sidelines cause I had already left, but I had been responsible for security and trust, including overseeing product security, all the controls and compliance and stuff like that.
And I, we always knew that I'd say the traditional controls didn't directly protect the models themselves from different things, but all the testing and validation and all the things that we could do, based upon being a very smart AI-first company and a security company, we thought were reasonable to mitigate the potential of attacking the models, even though we knew it was possible, but there was none that were really in the wild and it wasn't something that was occurring, but-
Paul Roberts: People weren't doing it.
Malcolm Harkins: Yeah. Have these things in place. So they then started having the concept of, if conceptually people understand EDR, endpoint detection and response, so why not AIDR? Because if we had a runtime protection capability for Cylance's machine learning capability, we would able been able to detect, disrupt, prevent that occurrence of that event that cost BlackBerry a significant amount of resources to mitigate.
And then, obviously it impacted customer trust and, some renewals and revenue. Incubate that, and then that became the concept for the company. We now have full AI security platforms. We've got, model scanners that can tell you if the model's been compromised in some ways, got embedded malicious code, malicious callouts, other things to give you that security development lifecycle assurance.
We've got runtime protection and both of these things work for both predictive AI, so traditionally AI, and all the generative stuff. And then we've got a automated red teaming capability that we've recently released. And a few other features and stuff like that. So a full platform that will again, span predictive AI, generative AI, whether your deployments are on-prem in the cloud, we can give you a holistic set of things.
Now, prior to my Cylance days, my life was relatively simple. I spent 24 years at Intel corporation, 10 years in various business and finance roles and some startup businesses. And then in late 2001, after CodeRed, Nimda, 9/11, I started running IT security and business continuity. And then over time put my arms around all of it to be Intel's chief security and privacy officer.
Paul Roberts: Very cool. Very cool. Yeah CodeRed, those were, that was a turning point right around the millennium.
Malcolm Harkins: Yeah, early days. Andy Grove was running Intel and like one of the- I don't have his book behind me, but Only the Paranoid Survivor is one of his books. He was one of those guys. And after 9/11, after Code Red and Nimda, he frankly started beating the crap out of some corporate officers, including the CIO, to deal with the availability risk issues, logically and physically, that had just in essence, change the risk perspective in corporate America. And the CIO called me up because of my knowledge of the business, my business acumen, my risk acumen to be able to shape the security efforts, which then again, over time as the whole cyber thing blew up, it became product security, all the traditional InfoSec, corporate emergency management. And a few other things, because to me, there are all intertwined risks between physical, logical, and the products and services that you create and provide to your customers.
Paul Roberts: It's a hugely relevant area and topic. You got your thumb on the heartbeat right now of IT and cybersecurity is all about figuring out the implications of AI and also, of course, leveraging AI for the good. So, HiddenLayer just released recently a whole report on looking at AI security risks. I was wondering, and you mentioned the inference attack six years ago that kind of gave birth to the company. Can you talk about some of the sort of shape and character of AI based threats and risks? You mentioned a couple of them here, malware, obviously, and supply chain attacks, but if you had to rank them, what are the big ones out there that organizations seem to be concerned about as they're looking to obviously embrace and leverage AI?
Malcolm Harkins: Yeah. Before I do that, I think it's also important, part of the threat report that we just released that's actually back behind my head over here. There was also some survey data that we did that I think, puts in perspective the state of the state, so to speak, around IT security and data scientists, because we had surveyed 250 director plus folks in, in those variety of roles.
And there was some very interesting data that came out of it. Getting to your point on achieving value and why people would want technology, we found that 89 percent of folks had indicated that the AI models, at least some of them that they have in production use in their organizations, are creating in essence, material benefit: their business critical. 100 percent of folks said models in production are significant to revenue generation. Now put on my risk cap, if I'm getting material benefit and it's compromised, it's material risk.
If I'm a public company, guess what? SEC has made it very clear, even though for years you should have done this, if you have a material incident or a material risk, you have public disclosure requirements in your financial filings, right? Your 10K for your broad systemic, macro risks, your 8K for the incident driven risk that could have a material impact on your business and thus your investors need to know. So people are getting material benefit, there's material risk.
The other thing that was interesting is risk is rising. So what we found in the survey, and this is also tied to some of the technical details in the survey, is that 76 percent of folks have indicated they've already had some level of, we'll say AI related security or breach incident at some level. 45 percent of those incidents have come from malware that was embedded in models that they downloaded from public repositories.
Paul Roberts: Wow.
Malcolm Harkins: And then there was 32, I think, or 33 percent that have indicated a chatbot internally or externally has already experienced some, in essence, poisoning or, injections and attempts at manipulating those chatbots.
So those are, being at least self reported aspects of things. The other thing that gets back to, again, some of the dynamics going on the, everybody's stood up, not everybody, but most organizations of decent side has stood up some type of AI committee the past couple of years since ChatGPT has gotten lost.
And one would think a few years down the road, people would actually be able to have a coherent, not only strategy, but ways to make sure that they're managing the risk and capturing the value a few years into it. But we found that again, vast majority of folks, over 70 some percent, had indicated there's still an internal debate as to who owns the controls and security around AI. Is it the data science teams, is it IT, is it security? That's also consistent with the International Association of Privacy Professionals found on a study they did last year. So I'd say all this stuff around governance- at this point, I'll be really negative, is governance because somebody says it is because you've got a committee, but underneath the committee, the decision making roles, responsibilities and accountability is all mucked up.
It's again, a bit of an overstatement, but I think it is somewhat true. Now, the positive side of it is we also found that 99 percent of folks said securing artificial intelligence is a high priority, and 95 percent are increasing their budgets to do so. It's material risk, risks are rising, we've got governance and decision-making accountability challenges.
But there's a big priority on addressing it and increasing the spending to manage the risks appropriately. Now I'll stop there and then we can walk into some of the other technical vulnerabilities and things that we've found.
Paul Roberts: Yeah. It's interesting. And it feels like early days for enterprise adoption of AI. I don't know, obviously your company's working with enterprises on this, where are most companies? Is it right to think of it as early days? Or are we actually further down the road than you might think?
Malcolm Harkins: I think it depends upon, if we talk about AI broadly, I'll go back into my Intel days, circa 2006.
I also, while I was chief information security officer, I also ran all the enterprise applications at Intel. Under that enterprise apps team, there was a nascent data science team. That was almost 20 years ago. So AI in some ways has been around for decades and we all know that it's just been more of the predictive, artificial intelligence capability.
You look at the cybersecurity industry, Cylance launched in 2012. One of the first companies to use AI to improve security- that happened 12 years ago, right? So I think there's aspects of AI that's been around for a long time. But I think just like with Cylance, even that predictive stuff that's been there, people don't realize how exposed the models are because the attacks weren't there that were visible.
Now you could argue if you don't have any capability to detect it, you don't know that it's there potentially getting subverted, right? Now, the thing that changed everything was the launch of ChatGPT. All of a sudden everybody's like, Oh my God, AI and all these things. And so I think that was a massive step in the evolution of AI, the capabilities for it and the potential to create broad economic and social and business benefit, and that's the Pandora's Box that's been opened. That's the race that has been started. That is the opportunity we have for that promise of potential of AI. And I think people are in that journey. Some of them are early. Some of them have been doing it since the releases. And I think that's what's changed the game now where you get to the technology cycle, and I had this dialogue a week or so ago when I was doing a threat report dinner, because we had just released the threat report and people are talking about how security always lags and stuff like that. And I'm like that's true. And we've seen it every time. HiddenLayer was launched several months before ChatGPT got launched.
How often has that ever occurred where a security capability meant to protect the emerging tech is there before the tech launches? I can't think of it ever occurring other than I can look at HiddenLayer and say, we were in front of the curve. We're at the forefront of the research. We're at the forefront of the protection.
And then the question just becomes the reality within organization to organization, their adoption cycles, their perspectives on risks and how they are progressing along that, but we're seeing strong customer demand in a wide variety of areas. And we've even launched internationally because we're getting pulled by, customers in different geographies outside of the U.S..
So I think, the potential to close the risk versus technology gap is there, and it's just really going to be dependent upon organization by organization, their risk appetite and what they're doing with AI and how they believe they need to manage and mitigate those risks.
Paul Roberts: And can you just talk about some of the types of stuff you do for your customers? You've got a product, there is a consultative piece of it as well, but you've got a standard platform.
Malcolm Harkins: So there's a few things. So we do have a services team and they do a wide range of engagements with folks. Some of it is just adversarial AI training.
Some of it is manual pen testing. Some of it is other risk assessments. Things like that and even incident response, right? Because the forensics, what you do, how you do it, how you, quote unquote, it's different to patch a vulnerability on an operating system or an application or device than it is to patch a model.
We know that from the Cylance experience. Massively different set of things that you need to do for remediation as well as the technical forensic stuff. But there's three primary products in the platform. One is a model scanner, think of what we do for vulnerability scanning, right?
But in this case, specific to the file types, the model types for predictive as well as generative AI, where you can go, is the model compromised in some way, or again, getting to that 45 percent of people have experienced malware in models they've downloaded from public repositories, right? Your existing scanners do not function there.
They don't work for that. They weren't built to. And so that's a purpose built capability that allows you assurance that the model is in essence uncompromised in those ways. And then we've got a runtime protection. So again, think EDR, but for AI, AIDR, again, predictive and generative, how do you create the looking for all the different attacks, whether it be different prompt injections, attempts at subversion, the equivalent jailbreaking, all that stuff. So there's a wide variety of attacks. There's behaviors. There's, things in the model over time. But there's the interaction with the entity interacting with the model. And a lot of people don't realize this, but it doesn't have to be a privileged user.
It doesn't have to be a data scientist. It doesn't have to be somebody who's in the data engineering team. It doesn't have to be something that is of that nature. It could be, Malcolm is a general user of an AI capability that is doing the inference attack, stealing the model, other injections meant to dump data from the model, backdooring it in some way, and then you could use the backdoor for a variety of things, distribution of other malicious code to impact others, you could backdoor the model in a way that, you know, one of the interesting research things... let's see, I'll geek out on this, one of our crazy researchers, super smooth, super smart guy, he had found a way that with computational graphs that are embedded in some AI models used for image recognition and real time object detection, that he could backdoor them in undetectable ways. So you might have all the things that you'd normally have in traditional controls, he could backdoor the models.
So this backdoor, let's say on the image recognition, you've trained a model and it's meant to recognize an image, a German Shepherd, doesn't matter size, age, color scheme. It's German Shepherd. Has high efficacy, German Shepherds are always classified as German Shepherds. Backdoor it and guess what? We can make it say it's a Pomeranian.
Paul Roberts: That would be devastating.
Malcolm Harkins: It could be in the wrong context. Now, the other thing that he was able to do with that same kind of backdoor technique is I'll give you an example of where it could be, it's real time object detection and he was able to hold up a mug or it could be as simple as a small pixel, I could have a button here that's blue.
And the AI that's processing Malcolm in the real time video stream that says that's Malcolm says it's something else or Malcolm no longer exists because of the backdoor. Now, again, you think about object detection, real time image. Now you go to customer and border patrol, right? And where we use, whether it's in shipping logistics or let's say it's customer border patrol is perfect.
Cause everybody had gone on planes and done stuff and you go, okay, the scanners. People are looking at comparing images. Is there a bomb? Is there a gun? Besides the people, there's machinery behind that. You have people walking across the border. Okay, is that, Malcolm who's the terrorist?
Is it Malcolm who's organized crime? Or is it Malcolm who's Chief Security and Trust Officer from HiddenLayer, right? But again, imagine you could backdoor those things at the border, what would happen to drug smuggling, weapons smuggling, other organized crime, child trafficking, terrorism, because you've potentially backdoored models.
Paul Roberts: Physical safety stuff too. Are the balusters up and preventing access? Oh, yes, they are.
Malcolm Harkins: Or even, you look at what people do in warehouses, right? There's a lot of image recognition for barcode scans. And other things like that, safety mechanisms that are doing it.
So there's, again, I think, real world implications of that vulnerability and exposure to businesses who rely and organizations who rely on it, either for public safety or for business operations. If you go back years ago, it was done differently. I'm sure they're doing some stuff go back years and years ago.
I'm in graduate school. My first chip company that I worked for before Intel was guess what? Frito Lay. You know how they spotted chips that they had a quality issue with. They had scanners as the Lays chips were coming through out of the hoppers and extruders onto the conveyor belts.
And they blew them out, right? And say, Hey, this has a burnt piece or whatever. Again, that was a visual scanner. That was 35 years ago. I'm sure that's been upgraded and probably has potential upgrades where AI is now used to enhance the quality of the image recognition spot that just like you're seeing in other industries, those type of scanning capabilities. Again, there's in these image areas, whether it be ultralytics, which is certain aspects of models that we found vulnerabilities in as well, plus the computational graphs. In the healthcare area, there are emerging capabilities, even with vision language models and stuff like that, where you go, how do I, if I'm a healthcare organization, and I've got 30,000 healthcare patients, and I do 500 X-rays or MRIs a day, and I've got 20 staff doing it, but the region's growing, I could use AI and some of these type of things to improve the efficacy and efficiency of the reading of the radiological results or the MRI. And then I could potentially also use a large language model vision, VLMs, to then start writing a draft report that then a technician and a radiological expert would review and validate.
But again, you go, boy, that's great. It would be great for medical advancements of spotting breast cancer earlier. Looking at that hairline fracture and going, boy, that could be a real problem. But again, if you could backdoor those models, and, or subvert the models or worse, ransom the models.
Why take out the entire imaging station where it might be easier to just basically take out the imaging process improvements that have been used by AI and then ransom you for that? So I think there's a wide variety of attack scenarios that again, have different implications, depending upon-
Paul Roberts: Haven't seen that yet, but to our knowledge, but that doesn't mean it hasn't happened of course.
Malcolm Harkins: No, to your point. One of the things that we also found in our survey, and this is I'd say disappointing, but it gets back to the governance thing. We found self reported from the folks we surveyed, 45 percent of organizations have not disclosed an AI related incident because of fear of public backlash.
Almost half of the organizations who've seen or experienced an AI related issue have not publicly disclosed it. Now the question becomes how many of them were needed to be publicly disclosed. Sometimes maybe they didn't, but I would argue in 250 samples and where there's personal information, where there's material potential impact to investors and shareholders.
That to me was startling that people are not reporting what probably should be reported. In some cases, publicly to either investors or consumers or customers that may have been impacted by a breach.
Paul Roberts: So let's dig in a little bit, because obviously, as ReversingLabs has done some research on, for example, Hugging Face and the pickles, it's a big kind of gap in the pickle scan application. What's really interesting to me about this is, and I started covering cybersecurity in 2002. But there's usually is, like you said, a gap between people speculating about the cybersecurity impacts of some technology once it's adopted and so on and actually seeing those things. But that gap has disappeared with AI, right? And so when we're talking about the sort of cyber physical consequences of broken or corrupted or, compromised AI models, it's not hypothetical. We can, like you said, there's probably already technology out there that's leveraging this and we could see real world consequences immediately.
One of the sort of discouraging things is it does seem like we're repeating some of the same mistakes that we made back in the 90s and early 2000s around prioritizing, features and feature development and cool new stuff over, security, integrity, quality. The pickle scan thing was a piece of that, Hugging Face has been, forthright and saying pickle files are really not secure. You really shouldn't use them. And yet they're still hugely popular. So I guess, how do we not repeat the mistakes of the past to learn the lessons of the past as the saying goes, and not end up in the same place that we are now with, traditional application security, which is a really bad place?
Malcolm Harkins: Yeah, the reality is you can't eliminate risk, right? Physically, logically in the financial market. I'm a former finance person. There's no risk free anything. So let's just be blunt on that and realize.
Paul Roberts: Walking across the street, right?
Malcolm Harkins: So I think it really becomes one of the acumen of the security team, the acumen of the data science team. And frankly, the accountability and decision making of the management team of an organization. Those things in combination will say whether or not we will continue the sins of the past. Now, I've been in the tech industry since 1992. So a long time, I think, going back then there was broadening ignorance, right?
Across everybody, around the realities, because they were ankle biter issues. Okay. I Love You Virus, annoying type things.
Paul Roberts: Blows up your email server, but.
Malcolm Harkins: The people that would see the fraud, retail and banking, they would manage it within their tolerances and limits. I was in the retail credit industry in the eighties, how we maximized revenue when I was running may department stores, Southern California credit operations, a few million credit cards, half of the company's revenue over a billion dollars a year. We maximized revenue by 2 percent bad debt. Half was people couldn't pay and half was fraud, but that's where we maximized the net income and revenue was basically allowing some level of fraud and bad debt to occur. So there's a reality is sometimes of maximization. That means you're actually intending to take a risk and have an impact, right? I think that's true across every business segment, right? And every-
Paul Roberts: This will be abused and we just have to recognize that.
Malcolm Harkins: Yeah, exactly. And then the question just becomes how do you manage the tolerances of those risk conditions relative to the value? So value versus risk. Now, the thing that I think has happened for a long time is a lot of organizations, they'll focus on risk to self, not risk to others.
And you go, okay it's going to cost me X hundreds of thousands of dollars to delay shipping this thing. And it's going to cost me time to market. And I ship it anyways. Ford when it shipped the Pinto for several years. So we've done this and other things beyond the tech sector.
And so the question just becomes when you're evaluating your risk, I'm a former finance person, I've always set design goals, and this is the thing that I've stressed for a long time with peers, I don't think organizations set good security design goals. They set a goal like, I want to be compliant with this. Okay great. That might mean you can sell stuff on credit cards because you met PCI compliance. It doesn't mean you actually manage the risk conditions that could impact you or your customers. It just meant that you managed to a standard that allowed you to, be able to use credit cards, two different things. Or people will go I want to hit a cybersecurity maturity level of 3.2. I don't know what the hell that actually means. And while those things are good, again, a former finance person, a business has business goals, revenue, gross margin, net income. I'll just stick to three of them, right? What's the equivalent of that for your cybersecurity efforts? For me, design goal number one, when I landed running IT security and business continuity in late 2001 and Intel was no material or significant event.
Paul Roberts: Material to the company.
Malcolm Harkins: Exactly what Andy Grove, the board, the shareholders frankly gave a crap about. You're always going to have operational risk stuff, but how do you manage your cyber risk so that there's a near zero potential for a material or significant event? Got to Cylance. Guess what? Same design goal. Other than we added one that only a nation state actor should ever be able to get in, they have to work for it. So what's the equivalent of your fire safe rating? So if you buy a safe at Costco or whatever, there's a rating. How long were your contents last at 400 degrees Fahrenheit, right? For me, those are the type of goals that should be set and managed to. The by product then would be your compliant. And, you hit all these other goals, but you go, what is the design goal that your organization has for cybersecurity? And I asked this over and over again, and people always give me compliant and manage the risk, safeguard the assets, maturity. And I'm like, I want it black and white, right? Revenue, net income, gross margin.
Now, if companies did that, then I think by and large the dynamic of how we manage risk would be different. And then even the new technologies, whether it be AI or whatever comes in the future, still the same design goal, right? And if you have those, then, because again, particularly if you're a public company you go, okay, if that's your design goal, then it doesn't matter whether it's AI or anything else.
That's the design goal. And if you have a decent enough potential for material impact, then invest to fix it or disclose it in your financial filings and with your investors and stuff like that. And for me, if we changed the behaviors at that level and had almost the equivalent of a Sarbanes Oxley for cybersecurity, right?
You have to attest to your state of controls, not only for the technology you're creating, but the technology you're using, that to the best of your abilities you are managing and mitigating material impact to your shareholders, your customers and potentially to society, then it will calibrate those things, right?
Paul Roberts: And we've seen that, more or less, we've seen the SEC adopt your standard from the early 90s as their standard, which is, is it a material breach? And if it is, you need to disclose it. They've been saying that for a while, but they recently, within the past few years, started putting money where their mouth is.
Malcolm Harkins: I'll give you an example. It's a shame that they had to do that. Yeah, go back in my lineage. February of 2010, Intel in its financial statements, disclosed a cyber incident in 2010. Yeah. Why? Former finance guy who also is responsible for all the system Sarbanes Oxley stuff. I knew when we were midway into an investigation of an advanced persistent threat that it had potential material impact to the company, and I went to the general counsel, we then read in the head of SEC law, and we got outside counsel specializing in disclosures in financial statements as while we were doing the technical investigation to bring them up to speed and then help us determine if we had the obligation to disclose this in our financial filings and the answer was yes, and we did, right? That was 2010. What it took to what a year and a half ago or, and you go, why is that? It's because a lot of organizations were sweeping that stuff under the rug. They would do a breach notification if it was privacy related, because there was already laws in the books to do that.
But were they disclosing they had a significant deficiency or material weakness in their IT systems that could impact the company and thus impact investors? No, right? And they weren't disclosing intellectual property breaches, right? Or in some cases, ransomware issues. And I go, if you paid a ransom, the only reason why you'd pay a ransom is because it was potentially materially impacting if you didn't. Right?
Paul Roberts: And the problem for us, for our society and our economy, of course, is that with software and services, your company is not an island, right? Like your customers and downstream companies that you supply your technology to rely on your technology to do what they're doing. So the ripple effects of a breach or a material incident go well beyond your company.
Malcolm Harkins: I totally agree. And even, we really released in our threat report, there's a set of recommendations in there. Some of them was looking at, again, the explosion of AI and everything, every technology company, even non technology companies are starting to embed and leverage AI. So that means now your third and fourth party risk items need to be managed, right? And so evolve your third party risk questions. And even part of the dialogues I've had with people on this they go to the traditional IT vendors, right? They've been embedded AI and a sales CRM solution, or video conferencing solution or an HR solution, whatever.
I'm like, great, knock yourself out, ask all them questions. Where did the model come from? Did you scan it for these things? How are you providing runtime protection? What are your incident response processes? How are you validating the integrity of the output? All those, things. But the other thing I said is go to other service providers.
And when I have this as a round table dialogue, you know, a few assistants are looking at me and they're like, you're lawyers, you're financial accountants. There's AI solutions that they're developing or utilizing to basically write a patent filing or a brief for an HR lawsuit or whatever.
I'm like, look, and I pull up startups that are targeting those areas. And I'm like, if those models that they're either doing, or they're procuring from somebody else, get breached, you're exposed. So ask even your non-IT providers how they may be utilizing AI and how it could impact your business.
I'll give you a good example of one. Harvard a year and some ago released a business case study, Thrupps, the steel company that also makes elevators. They published a case study, Harvard Business Review, that they were utilizing AI in elevators to improve people movement.
And they were seeing 30 percent efficiency in people moving up and down via elevators and high rises, right? Okay. My facilities is using AI, right? I might be in a leased building. I might have my own buildings, but have a facility provider who's providing technical solutions for security or elevators and stuff like that, right?
You go ask them. They're a non traditional IT provider, but a base of what they're doing has technology in it to provide you facility services.
Paul Roberts: How about your automobile? Accident prevention and lane assist features, yeah, absolutely. So one really interesting report you all did recently was on the, so one of the, one of the conversation threads that's going on in the AI space now is between this sort of North American proprietary AI models and the very vibrant community of open source AI, exhibit number one is DeepSeek, which is a Chinese AI startup. You did a really interesting report on DeepSeek when it first came out. It was, Oh my God, it's just as powerful as open AI. It's a fraction of the cost. It's. Blah, blah, blah, blah. Everybody downloaded the app.
But you guys had a really interesting report on it. Maybe just sum up what the findings were for DeepSeek AI.
Malcolm Harkins: Yeah. I think there's, obviously it was again, an eyeopening moment for everybody. And I think, it was oh crap. You saw Nvidia stock drop and stuff like that.
And it's okay boy, the cost of compute now, again, there's some arguments around whether or not their articulation of the economics was totally accurate. Let's just say that it's, it was reasonably, there's-
Paul Roberts: Always a question, right?
Malcolm Harkins: There's always a question, but let's just say it was reasonably accurate because they are using cheaper compute for some of the stuff. Like we've seen with everything else, the the work that people have done from the past paves the way for people to do things in the future, right? To give you a different analogy, go back, what, 20 years ago, mapping the human genome the first time took 10 years and a couple billion dollars, right? I could get a DNA test and all that stuff for what? A hundred bucks and get it turned around in a week, right?
So there's always this, you're building upon the, what other people have done, which makes your stuff faster, better, and cheaper potentially than theirs. So let's just assume that there's some dynamic of that is what occurred. But the bigger issue with things like DeepSeek and frankly, it could be other countries as well depending upon the social and legal and political things.
One, we found within hours when we started playing with it, that if you put into DeepSeek something like Tiananmen Square depending upon the language you used yeah, let's just say the realities that were on the ground that day were not accurately given back to you. And the Chinese government, again, I not, no knock on them per se, their own independent country with their own political party and beliefs.
They stated years ago, and they codified policies. That the AI in China will follow the doctrine and perspectives of the government, right? As a level of social and political control and stuff like that. Some could say manipulation. I'm sure they don't, but they look at it and say, Hey, we want to keep things within the way in which we want China to operate and run. Their right. They're doing it. Guess what? It was developed there. You're seeing aspects of that intentional bias that was built into the model. Why? Because that was the law, right? Okay. Proliferate that wherever it goes. And guess what? You can start also altering history, right?
Think of that. Let's just take DeepSeek and extend this out. If we weren't, didn't take actions to mitigate things like that. A generation or two from now, my kids might actually believe that Tiananmen Square didn't happen and somebody didn't stand in front of a tank. Why? Because the things that were used, that then just kept getting seeded with that, in essence erasure of fact, it's proliferated. So that's, I think, a big risk that we have when we start looking at models and who's created them and what their countries or their values are creating, intentionally bias or manipulation of certain things.
That's certainly one thing. We found it's a concern for a variety of reasons.
Paul Roberts: There was also this very interesting chain of thought feature that your researchers found out how to manipulate it in a couple different ways. One would just be to consume credits and burn through your wallet quicker than you might otherwise.
But there was also some really interesting stuff, including some suggestions that maybe this AI model was trained on OpenAI or Microsoft data that came out of that chain of thought. So talk about what is this chain of thought approach and what does it reveal from an attacker's standpoint, how can that be manipulated?
Malcolm Harkins: Yeah, let me give you another example. Cause you did potentially, identify the misappropriation of intellectual property, that could have and some of that is there. Again, it was about a year and a half ago, OpenAI suspended ByteDance's account, parent company of TikTok. Why? Because they were basically stealing, I think it was ChatGPT 3.5 or 3.0, one of the models. So again, that is occurring. And that's where you have to really start thinking about those things and really understanding again, how are you protecting the model? By looking at the behaviors of what an entity is doing to potentially manipulate it, let alone do an inference attack, like what happened to Cylance, but in this way, a different way, or, do some things that again would be, impacting, of the millions or hundreds of millions of dollars you spent on the intellectual property.
And so I think again, this become the chaining of leveraging other people's reasonings, leveraging other people's models. And this is where, again, if you look at it and even look at what the Trump administration's doing with its executive order, wanting to provide assurance and encouragement for AI in the United States to be the dominant AI in the world.
If we're do that and we're, investing tens of millions, a hundred millions of dollars or billions, and that's able to be taken, again, it has impact on the competitiveness of the United States and again other implications. But again, getting back to this chain of thought thing, it's how do train differently, reflecting on the inputs and identifying and potentially, adding flawed reasoning and hallucination. So there's all these other implications you can start doing with that, that again, let's just say you provided yourself up assurance that the initial model looked right, can you use this chain of thought type stuff that if you don't have the runtime protection, you can still start tilting and doing all these other, things that have, implications either on a business process, on society or whatever, again, depending upon the condition of what you're doing in the context of the model itself.
Paul Roberts: And as you say in the report, companies just need to be very mindful of that as they're, implementing this technology within their own organizations of what certain types of prompts might reveal and what they are comfortable having revealed or not. As they're feeding these models. Really interesting. It's a super interesting report. People should definitely check it out.
Right now, modern enterprises have a whole range of security technologies unless they are HiddenLayer customers- most of it is not really tuned around AI based risks and threats.
So I guess what would you say about the level of preparedness or awareness of organizations, enterprises that are, again, rapidly embracing AI technology to actually address some of these threats and risks that we're talking about?
Malcolm Harkins: So I'm bipolar on this. I'm a glass half full, glass half empty type person.
I think there has been an, and to some extent in some organization continues to be a, misunderstanding of what controls do what in certain contexts. Now I wrote an opinion piece kind of paper that was posted on the RSA conference site a couple of months ago that basically said traditional controls do not protect AI.
I know this from my days, even at Intel. I know it from the experience of Cylance. I know it, so I don't care if you had EDR, NDR, SAS, DAST. All the traditional controls. I actually have done a control matrix that looks at, let's say the top 25 enterprise security controls on the application side, the infrastructure side, your SIEMs, your SOARs, your scanners, your threat intel and all that stuff.
And guess what? You can do a strength of present control analysis and say, here's the attacks against an AI model. Is it red? Meaning it's not designed to manage and mitigate the risk. Is it yellow, meaning it provides indirect protection or partial coverage under some circumstances or green designed to control for the risk?
You do that across all the traditional things we've spent our, $ 226 billion of InfoSec spend annually on. And guess what? It's a sea of red with a little bit of yellow, and again, I'll give you an example, web application firewall. Do you think it's going to stop a prompt injection?
No, but are some WAFs extending to do that? Yes. But are they going to catch what I'll say the simple commodity ones? Yes. Why? Because the WAF wasn't built to protect the AI model, right?
Just like a WAF doesn't basically do a good job of really providing web application security. It only provides a limited aspect of web application security because the attack is actually occurring in the browser, right?
In most cases, right? And so again, you have to look at your existing stack, look at the context of the model, look at is it internal facing, external facing, and even if it's internal only, this is a dialogue I had a week or so ago. And as I was having this debate with the peer, they were in flight with looking at us, but they're like, yeah we don't think we need to deal with it.
It's the AI models used internally. Okay, but it's used internally for material benefit for the company. Yeah. Okay. So you're going to rely on all these infrastructure controls and existing security mechanisms to provide indirect protection of the model. They're yeah, but Malcolm, it's an internal deployment and I'm like, okay, pause for a second.
Do you do enhance database security for databases that are highly sensitive and have a lot of data? Yes. Do you do enhance security for your factory automation systems? Because if they go down or got ransomed, you're screwed and you're losing output revenue. Yes. Why? It's on an internal network.
The only thing that could impact them is an internal system. And your internal systems are protected with advanced endpoints, and firewalls, and scanners, and all that other stuff, so you're good, right? They're like no. And I'm like why do you think it's- you know if again, it's the model is used for material benefit, just like a factory system is used for material benefit, a sensitive database has material impact, and you're doing enhanced controls there, and the model you've got on an internal network is used for material benefit- you're telling me you don't need enhanced controls to protect the model? That makes no sense. And by the way, where'd you get the model? We, foundational model. Oh, you got it from a public repository to start with. Yeah. Okay. Guess what? How many times does a PDF get scanned or any attachment gets scanned before it opens up on your network?
Probably a half a dozen times. At the gateway, at every network bit and byte. Your client firewall and pre execution, that thing is scanned several times and it's a file that's coming in, but you're going to go get a model from a public repository and then build on it for material benefit. And you're not even scanning the model to see if it has embedded malicious things that could be used for a lateral movement in your organization, let alone.
And then I basically decimate them like. You got your head up your ass. I'm sorry. Go tell your management what controls do what and what they don't do and have a real dialogue around risk, value, and sufficiency of control.
Paul Roberts: It seems like there is this kind of mental game people are playing where they think that because it's AI, we have turned a page from the, very messy page of application security and, software supply chain security, and suddenly it's this green fields and everything's fine. We can just play around with it and use it.
Malcolm Harkins: Or there's another dynamic that you, and you could argue it's reasonable, I don't see, the news headline that says this AI was taken down and my business is on its knees. I don't see, so there's a little bit of a chicken and egg thing here that sometimes can go on, but I also look at it and I go, if you don't have the sensors meant to detect it, you don't know that it's not occurring.
And even, go back 10, 15 years ago, right? We were all worried about dwell time, why? AV was going to crap, the scanners and other things we had in our environment weren't great. And the dwell time was hundreds of days. We upgraded all that stuff and the dwell time is strong. Guess where the new dwell time is that's going to create even in a low risk model, when low impact that I have a long dwell time, the potential for that model to be used as a pivot point of exploitability.
For long term exposure to your organization, your customers is enormous.
Paul Roberts: Okay, so one thing, you mentioned this whole elevator example. One of the things that you've talked about a lot is a sort of shadow AI problem, right? Which we already know about the shadow IT problem.
Is this just a extension of the shadow IT problem, or is there something different? What is shadow AI and what should enterprises know about it?
Malcolm Harkins: Yeah, shadow AI is where are your users or your business goes and grabs AI models or uses AI models. And, in some cases without your knowledge or without your approval.
Now, I think some portions of, I'll say corporate America created this problem because when ChatGPT got launched, a lot of organizations, I have this debate with a lot of people. They banned it. You can't go touch all these things. And I'm like, that's the ineffective. Your user is going to go around you.
And so you now become blind to what they're doing. You become blind to the potential use cases. So for me, I've always been a run to the riskiest thing in order to shape the paths of the risk. And then, because that, and I think if the security team has that mentality and the companies themselves have a controlled mentality around.
Sometimes you really need to block stuff, but it needs to be tactical and temporary because you run the risk of making yourself blind and making people go around now. And then I think that exacerbates the shadow it or shadow AI problem. But I look at it and I go, again, I think it's a reality.
And I think the it teams, the corporations and the security team needs to embrace it, needs to encourage it. And then have the boundaries. If people go out of bounds, have a way to spot it, have a way to bring them back into bounds and have a way to do disciplinary actions. If it's egregious enough, and they really ignored things that they shouldn't have.
Paul Roberts: I feel like a lot of that was driven by fear of IP theft and that type of thing, rather than, we don't want you using ChatGPT to write this email. It was more about what are you feeding to this model? And are jeopardizing our business?
Malcolm Harkins: Yeah, but I have this theory and I started it again a couple decades ago. So go back to the early days before the early days of social media. I, again, former finance guy, economics background. I had this theory that I called my roundabout theory. If you look at the transportation economics, a roundabout is more efficient and effective for traffic because people move, they burn less fuel and guess what, even when there's collisions, cause you can't eliminate risks, you can't eliminate accidents, the damage is generally less.
I have my roundabout theory for social media, and I think the same thing is true for ChatGPT and the launch and all that stuff. You go shape the path of the user, make them risk aware. When you go into a roundabout, once you get past the infant mortality of people who've never driven in one before, they become risk aware.
They slow down, they keep moving, the risk, you go, okay what I would have done was continue that theory and say. Okay, let's just say I create a screen that if you're going to any one of these open models and using it, it pops up and says, you're now, going to ChatGPT, or it could be any other one.
Do not disclose personal information. Don't disclose intellectual property. We are trusting in you, but if you make a mistake, you'll be going to be held accountable. They go, ah, okay, great. I was going to go use it. I'm an annual review cycle. I was going to go write it or use it to write a review. Maybe I shouldn't put, tell me how to give Malcolm Harkin's coaching on his sometimes aggressive behavior and managing risk and his language, right?
You wouldn't want to put that, but you could say in ChatGPT write a paragraph for me on how to give coaching to an employee whose language can sometimes be abrasive and seem pushy. Okay, great. Guess what? Because I interrupted Malcolm and made him think about it before he put somebody's name in it to write a review with the context of who he's working for and everything else.
I can go, Oh, great. Now he gets the value of it. He averted the issue and everybody wins. There were simple things people could have done rather than block it to knock down probably 95 percent of the risk. Now, if you have a user who's just frankly stupid or doesn't care, fire them. But at least you've got the warning and they say, yes, I understand.
And then if you find that they violated it. You now have, you've given them warning and they've ignored it. And I would use that do performance actions against them.
Paul Roberts: Okay. Final question. So we talked a lot about the cyber risks around artificial intelligence. I'm interested in your thoughts on the benefits for organizations, particularly in regard to not generally, but in regard to cybersecurity, how you see AI maybe being leveraged to improve our ability to defend against threats and attacks.
Because obviously, as you read the headlines, companies are struggling. Attacks are very sophisticated. There are a lot of them. Where do you see AI boosting cybersecurity preparedness?
Malcolm Harkins: Yeah I think it has enormous potential, cause I've always believed that, the more you can automate things, the more you can reduce costs.
And then the question just becomes just like whether it's in the sales process, imaging, like we were talking about before cybersecurity, you go, where can I use predictive AI to improve the efficiency and effectiveness like Cylance was doing, a dozen years ago and CrowdStrike and SentinelOne and a bunch of other players embedded AI in their solutions to improve the efficiency and effectiveness of control.
Great, continue to do that. Now with LLMs, how do I, and particularly agentic AI, how do I potentially take the sea of vulnerabilities, the sea of alerts, the reporting and all that stuff and start potentially using LLMs to again, drive a level of efficiency and effectiveness of control. I think the potential to do that's enormous.
And then the question just becomes, what are those use cases? Who's selling those solutions, or how do I develop one myself using a foundation model? But if you go get a foundation model, make sure you scan the damn thing. But again, I wear a risk cap.
Security solutions are privileged solutions that when compromised will absolutely create material impact. So, if you're going to go down that path, you need to ask every one of your security vendors, including the ones you've got today, that have predictive AI: Where do their model come from? How have they validated the model is not tampered with or has embedded malicious things? How are they going to do incident response? Because I look at the Cylance incident, right? That was incredibly impacting to Cylance, to BlackBerry, and it's customers because they were exposed and while we were doing a lot of things and all the things we could have or should have done again, that was a problem we can learn from that.
It's a case study in the MITRE Atlas framework, that again, for people that are familiar with MITRE Atlas is the AI attack framework. I would go leverage that stuff. I would leverage it for benefit, but then I'd also recognize those security solutions create a risk condition. And I would go figure out how the solution providers managing those risk conditions. And then how me as a user, again, just, look at even the impact that CrowdStrike had, last year.
Paul Roberts: It was a bad update. Yeah.
Malcolm Harkins: Yeah. Some of that was CrowdStrike's problem. The people that, my opinion, the organizations who suffered more dramatic problems because of it, they weren't managing their side of the risk side of it.
We should always have assumed that a bit or a piece of code could corrupt and take things down. Business continuity disaster recovery processes in some organizations wasn't done right, and they had catastrophic issues. Even a third party, you can't eliminate risk. So you have to then still be responsible and co-responsible in many cases for incident response should a problem occur.
And people don't realize that when it comes to security tools. And I think the CrowdStrike issue was a clear example of that for some organizations that didn't do their job to manage the risk of what could have always occurred.
Paul Roberts: Right? And again, that was merely a flawed update. It was not a malicious compromise.
But your EDR client is the most powerful Trojan in the world. I mean-
Malcolm Harkins: McAfee had a bad incident in 2010. I had 26,562 systems bricked in seconds because of it. And guess what? Intel had no material impact. Why? Because that was the design goal. I'll go back to my design goal.
No material significant event. And guess what? I had something that one could argue was the equivalent of a ransomware event. And it took out a quarter of the environment of office workers and no material impact. Why? Because that was my job.
Paul Roberts: Really important point to wrap up on. And before we do Malcolm, is there anything I didn't ask you that I should have, or something you wanted to say that I can give you a chance to say?
Malcolm Harkins: I think, goes through the whole thing that we were talking about and I'll just have a closing comment. Again, there's a psychology of risk, right? That, I think affects decision makers and affects the economics and all that stuff. But I've always had this view, and I wrote it quote in a couple of the books I published on managing risk and information security: "Risk surrounds and envelops us, without understanding it, we risk everything. Without capitalizing on it, we gain nothing." And I think that's exactly at the heart of this stuff. How do we capitalize on these perceived risky things? Because we understand the risk, and then we take action to appropriately manage it, given the context of the impact and the blast radius.
Paul Roberts: Malcolm Harkins, HiddenLayer, thank you so much for coming on and speaking to us on ConversingLabs Podcast, it's been a pleasure and we'd love to have you back on.
Malcolm Harkins: Anytime, thanks Paul.
