Post by @tinker@infosec.exchange

Well, genAI has finally arrived at my job's doorstep.

fuck.

I test computer systems to see if they can be hacked.

I do this by actually hacking into them.

I am NOT being asked to use AI tools to do my normal day to day job, specifically (though many in my field are doing that and advocating for those tools' use).

But.

I *am* being asked if our systems can be hacked by the current genAI hacking tools.

And the only way to determine that is by running those tools against our systems.

And, from a professional standpoint, I agree with this. We need to know if the current genAI tools can successfully allow a "hacker" (read script-kiddie... or AI-kiddie in this matter) to actually hack into our systems.

My role in my job has always been to become the "bad guy". I do what the bad guys do. And if the bad guys are using genAI hacking tools....

...then I have to use genAI hacking tools.

At least insofar as to see if they are effective.

I hate my job.

25 Replies

3 Boosts

John Francis 🇨🇦🦫🍁

@johnefrancis@cosocial.ca 42 min. ago

@tinker prompt-kiddie, or juvenile-sloperator

CMDR Yojimbosan 🅅⁂

@yojimbo@masto.hackers.town 1 hr. ago

@tinker I'm not quite in the same boat, but close.
Our threat modelling & risk conversations normally consider an authenticated user risky when compromised, or "disaffected", and these are rare. Now we need to add "account might be using an Agent, which unilaterally starts attacking our infrastructure just because it can't figure out how to do the job properly" (i.e. the Sydney Gym Booking problem)

CMDR Yojimbosan 🅅⁂

@yojimbo@masto.hackers.town 48 min. ago

@tinker Lemme follow up a little from that - $dayjob runs a B2B service, no online signup, customers' company identities are established and finance checks etc all happen before they get an account on the system. So usually we don't need to consider our system users as particularly malicious, especially compared to more open platforms.

The risk of misfiring agents ends up being a significant increase to our normal stance.

Tinker ☀️

@tinker@infosec.exchange 43 min. ago

@yojimbo - yeah makes sense

Tinker ☀️

@tinker@infosec.exchange 60 min. ago

@yojimbo - Yeah, a dear friend is on that side.... covering governance, risk, and compliance around the use of genAI. Trying to set up frameworks to ensure that internal employees that use genAI are not introducing vulnerabilities or risk into the environment.

None of these are real jobs. Hahahaha. What a weird convoluted made up world we've created.

CMDR Yojimbosan 🅅⁂

@yojimbo@masto.hackers.town 57 min. ago

@tinker I haven't found any way to talk about "the risks of AI" to individuals who are all sold on "but my productivity goes up! I can produce more output!". I haven't found a way to get them to see that copious output != valuable output.

However, you're right that "securing the IT stack" isn't a fun job any more if the IT stack itself is a the problem :-(

Tinker ☀️

@tinker@infosec.exchange 50 min. ago

@yojimbo - That is such a classic (and prevalent as you point out) problem.

Interestingly my friend is really good at that part, heh. She is able to dive in deep and basically say "fuck your ignorance, it is an issue, here's how we fix it. Get out of my way" - but in really tactful corporate speak that seems to work.

Blows my mind.

1 more reply

G1N&T

@g1n@infosec.exchange 1 hr. ago

@tinker same here, although we are actually being "incentivized" to build agents and use the tools. Feels like we're training our replacement (and we know the replacement sucks at the job)

Tinker ☀️

@tinker@infosec.exchange 1 hr. ago

@g1n - Yeah. I'm a bit lucky on this front.... at least for now... in that I both test AND help lead remediation.

So someone has to run the tools - and that would be me. So even if I train the replacement... I'll be the one to run that replacement.

And even then, I'll be the one to ensure that the vulnerability gets fixed and keep running the tool until it can't hack into the thing anymore.

But this was already a niche niche niche field... and this just makes it even smaller. So who knows.

G1N&T

@g1n@infosec.exchange 33 min. ago

@tinker yes, a narrow field getting narrower. I guess experienced and well established people will sort it out. Not so sure about the ones beginning their career or wanting to start it.

Tinker ☀️

@tinker@infosec.exchange 20 min. ago

@g1n - That's another issue with all of this. The adoption of genAI is absolutely causing issues with entry level positions, right?

In so many industries, genAI is replacing new hires / entry level and junior folks (whether this is justified or not, and whether there will be a meaningful swing back, etc).

But teams are cutting themselves off at the knees. Today's senior folks were yesterday's junior folks, of course. And without new people coming in, then there won't be senior folks down the line.

1 more reply

/usr/local/goth (new server)

@ladytel@masto.hackers.town 1 hr. ago

@tinker honestly this might be the rare instance in which I don't find this objectionable. Someone else said this but so far genai hacking has come out of the labs. I think they're full of it so I'd love to see some independent writing on the subject (even if you can't share because $JOB)

Tabby

@0x09@hachyderm.io 24 min. ago

@ladytel @tinker a friend did some work on this in a specifically Kubernetes context and presented it at KubeCon EU: https://www.youtube.com/watch?v=jgx4J1z7POM

View

Tinker ☀️

@tinker@infosec.exchange 19 min. ago

@0x09 - Oh, I'll take a look at that! Cheers!

One big thing for me is seeing the difference that only 8 months has made. I saw a demo of Claude hacking several months ago.... and it did not impress.

Now though... hmmm....

Tinker ☀️

@tinker@infosec.exchange 1 hr. ago

@ladytel - So far, genAI hacking tools are used in two main approaches:

1) Similar to a common automated vulnerability scan with follow up exploitation attempt. So very similar to running say... OpenVAS and then having it autoverify.

Neat. Point tool at target and say "hack me".

I'll compare the outcome against Nessus, Nexpose, Qualys, etc.

2) To assist experienced pentesters / hackers in a similar way that its used to "assist" code developers.

So running specific prompts at systems, interpreting the output, and then dialing it in.

I hear it's shit at 1) and amazing at 2).

Purrbiscat

@silver_buttercat@kolektiva.social 1 hr. ago

@tinker Grandma here is offering virtual condolence hugs. Such a screwed up situation. At least take comfort in the knowledge that you are using the evil tools to make sure the evil tools can't hurt you (or your employer as case may be).

Tinker ☀️

@tinker@infosec.exchange 1 hr. ago

@silver_buttercat - Yeah, that's always been my role. Be the bad guy in our back pocket.... do what the bad guys do BEFORE the actual bad guys do it.... so we can fix it so the bad guys can't do it.

It's always been an icky job, but I've never caused harm before by doing it.

Now, just by using the tools, I'm causing harm.

ness

@ness@ioc.exchange 2 hr. ago

@tinker im really curious though, because it feels like most of the info on this is coming from "ai" companies who of course arent neutral

Tinker ☀️

@tinker@infosec.exchange 1 hr. ago

@ness - I mean... exactly. So I'm being tasked with answering the question "Are they full of shit or this is a real threat?"

Bill, organizer of stuff

@wcbdata@vis.social 26 min. ago

@tinker @ness Both!

Tinker ☀️

@tinker@infosec.exchange 20 min. ago

@wcbdata - Hahaha.

Yup.

s00ner🌈

@jeffers00n@tiny.tilde.website 1 hr. ago

@tinker @ness it's also insanely difficult to test because the AI systems are nondeterministic. There's not guarantee that because I tried really hard to make the chatbot hack us and it failed, that it will also fail for the next guy trying the same thing. The money is better spent on hardening and detection than the hackbot slot machine. IMO.

Kallisti

@kallisti@infosec.exchange 1 hr. ago

@tinker @ness
On the bright side, you may end up being able to report "They are full of shit" and have the receipts to prove it

Tinker ☀️

@tinker@infosec.exchange 1 hr. ago

@kallisti @ness - That's how it was a year ago when I observed it.

Last month I spoke with a dear friend who is exceedingly more talented and experienced than me...

...and he says the current genAI hacking tools are amazing.

So. I don't think that's going to be result. But. I will find out. One way or another.

Gary Blosser

@zombie042@infosec.exchange 2 hr. ago

@tinker I'm pretty much in the same boat. I wish you luck with my pain... 😫

Discussion

Discussion