Automated Testing of Defences and Alerting

Yes, defenCe, not defenSe, I’m British darling.

When I was a SOC team leader (before SOAR existed) I tried to build automated processes to confirm technology and process worked as excpected. Even though I suffered from scalability, my aim was to test:

  • Was existing technology blocking known bad as designed?
  • Were alerts being raised to my Analysts?
  • Was the team reacting quick enough?

Technology Configuration Testing

Over time, policies and allow/block lists get abused by inexperienced staff making unsafe/incorrect changes.

(I once saw “allow encrypted PDF” at the top of a proxy config. #Fail)

Imagine a playbook that could:

  • Test Web policies by downloading an encrypted zip
  • Test AV by downloading Eicar
  • Test firewall policy by connecting inbound HTTP 80 to your DMZ
  • Test SSL policy by connecting to an invalid Certificate

We could run this playbook every 60 minutes, and any test that “fails” can create a Critical Severity incident for the team to investigate WHY it was successful.

Testing Alert-Workflow

Referencing the “Connect to known C2C” validation above as an example, this should be blocked, but even when it is blocked we can test more:

  • Was the HTTP block logged in your LogStore/DataLake/SIEM?
  • Was this malicious request raised as a new Alert to your analysts?

Can we check this automaticaly, and check whether the alert creation is happening quick enough?

This kind of playbook can be left running for weeks, and you only get involved if an alert fails to be created. That’s a lot of peace of mind for a very small amount of effort.

Alternative Usecases

The list of actual test is endless, but if I still ran a SOC here is a simple list I would want to create for endless validation:

  • Bruteforce a random account and test if it becomes locked out
  • Is inbound password spraying detected?
  • Test inbound checks of SSL Cert validity and TLS1.0 handshakes
  • Inbound port scans, unsecure protocols
  • Add a new account to a sensitive OU (e.g. Domain Admins) and see if anyone notices
  • Run encoded/obfuscated PowerShell against endpoints
  • Probe internal lateral movement to sensitive networks
  • Large file transfer, transmit easily detected PII

What else could you test?