SANS Penetration Testing

Dealing with the Many Stages of Pen Test Result Grief - Part 2

By Ed Skoudis

In this series of articles, we're looking at some of the grief that penetration testers often encounter when they deliver their results and recommendations. Our premise? You, a great pen tester, work your tail off to conduct a wonderful, high-value, technically awesome pen test. The result? Target system personnel vomit all over your findings, push back on your recommendation, and just plain don't see the value of what you've done. The series, which began with article one here, focuses on practical tips you can use to avoid such situations up front, or, if they do occur later on, methods for defusing the situation and demonstrating the real value you are providing.

Article 1 in the series sets the backdrop and talks about Stage 1: Denial and Stage 2: We Meant to Do It That Way. Here in article 2, we'll pick up where we left off by exploring Stage 3: The Blame Game (Blame the Messenger) and Stage 4: That's Not FAIR!

Stage 3: The Blame Game (Blame the Messenger)

Target System Personnel Reviewing the Report: "You've actually caused the issue at hand by running your freaky-deaky ultra-scary hacking tools. Something like that would never happen in a real-world attack."

This complaint sometimes comes from target system personnel not understanding how widespread and common hacking tools are today and the value they can provide in actually improving security. You can hardly blame them. For some people outside of the information security community, hacking tools are mysterious, edgy, voodoo magick. Compounding the problem, many penetration test reports reinforce the voodoo magick misperception, because they don't go into any details about how the tools found or exploited the flaw. In my work reviewing pen test reports, I see a lot of deliverables that blurt out critical findings as though they just fell out of the sky, with little information for target system personnel to understand the true nature of the risk they face.

To head this one off in advance, you as the penetration tester should try to help educate target system personnel by including some relatively small but very important details in your report. Make sure your report includes the names of the tools you used during the test for each finding. Also, for particularly critical findings, explain at a high-level the underlying technique your tool used to discover the finding, along with how common that technique is applied in the wild. Explain how real-world attackers use the same or similar tools to exploit these kinds of flaws every day. Your goal is to make the critical issues you've discovered real to your report readers, tying them to real-world threats. These small tweaks to your report will not only help inform target system personnel about the validity of your findings, they will also help them better understand what they are up against in defending their environments.

That's one form of the Blame Game that pen testers may encounter. We'll see another variation of the Blame Game in a follow-up article in this series, where target system personnel blame not the pen testers, but another organization or entity outside their control.

Stage 4: That's Not FAIR!

Target System Personnel Reviewing the Report: "Well, well, well? Sure you can come up with such findings if you cheat. The only way someone could discover such a flaw is by utilizing some super secret inside knowledge, which we gave you in good faith at the outset of the pen test before you betrayed us, Benedict Arnold. No real bad guy would ever be able to figure that out and mount such an attack."

Here's a classic response that comes up surprisingly often. During the initial scoping of the project or during project update debriefings, target system personnel will almost always share information that proves extremely useful in finding juicy flaws during your penetration test. They might mention an architectural problem, a broken process, a particularly weak system, or more, allowing you to find and exploit the crap out of the target environment. When you report on your results, sometimes target system personnel turn on you and say, "That's not FAIR!"

To deal with this, I always start a project by talking with target system security and operations personnel trying to defuse an inherent conflict. You see, target system personnel sometimes fear pen testers (as they worry that we may accidentally break something like a bull in a china shop), and other times they just plain loathe us (as we look through their work to find flaws which could get them in trouble). This mix of fear and loathing can make for... er... rather complex emotions during the test, to put it lightly. At the outset of a project, I usually say in a nice way, "Look, we both have the same goal — to help improve the organization's security and its operations. Let's work together to make that happen."

I go even further, saying, "You know this environment better than I ever will? I'm sure there are some things you know that would help improve security, but you haven't been able to get management to invest the resources needed to do so. Use me. I can be a vehicle by which you can move your organization to improving its security stance, if we work together." It's a sincere pledge on my part, and being explicit about it up front can help you identify important issues early on during a project. You are planting the seeds for a good working relationship with target system personnel so that you don't find yourself with a "That's Not FAIR" claim.

Then, of course, when you write up the report, you should point out those issues that were identified during the beginning of the project, emphasizing the importance of the issues and their associated defenses. Only mention that you learned of the issues from the in-house team if the individuals on that team have agreed to such a mention. You also have to show, in your report, how a real-world bad guy could find and exploit the security issue that was, for your pen test, kind of a "gimme" by target system personnel. Help them to understand how real-world attackers working over months may find this same issue, which you modeled in your penetration test over the course of a week or two.

Even if you do all of this flawlessly, you may still hear a cry of "That's Not FAIR!" If this happens to you, remember to keep calm and never get your back up. You are in the right, as long as you've stayed within scope, followed your rules of engagement, and utilized real-world attack techniques.

Here's what I like to do when this occurs: When someone claims, "That's Not FAIR," I translate them in real time, in my head, using an internal voice that sounds like a bad voice over from an old kung fu movie to, "You have bested me with your impressive kung fu, and I congratulate you on your success. Please help me understand how you have done this great feat so that I can improve the security of my enterprise." Yeah, it's a little wordy replacement for "That's Not FAIR," but it helps me smile instead of reacting negatively. You can then proceed calmly to explain how a real-world attacker would find the associated issue using publicly available tools and information, along with your recommendations to help improve the organization's security stance.

So, that's Stage 3 and 4. In part 3 of this series, we'll discuss some tips to use when dealing with Stage 5 ("I don't think that finding means what you think it means") and Stage 6 ("Scope-a-Dope"). Please stay tuned.

In the mean time, if you are interested in all things penetration test related, I'd love to see you take the SANS Security 560 course from me.

-Ed Skoudis
SANS Institute Fellow
SANS Penetration Testing Curriculum Lead
Author and Instructor, SANS Security 560: Network Pen Testing & Ethical Hacking

Upcoming SANS Special Event - 2018 Holiday Hack Challenge


SANS Holiday Hack Challenge - KringleCon 2018

  • Free SANS Online Capture-the-Flag Challenge
  • Our annual gift to the entire Information Security Industry
  • Designed for novice to advanced InfoSec professionals
  • Fun for the whole family!!
  • Build and hone your skills in a fun and festive roleplaying like video game, by the makers of SANS NetWars
  • Learn more:
  • Play previous versions from free 24/7/365:

Player Feedback!

  • "On to level 4 of the #holidayhackchallenge. Thanks again @edskoudis / @SANSPenTest team." - @mikehodges
  • "#SANSHolidayHack Confession — I have never used python or scapy before. I got started with both today because of this game! Yay!" - @tww2b
  • "Happiness is watching my 12 yo meet @edskoudis at the end of #SANSHolidayHack quest. Now the gnomes #ProudHackerPapa" - @dnlongen


Posted August 7, 2014 at 7:20 PM | Permalink | Reply


Another fantastic write-up. Thank you! Also glad to see that you're catching onto the JLV hair trend. I am beginning to believe this is a huge factor in pentesting success.

Post a Comment


* Indicates a required field.