SANS Penetration Testing

Dealing with the Many Stages of Pen Test Result Grief - Part 1

By Ed Skoudis

If you've done penetration testing for any length of time, I'm sure you've encountered it. You perform a beautiful penetration test - technically rigorous, focused on real business risk, all wrapped up with a solid report. You don't wanna brag, but you feel pretty darned proud of completing a job well done.

And then... it happens. Target system personnel, the very people you've labored to help secure, blindside you with a barrage of criticisms of your findings in your draft report. Some penetration testers are shocked as target system personnel, both business decision makers and the technical people responsible for acting on the pen test findings, reject your results. It's almost as though they willfully don't understand your findings and the associated business risk. Your findings make perfect sense to you, yet they just don't get it despite your efforts to explain things as best you can. And, you still have to turn your draft report into a final report that is meaningful for the target organization. I've been there, my friends, right along side you. It can be soul crushing.

Over the years, my colleagues and I have developed our pen test methodologies and reporting techniques for avoiding and dealing with these situations. In this series of articles, we'll look at a variety of negative responses to penetration testing results often held by target system personnel. Most important, we'll discuss how to avoid these issues proactively in the way you conduct the test and report your findings so you can head them off in advance. And, if they still come up, we'll offer tips for how to respond to each situation so you can adjust your draft report into a really meaningful, useful final report for the organization.

In this series of articles, we'll present each of these situations as a stage, analogous to the stages of grief people go through when they suffer a loss. It's important to note, however, that not every group of target personnel will move through each and every stage. Often, people reading your report will fixate on a single stage and linger there. Your diligence can help them move off of the given stage and onto improving their security stance.

Ultimately, that's the bottom line - the goal of an effective penetration test is to help organizations improve their security stance in light of real-world threats and attack vectors demonstrating important business risk. As you'll see as this series unfolds, we'll keep coming back to that concept.

Another important note is worth mentioning: it is entirely possible that target system personnel are 100% right about their responses to your findings. As a pen tester, remember to keep your ego in check and actually listen carefully to target system personnel. Yes, some of their push back is just the classic stages of grief as we'll discuss in this series. But, sometimes, what they tell you is actually real and well founded. Your intelligently incorporating their responses into finalizing your report for this project and adjusting your approaches for future pen tests will make you a better penetration tester.

Oh, and one last point before we dig in. Thankfully, whenever issues like this come up, they seldom apply to all findings. Instead, they apply to just one or two particularly important findings in your penetration test. Thus, it's extremely unlikely that you'll have to use all the techniques we describe in this series for every finding. Instead, you should use them for your most critical and important findings, those that need action.

OK, with that background, let's get started by jumping to the single most common reason for rejecting pen test findings: denial.

Stage 1: Denial

Your so-called "vulnerability" is not really there. In fact, even it the situation you describe is real (which is doubtful), it's not really a vulnerability, but instead a feature of our environment.

To proactively avoid this kind of claim, make sure your report includes, for each critical or important finding, the following items:

a) A description of the attack steps needed to exploit the finding

b) A description of the business risk (in business terms) associated with the finding, particularly how it could damage the organization's mission, such as by costing it money, damaging its reputation, inviting regulatory oversight, etc.

c) One or more screenshots showing that the finding is real. Your screenshot should include annotation with arrows and text pointing to the proof on the screen that the vulnerability is really there. A picture is worth a thousand words, they say. Make your screenshot really illustrative that your finding is a problem and poses a business risk.

d) If available, a description of how other organizations suffered from attacks against the given issue.

You don't have to provide these items for each and every finding, but it's vital to include them for your critical and important findings, those that you really think require attention and near-term action.

Even if you provide the items described above in your draft report, you may still face target system personnel who play the "denial" game. If that happens, I recommend you double check the quality of each of the items in the list above, in particular, item b. Also, make sure item a provides a realistic, step-by-step narrative of how an actual, real-world threat faced by the target organization could exploit the given situation to cause harm.

Additionally, if you still get pushback from target system personnel in the form of denial, consider a brainstorming session with them (either face to face or via conference call) to discuss different remediation recommendations. It's often the case that their denial hinges not on your specific finding, but instead on the operational difficulty in implementing and managing your recommended fixes. If that's the case, brainstorming about alternative remediation efforts that can still fix the situation while being operationally tenable can help you achieve your overall goal of improving security in a cost-effective manner. Sometimes, you'll have to back off on your original recommendation (not your finding, but just the recommendation) and work with target system personnel to come up with clever and artful alternatives that still mitigate the flaw.

Stage 2: We Meant to Do It That Way

We made a conscious business decision to do things this way, so your "finding" is really just underscoring our design decisions. Thus, your work is unimportant, and we don't have to take your so-called "discovery" into account; the concept has been baked into our plans since the start.

Ah yes. This is a classic. To avoid it proactively, make sure your critical findings include each of the items we discussed in our "denial" segment above. Even so, this may still come up after you deliver your draft report. When confronted with this kind of response, remember to maintain your cool and be supportive. Let them know that you are happy to hear that they anticipated this situation, and that your pen test confirmed it. I like to use phrases like, "It's good to see we're on the same page then" and "Excellent. Let's work together to come up with a good mitigation strategy that works for you."

Then, as before, put together a step-by-step narrative showing how a real threat could exploit this issue causing business harm to the organization. Suggest a brainstorming call for different remediation tactics. With this response, it is more likely than ever that the organization really needs help in finding a remediation plan that they can operationalize around. Your job as a penetration tester is to help them find a practical way to thwart the tactics you used. I sometimes hear from pen testers this notion: "Look, I'm an offensive guy. How the heck do I know what to do defend against this stuff?" I find such an attitude is most often merely an excuse or a dodge, and I push back. "You're a very smart attacker," I truthfully say, "and I'll bet you could come up with half a dozen techniques that'll thwart the tactics you used here." Most of the time, the penetration tester agrees.

But, even if you can't come up with an acceptable and deployable approach yourself, in your mitigation brainstorming conference call or meeting, invite particularly clever and smart operations personnel that you've allied with in the target organization to help come up with solutions to your finding. Those interactions will be like gold as you tailor your final report recommendation to provide maximum value.

In our next installment in this series, we'll look at Stages 3 and 4 of pen test findings grief: Blame and "That's not FAIR!"

Read: Part 2 - Dealing with the Many Stages of Pen Test Result Grief

Finally, if you want to learn how to conduct a technically rigorous penetration test using an industry-proven methodology that provides real business value, you may want to check out my SANS Security 560 course Network Penetration Testing and Ethical Hacking.

It really is the must-have course to amp up your skills to become a GREAT pen tester. In addition to discussing issues like those in this series, we also perform dozens of in-depth, hands-on labs in which you'll build great technical skills in recon, scanning, exploitation, post exploitation, and much more. This class covers professional penetration testing end-to-end, and helps you get ready for conducting excellent pen tests.

-Ed Skoudis
SANS Institute Fellow
SANS Penetration Testing Curriculum Lead
Author and Instructor, SANS Security 560: Network Pen Testing & Ethical Hacking

Upcoming SANS Special Event - 2018 Holiday Hack Challenge

KringleCon

SANS Holiday Hack Challenge - KringleCon 2018

  • Free SANS Online Capture-the-Flag Challenge
  • Our annual gift to the entire Information Security Industry
  • Designed for novice to advanced InfoSec professionals
  • Fun for the whole family!!
  • Build and hone your skills in a fun and festive roleplaying like video game, by the makers of SANS NetWars
  • Learn more: www.kringlecon.com
  • Play previous versions from free 24/7/365: www.holidayhackchallenge.com

Player Feedback!

  • "On to level 4 of the #holidayhackchallenge. Thanks again @edskoudis / @SANSPenTest team." - @mikehodges
  • "#SANSHolidayHack Confession — I have never used python or scapy before. I got started with both today because of this game! Yay!" - @tww2b
  • "Happiness is watching my 12 yo meet @edskoudis at the end of #SANSHolidayHack quest. Now the gnomes #ProudHackerPapa" - @dnlongen
kringle_02

11 Comments

Posted April 27, 2014 at 2:34 PM | Permalink | Reply

Jabberwock

Well spoken! This is a nice complement to your SANS 560 material. Thanks for the great read!

Posted April 27, 2014 at 8:01 PM | Permalink | Reply

russell eubanks

Love to see the Kubler-Ross model ''" the five stages of grief applied to other areas.

Posted April 28, 2014 at 2:34 PM | Permalink | Reply

Ed Skoudis

Thanks, Jabberwock

Posted April 28, 2014 at 2:36 PM | Permalink | Reply

Ed Skoudis

Thank you, Russell. I used the Kubler-Ross model for inspiration, but you'll see that I didn't follow it with exactitude. That's on purpose. I found that, in a business context, some of its phases don't really apply, such as depression. Further, I found that I had specific tips for sub variants of some of its phases (such as many forms of denial or blame). Thus, I figured it would be most valuable to use the model as a rough guide, but get particular and diverge where it would allow for more specific tips and recommendations. Thanks again!

Posted April 28, 2014 at 2:37 PM | Permalink | Reply

Ed Skoudis

Thank you, Russell. I used the Kbler-Ross model for inspiration, but you'll see that I didn't follow it with exactitude. That's on purpose. I found that, in a business context, some of its phases don't really apply, such as depression. Further, I found that I had specific tips for sub variants of some of its phases (such as many forms of denial or blame). Thus, I figured it would be most valuable to use the model as a rough guide, but get particular and diverge where it would allow for more specific tips and recommendations. Thanks again!

Posted April 28, 2014 at 6:14 PM | Permalink | Reply

Sandro Gauci

Great to see this discussion as this is something I think all pentesters face often enough.
Wanted to add some points: I tend to tackle denial by showing them the real impact of the top vulnerabilities, i.e. how bad things can get. Examples include password lists (especially those belonging to the people you are dealing with when possible/allowable, making it personal and eliminating denial ), videos showing the attacker and the victim in a exploitation scenario (very useful for webapp vulnerabilities such as XSS

Posted April 29, 2014 at 1:05 PM | Permalink | Reply

Simon

Sandro, I think the "showing them the issue" approach needs care.
For some findings this is inherently difficult, or you may lack the time, skill, or even inclination to do so.
Because I can't hack you via my findings doesn't mean someone smarter than I can't. I'm not Dan Kaminsky, and never will be, but I can still find where the kinks are, find out where the software is sub-optimal, find out if your system admin understands how to deploy TLS correctly etc.
Findings are findings, best you can do is indicate how risky you think something is. Often a good example of a similar hack in the real world will sway minds, especially if they are management minds which don't understand the technical problem, a finding of "XSS" isn't meaningful to them, but a "this is how Paypal accounts were accessed in 20XX" focuses their minds that maybe even findings that the automated scanning tools rank as "minor" may get them into the newspapers (in a bad way).
Then I'm on the other side these days, the thing the Pen Tester needs to do is talk to me, because I will tell you exactly where I think the weaknesses are, and whilst you need to check everything, if you want to improve your odds just ask.
But then we are in the business of trying to ship well-secured software, not simply trying to ship software (we sometimes fail but the ethos is right), and we don't see Pen testing as "adversarial" in the way many businesses seem to. These are the one "bad guy" attacking your systems who is contractually obliged to say how he did it, and will tell you as soon as possible. I think the main issue is everyone does this too late, so everyone is in a hurry, so the last thing they need is uncertainty when deadlines are looming.

Posted July 24, 2014 at 6:29 PM | Permalink | Reply

Matt

Good points, but food for thought''.a little humility might be in order too. You might actually be wrong on occasion

Posted July 24, 2014 at 6:54 PM | Permalink | Reply

Ed Skoudis

Absolutely, Matt. That's why the article says near the beginning: "Another important note is worth mentioning: it is entirely possible that target system personnel are 100% right about their responses to your findings. As a pen tester, remember to keep your ego in check and actually listen carefully to target system personnel."

Posted July 24, 2014 at 8:37 PM | Permalink | Reply

Matt

Missed that Ed. Must have jumped to denial. Another point well made.

Posted July 24, 2014 at 9:25 PM | Permalink | Reply

Ed Skoudis

I appreciate your input, Matt. Thanks for reading

Post a Comment






Captcha


* Indicates a required field.