Talk transcript

I'm Pete. I run an information security company here in Wellington. We do a mixture of pen testing, audit work, maturity reviews, and a whole bunch of other things along those lines — working alongside government right through to software development companies and everything in between.

How this talk came about

This talk came about partly because of Jim Rush, who used to work with me. He quite regularly would tell people: if you give the same app to five different people, you're going to get five different outcomes. No two testers are the same, and you're never going to get exactly the same results every time. There's probably a baseline of things you will find, but not always.

I'm talking about this from a security testing perspective today, but I think there are a lot of similarities with other areas. Development, design, arts, medicine — give the same requirements to five different people and you're going to get five slightly different outcomes.

For context: penetration testing is security testing. We do it to get confidence in the security of applications and systems we've developed. One thing we often find is that when we go and do some work with an organisation and ask for their previous pen test report, we'll often find a whole bunch of issues that haven't been seen before. Ask for the last five years' reports, and you'll probably get five wildly different results. Sometimes that's because they've added new features and functionality. Sometimes it's just because different people have been looking at it each year.

This talk is a dig into why that might be the case, what some of those reasons are, and how you can try to mitigate it.

Talk overview

We'll cover: the people factor, how scoping can change things, how security is a bit of a moving target, how methodology differs from tooling, real constraints, and some takeaways.

The people factor

Who's in your team is really important — and who a tester is matters a lot. Everyone has different backgrounds, different skill sets, and different things that influence what they do. People think and act in different ways.

When you engage a company to do testing, in theory you're engaging the whole company, not just one person. In practice, you might only have one person doing the actual testing, but there's usually a QA process, peer review, or just someone wandering past and saying, "Oh, have you tried this?" or "Anyone familiar with this technology want to have a look?"

To illustrate: imagine two fictional pen testers, Steve and Shelley. Who would do the better pen test? Shelley's wearing a sweet hoodie, which is pretty hacker. Steve looks a bit like a nerd, but maybe he's just a sweet hacker who really loves suits. The point is you can't tell from the outside.

Their backgrounds and experience will heavily affect what they find. People come from different places and have done different things before getting into security. Pete himself was an electrical engineer. He knows people who were welders before becoming pen testers, people who came from music, from all kinds of different industries. Those past experiences affect how you think about things, what parts of your brain you use, and the context you bring to a test.

Training makes a difference too. There are hundreds of qualifications in the security space — OSCP (Offensive Security Certified Professional), CEH (Certified Ethical Hacker), and many others. Some people have a university background before getting into security. Others came straight from high school, straight from work, or found their way in through bug bounties. All of these different paths lead to different ways of approaching a test.

People also think differently. Whether you tend toward a more creative or more logical approach is going to affect what you find. Left-brain, right-brain, however you want to describe it — it makes a real difference.

Fresh eyes: junior tester story

Experience level obviously matters. If this is your first test, you're probably not going to find the same things as someone who's been testing similar technologies for 15 years.

But there's a flip side to this. If you've been testing for a long time and you've become a bit blasé about things, you might start missing stuff.

A good example: we had a junior tester — fairly fresh, maybe six to twelve months in — who always worked through a checklist on every test. He found an issue in an app that had been quite well tested. It had been getting tested annually for the last five years.

He found an issue with the ECSOimplementation, where it wasn't doing a check on the nonce. That essentially meant you could bypass the authorisation checks. As long as you had any account, you could give yourself admin access.

Other testers in the team hadn't caught it. They technically knew the attack existed, but had dismissed it as something that never actually happens in practice. This junior tester was just working down his checklist and went, "No, this is on my list of things to check." He checked it, it wasn't implemented properly, and he found a way in.

That finding, paired with another person in the team, ended up achieving remote code execution (RCE) in the app.

Sometimes fresh eyes find things that more experienced testers miss — simply because experienced testers have already decided something isn't interesting, or isn't something they'd normally pursue.

Bug bounties vs consultancy

Bug bounties are a model where organisations open up their systems to the public — here's our scope, find bugs, get paid. Some people make a full living doing bug bounties. The efficient approach is to run broad enumeration across a lot of apps, then go deep on a particular niche — timing attacks, logic bugs, specific types of auth bugs, or known vulnerable components.

Someone with a bug bounty background is probably going to find different things than someone who's spent their whole career in a consultancy. Bug bounty hunters tend to find really niche, interesting bugs and do great enumeration. Consultancy testers typically work with wider scopes and less crowded applications, so they might find different things at different depths.

Neither background is better — they're just different, and they produce different outcomes.

Going back to Steve and Shelley: Steve had a previous life as a mechanic, spends his spare time doing bug bounties, and loves web app testing — but has no CVEs to his name. Shelley just came out of university, has her OSCP, loves network testing, has never had prior work experience, but has a number of publicly disclosed vulnerabilities. Completely different profiles, completely different things they'll likely find.

Time allocation

The way people allocate their time during a test also has a big impact on outcomes.

If you've got a fixed amount of time and you spend most of it on enumeration at the start, you'll find a different set of things than if you do quick enumeration and then dig into areas that smell a bit off.

One day on an application will get you the surface-level bugs. Three, five, or twenty days gives you the chance to dig in and find something quite cool and deep. In New Zealand, budgets are typically more constrained, so we tend to work with shorter engagements — and that shapes what we find.

Different testers also have different risk appetites and bug-class interests. If you're particularly passionate about auth, you're going to target that area immediately. Depending on who's looked at it before and how the developers built it, you'll find very different things.

Some testers lean heavily on tooling — this tends to be more common on the junior end. Others go by intuition, code review, or first principles. Tooling gives you one kind of result; manual, considered analysis gives you another.

The artist analogy: different approaches

Give five different artists the same scene to paint and they'll go about it in five different ways. Some start with the background, some with the foreground. Some use paint, some coloured pencils, some might finger paint, some use fine brushes. You end up with very different results — not because any one artist is wrong, but because each brings their own approach.

If you layer constraints on how they work, you'll get more consistency. But you might also lose variety, and potentially lose some of the best work.

This isn't just security testing. The same thing applies to functional QA testing. Give the same requirements to people with different test plans — or no test plan — and you'll find different things depending on their background and experience.

Financial auditing is similar. Technically, auditors work to frameworks, but different levels of knowledge and experience still lead to different things being found. And financial auditing has been around for a very long time, so there's been much more time to develop rigour and settle on what "good" looks like.

Pen testing in New Zealand is comparatively young. The first pen testing company here was formed in 2005 — that's only about 20 years. As technology evolves, we'll probably see more appropriate guardrails and methodologies develop. But there will always be a need for both art and science. Nothing is ever truly cookie cutter.

Frameworks and standards

There are a bunch of frameworks in the security space: OWASP WSTGs (Web Security Testing Guidelines), the Penetration Testing Execution Standard (PTES), and the NIST guidelines, among others.

Do most people Pete knows actually follow these strictly? No! There's a level of formality that doesn't always suit a relatively young industry, and sometimes these frameworks put constraints on testing that aren't helpful.

On the AI side: AI-assisted tooling is the new hot thing, and it's getting better at finding issues faster. But you still need to know the technologies you're looking at. You need to be able to look at results critically. AI tends to find known patterns and known vulnerability classes well. Novel bugs and logic issues are still most likely to come from manual testing.

Scoping

Scoping is really important. Pete has been in scoping meetings that lasted five minutes and ones that went for an hour. The difference in outcomes can be significant — more context means more focused testing.

Some clients can provide source code and design diagrams (this is called white box or open box testing). Others hand over a URL and say, "Here's the thing, have at it" — that's black box testing, where you only know what the public would know.

One of the most important questions in scoping is: what outcome are you actually after? Do you want to genuinely improve your security posture and protect your users' data? Or do you need to demonstrate PCI (Payment Card Industry) compliance so you can take credit card payments? Those are two different goals, likely to get two different amounts of investment, and two different testing focuses.

One client didn't want to remove their web application firewall (WAF) during testing, which is something that's usually helpful to do so testers can focus on the app itself without being blocked by rate limiting. One tester took that as a challenge and went looking for WAF bypasses. He found some interesting bugs as a result. But if they'd just removed the WAF upfront, he probably would have found a broader range of issues — and he might not have been nearly as motivated to dig as deep as he did.

Real constraints

There are always constraints, and this is true across all kinds of testing — functional, non-functional, security. Limited time, limited budget, needing to fit testing into the software development lifecycle.

Can we only test in non-production? Can we use real data? Are there things you won't tell us about the system? All of these shape what's possible.

A three-day engagement versus a five-day versus a twenty-day engagement will get you very different results. In New Zealand, budget constraints tend to be real, and that affects outcomes. Something worth being aware of on both sides of the table.

One thing Pete's noticed from his own side: testers sometimes assume clients don't want to spend money, or won't want to give much time. The better approach is to just ask. How much are you looking to invest in this? What areas do you most want us to focus on? Those questions often surface things that change the entire scope of an engagement.

The type of application matters too. Payment applications → the integrity of the payment flow is critical. Government systems → reputation, integrity, and confidentiality are the key priorities. A time-sensitive app — like an election app or a nominations platform that only opens for one day a year — needs to prioritise availability. If you don't know these priorities going in, you're less likely to focus testing in the right areas.

Security is a moving target

Just because a vulnerability didn't exist yesterday doesn't mean it won't exist today.

Some examples:

Log4Shell happened about five years ago, just before Christmas — the worst possible timing. It was a critical vulnerability found in a popular Java logging library and was present across a huge number of apps and environments. There was a massive scramble to patch things over the holiday period. The issue had existed for years before it was discovered. Just because we didn't know about it didn't mean it wasn't there.

The XZ Utils backdoor was found by an engineer at Microsoft who was doing benchmarking and noticed SSH was taking a slightly longer time than expected. A very subtle deviation. Most people would not have thought twice about it. But this person's background made them care about it, and they dug in and found a deliberately planted backdoor.

React2Shell was found by a Kiwi bloke Lochie, after working on a couple of engagements where React was involved. He found a vulnerability that ended up affecting a huge number of applications across the internet.

The common thread: these issues existed before they were publicly known. Lots of people had probably done pen tests on systems using these components. But without the specific focus or background to look at that one thing, they didn't find it.

What can we actually do?

So, given all of this — what can we actually do about it?

Define scopes collaboratively. Ask the right questions. Make sure you actually know what outcomes you're after before the testing starts.

Talk about methodologies upfront. Are there frameworks you particularly care about? Do you want a structured, compliance-driven test, or do you want the tester to have more freedom to explore novel areas? Knowing this upfront shapes how the work gets done.

Be explicit about priorities. Do you care most about the availability of the application? Confidentiality? The auth flow? Where do you most want effort placed? Say it out loud — don't assume the tester will guess correctly.

Treat reports as conversations. The same finding can have different severity for different clients. A recent example: a previous tester had looked at an IP whitelisting issue and marked it as acceptable because of other controls in place. When Pete's team looked at the same issue and had a conversation with the client, it turned out the client had no idea it had been raised before — and considered it a serious problem. Having that conversation can completely change how findings are prioritised.

Retest after fixing. When you fix issues, you can sometimes introduce new ones. Don't just close things off in your risk register and assume you're done. Getting someone to look at the fix is important.

Layer your controls. No single pen test finds everything. Combining code analysis in your pipeline, regular pen tests, a bug bounty programme, plus good security hygiene — knowing your software components, keeping things patched and up to date, having logging in place — gives you much better coverage than any one approach alone.

Even Google Chrome — one of the most scrutinised pieces of software in the world — has someone sitting at the top of their bug bounty leaderboard right now finding novel issues. Nothing is truly safe from everything. But with layered controls, you're in a much better position.

Talk references

• Privsec Consulting

• OSCP (Offensive Security Certified Professional)

• CEH (Certified Ethical Hacker)

• ECSO (European Cyber Security Organisation)

• RCE (Remote code execution)

• React2Shell