Attestations Don’t Always Measure your Defensive Posture

An attestation, by definition, is an indication that makes something evident. In the case of the security, specifically security programs it means to certify in an official capacity.

People often ask me what makes a good security program. As much as I would like to point to one aspect of my security perimeter to use as an example, there are multiple items to highlight. The industry relies on attestations and certifications to measure your security defenses. Engineers and operators will tell you that your actual security perimeter and threat assessment capabilities define your security program. I will tell you it is both compliance attestations as a measurement and the operational capabilities of your security team that define your program. Though attestations alone are not an accurate benchmark to measure a program.

Attestations are an industry necessity to ensure compliance with federal, local and state statutes as well as industry best practices. ISO, NIST or DoD standards form the baseline of most attestations. NIST, for example, publishes a set of standards and technical guides to help organizations build perimeter defenses that are “acceptable” to the government. As I will outline however, just because the standards are set doesn’t mean implementation is always stellar.

Deployment of a Tool Doesn’t Mean it is Providing Value

Controls allow for flexibility in implementation and operational growth and innovation over time. Unfortunately some organizations use the flexibility to check the box, but have no real defenses in place.

A prime example of this problem is intrusion detection/protection systems (IDS or IPS). Like virus scanners, most organizations invest in IDS/IPS as a matter of standard security practices to guard against malicious traffic and data exfiltration. The industry is replete with vendors making various forms of IDS/IPS systems. However, some organizations build systems rather than buy.

I recently left one such organization that “built” their own intrusion detection system from open source tools. Auditors were told the system was a “fantastic tool”, and even given examples of traffic. When I dug deeper into the telemetry the tool was providing, I realized that traffic was not being analyzed at all. Rather, passing through the sensor as it was not configured to capture any traffic or alert at all. Furthermore, the credentials used to administer the tool were set up by a previous employee and were never updated after his departure. So essentially, the tool was sitting idle for months without any human intervention. Not only does this put the company at risk, but it also compromises the perimeter.

A savvy auditor wouldn’t have caught the issue because the attestations don’t look for “operational” information on all systems – the standard is literally one layer of question and answer. In fact, most attestations measure simply if the tool exists, not operational viability. Additionally, most auditors are not technical enough to discern a functional IDS/IPS from a non-functional. The meat of the audit relies on the company to put their best foot forward rather than answer tough questions. Auditors also have to cover a vast array of controls during an audit so time is a large factor in the quality of their analysis.

An attestation alone will tell you that a company has a mature security program with controls. Requiring a potential partner to complete a vendor survey won’t provide you confidence either. Surveys merely outline the same information in a different format. So how do you evaluate a mature security program?

Evaluate the Entire Cloud Security Program

First, you should review at a minimum the attestations and the findings report, not the executive summary. That will provide you with an overview of the program reviewed by a third party. Second, you should definitely review if the company undergoes a third party penetration test or bug bounty program. Personally I am not a fan of bug bounties, but I am a fan of third party penetration testing on an annual basis. Pentesting provides you with a structured test of your defenses and real feedback on vulnerabilities. Finally, review the security documents (usually table of contents) the company utilizes as a basis for implementation. This includes (but certainly is not limited to) a security policy, incident response and vulnerability management. An experienced security team will offer to share those documents and artifacts as a part of normal business.

I make it a matter of course to evaluate every vendor and partner from the perspective of access to company data. Meaning if the partner or vendor manages company data, they’re subject to more scrutiny than a vendor that does not. Keep in mind the business purpose when evaluating a security program. I review the business purpose and type of information involved, then evaluate from that perspective, rather than handle all partners and vendors the same. When in doubt, always ask for more information.