How can security incident essentials help mitigate major breaches

From detection to resolution

At Flexion, one of our core competencies is keeping data safe and secure. Many of our projects involve the U.S. federal government, so we keep security at the forefront of everything we do. As mentioned in the Oath of Allegiance, we see ourselves as supporting and defending the Constitution and laws of the United States of America against all enemies, foreign and domestic.  We’re not just protecting our nation, but we’re protecting the people who call this country home.

Security incidents happen

Security incidents happen. It’s unfortunate and it’s unavoidable. We’ve plenty of tools and processes to help (e.g., MegaLinter, pre-commit), but an important thing to remember is that technology can only go so far. Behind passwords and firewalls are real, failable people. When security incidents happen, we all have a part to play in mediating the impacts.  This solemn responsibility belongs to each one of us, not just a team of DevSecOps Engineers. Being able to respond calmly and decisively to intense, high-pressure situations is important. A huge skill we learn and practice is how to communicate effectively when a security incident happens.  Reacting quickly is important, but so is being able to prevent future incidents from happening again. Communication needs to be clear and measured and convey not only information but also provide a level of reassurance that the situation is under control.

This is not an Incident Response Plan. Each project should have an Incident Response Plan or IRP. An IRP is typically included in the applicable Authority To Operate (ATO) and/or Risk Management Framework (RMF). It spells out roles and communication requirements and steps to take to mitigate the risk of an active incident. Typically, an IRP is written by and for security folks. It’s important – really important – but it’s not an elixir to solve every problem, nor is it equally applicable to everyone involved. This isn’t that. This is an overview of what’s involved so that everyone has an understanding of the essentials for when something happens.  This does not replace an IRP, nor does it alleviate the need for planning and documenting procedures for handling incidents.  

Security incident core concepts

At Flexion, we’ve honed a set of practices that consistently drive success in complex environments. These practices aren’t just theoretical—they’re actionable steps we’ve refined through experience, ensuring efficiency, clarity, and adaptability in every project. Whether you’re navigating policy changes or tackling technical challenges, the foundation remains the same: mitigate to control scope, document to capture critical insights, communicate to align with stakeholders, and reflect to continuously improve. In the sections ahead, we’ll break down these core concepts and show you how to apply them effectively to streamline workflows and deliver impactful results, just as we do. 

1. Mitigation (limiting the scope)
2. Documentation (taking notes)
3. Communication (collaborating with stakeholders)
4. Retrospective (what did we learn)

Let’s dive deeper into these.

1. Mitigation

This is where we prevent the situation from getting worse. Something bad happened, yes, but we don’t want it to get any worse if we can avoid it. This can be a little tricky as it’s possible to go too far. We want to stabilize the situation, but we also need to be careful not to damage or destroy potentially forensic or exculpatory evidence that may be needed later. Maybe there’s an investigation or maybe the logs need to be reviewed to see who did what. We may not know at the moment – we just know that we don’t want things to get worse. Mitigation can take many different forms. It can include changing passwords, updating access controls, revoking user privileges, modifying log retention settings, or even bringing systems offline.

Mitigation is whatever we can do to avoid making the situation worse. At the same time, it’s being mindful not to destroy something we may need later. It’s important to act quickly and decisively, but not rashly or hastily. Another factor to consider is the outward appearance of mitigating activities. It’s important to avoid the appearance of covering things up or obfuscating truths. At Flexion, we believe that people make mistakes and that’s to be expected; however, when things turn to covering up what happened, that’s when the situation gets ugly. An excellent tool that can help inform mitigation efforts is the project’s Threat Model (TM). The Threat Model documents what threats may be applicable to a project and what risks they represent. Knowing what aspects of the project are related to what threats can remind folks about how an incident over here may have results over there.

2. Documentation

When a security incident happens, it’s important to document everything. Document all the things. Part of the after-action conversation is knowing what happened when, who had access to what, what systems were compromised, what data may have been tainted or stolen, etc., so take notes about the people involved, the conversations that take place, the nature of the systems and/or data that are involved, and any other relevant details that may be needed later.

One of the first things that will be discussed is when something became accessible or vulnerable and when that vulnerability was mitigated. This directs forensic activities and verifies that the people who need to know, know. Documentation also includes capturing evidence and proof. This can include log files or screenshots. It’s important, however, to avoid making the situation worse. Don’t have documentation efforts work at cross-purposes to mitigation efforts. Taking a screenshot of a password, for example, means that the screenshot is now a part of the problem.

It’s also important to avoid sharing sensitive details (e.g., API keys, passwords, tokens, credentials, etc.) over unsafe communication channels like email or instant messaging. Documentation also includes determining what bad actors could do if they knew about this situation. The potential consequences (scope) of an incident could be radically different if one operator noticed an API key in a log as opposed to a data dump of API keys, Personally Identifiable Information (PII), Controlled Unclassified Information (CUI), Personal Health Information (PHI), Personal Financial Information (PFI), etc. that could be indexed by a web crawler and potentially appear as results in an online search or be incorporated into a dataset used by a Large Language Model (LLM) used by Artificial Intelligence (AI).

3. Communication

Communication is the lifeblood of all incident responses.  It’s keeping our teams informed (we don’t want people working at cross-purposes or engaging in redundant activities) and we don’t want our colleagues to make the situation worse. It’s also about reassuring our stakeholders that the situation is under control. It’s also about making sure that upstream and downstream teams, offices, and individuals with related accountabilities are kept current as the situation unfolds. One of the most frustrating things is when stuff is getting crazy and the phone starts going off as everyone is asking questions and they all want a status update. The whole process falls apart and nobody is happy. Our process for situations like these is to have a single person be responsible for continually communicating with stakeholders. This person serves as a single point of contact through which all questions and updates flow. Folks on the team who receive questions or requests are asked to forward them along to the single point of contact for the incident so that they can manage the conversation.

It’s critically important to be transparent about any and all communications. The single point of contact needs to be fed accurate, complete, honest, and timely information. Individual developers, administrators, operators, etc. need to be shielded from questions so that they can focus on mitigating and documenting the situation. It’s not a good use of their time to be involved in direct communication. The team speaks with a single, authoritative voice. It’s also extremely important that the person responsible for communicating about the situation not be involved in the situation itself. This shores up trust and confidence in the team while also minimizing the potential for those involved in the incident to selectively communicate or act from a place of pride.

Another helpful tool that can assist during a security incident is the outcome of a recent Table Top Exercise.  A Table Top Exercise is a training exercise where the team supposes that an incident is happening and then goes through the motions of responding to it. This includes bringing out the Incident Response Plan (IRP), establishing communication channels (including who would be speaking with whom), verifying contact information, etc. Not only does a Table Top Exercise shine light on any missing details, but it also establishes a pattern and a rhythm where folks can become comfortable (as comfortable as can be expected during a security incident) with the process. Whenever you’re drafting an IRP or a Crisis Communications Plan (CCP), be sure to run it past not only your team so they know what to do, but also the folks who will be on the receiving end (e.g., your Information System Security Officer (ISSO), Incident Response Team (IRT), Computer Emergency Response Team (CERT), Network Operations Center (NOC), etc. as they may have unanticipated needs that can be addressed before an actual problem happens.

4. Retrospective

The final component is to take a look at what happened and have an open conversation about what could be learned from the incident. It involves all of the other components while converting the focus from what happened to the future. The most important goal of the retrospective is to learn and grow. The retrospective is not about assigning blame or determining responsibility. Ideally, names shouldn’t come up. When an incident happens, it’s on the team, not individual team members. Assigning blame is destructive. Assigning blame is counterproductive. Assigning blame undermines team psychological safety. Assigning blame dramatically reduces the chance that future incidents will be handled well.

“People get defensive, especially when they’re scared – scared of punishment shame humiliation, or loss of reputation.”

The natural tendency is to operate from a perspective of self-preservation; when that happens, focus is taken away from protecting the team the client, or the company and that’s the exact opposite of what we want to have happen. It can be helpful to have a template for a retrospective. Knowing what questions will be asked can help the team be thorough and efficient.

The goal of the retrospective is to learn and grow. We learn about what we could have done better and what we could do differently to prevent something like this from happening again. Lessons learned today result in a better, safer tomorrow.

Takeaways

1. Mitigate: prevent the situation from getting worse without destroying evidence
2. Document: take notes about all of the things
3. Communicate: a single person provides open, honest, transparent communication
4. Retrospect: learn and grow but never shame or blame; avoid naming if possible

Flexion, while not a certified Third Party Assessment Organization (3PAO) can help:

• Draft IRPs and other compliance-related documents
• Threat Modeling
• Table Top Exercises

If you’d like to discuss a partnership with Flexion to up your DevSecOps game, contact us today!