Incident Response

Outline
• Introduction
• Section One – What you need to know about
preparing well in advance of an incident
• Section Two – General incident response
process
• Detection and Analysis
• Containment, Eradication and Recovery
• Section Three (time permitting) – Four use case
scenarios of real incidents
Role of Incident
Response
•Specialized service for incidents
• Systematic, repeatable, consistent lifecycle
• Compliance with policy/law to have capabilities to
respond to incidents
• Prevent, detect, and respond
Types of Incidents
• Phishing • Acceptable Use violations
• Worms • Property Loss/Theft
• Vulnerabilities • Data Loss
• Insider Threat • Misconfiguration
• Hacked accounts • Abuse
• Pivoting
• Brute Force
• DoS
Prevention – First Step to Being
Prepared
• Understand and document project/organization assets and
the threats and vulnerabilities to those assets. Also known
as a risk assessment.
• Identify security and related policies.
• Development of a Cyber Security Plan to protect against those
risks.
• Develop a Cybersecurity Incident Response Plan
Prevention – First Step to Being
Prepared
nt.
• Identify security and related policies.
• Development of a Cyber Security Plan to protect against those
risks.
• Develop a Cybersecurity Incident Response Plan
Policy Examples
• Information classification policy - Classifies types of data and what risk
exposure is to them.
• HIPAA, FERPA, ITAR, EAR – Federal government regulations
• Information disclosure policy - How and to whom project information
might be shared.
• Media policy - Who and how will interaction with the media be conducted.
• Privacy policy - Address the privacy expectations of participants in the
project.
• Security policy - How will security be treated by the project.
• Disaster recovery - How will the project recover from any type of disaster.
Cybersecurity Plan
• A plan/strategy to address the risks your project/organization have identified.
• Cybersecurity best practice – as a starting point
• [Link]
• For more details see our tutorial on developing a cybersecurity plan.
• Monitoring and alert systems
• These feed your incident investigations
• In open environments monitoring is a key security backbone.
• Networks & firewalls
• Hosts & application logging
• Intrusion detection systems
• Scans
• Security threat lists and feeds
• Much more on this later in the talk
Prevention – First Step to Being
Prepared
• Understand and document project/organization assets and the
threats and vulnerabilities to those assets. Also known as a risk
assessment.
• Identify security and related policies.
• Development of a Cyber Security Plan to protect against those
risks.
• Develop a Cybersecurity Incident Response Plan
Incident Response Plan
• Guide – NIST SP800-61
• Identifies
• Process used to investigate an incident
• Incident mitigation to remedy the situation, including potentially patching, blocking off, and
recovery
• Documents what was found, how, when, and what was done to remedy it
Incident Response Plan
Categories
• Incident Alert Mechanisms
• Forming a team – makeup and skills
• Contact information & Incident reporting mechanisms – who,
what, when, how……
• Team coordination & communication
• Secure information channels
• Monitoring - Information sources & tools
• Log Management
Forming an IR Team
• Decide who is on the team
• Someone to lead investigation (may depend upon experience)
• Network expertise
• Key application expertise
• Security knowledge and responsibility
• Define roles and responsibilities
• Who is leading
• Analysts
• Containment
• Restore
• Define expectations of outside resources
• Identify subject matter experts
Types of IR Teams
• Managed Security Team
• This team has experience with security and incident response.
• Likely more than a single individual
• One (maybe full-time) security person.
• Several other members who know they will be called on during an incident
• A person who is assigned as part of other duties

• Distributed Team
• May be a coordinated response by multiple teams
IR Team Skills
Team Leader Team Members
 Communication skills  Networking Expertise
 Diplomacy  Knowledge of Key Applications and
 Organized Services
 Host & System Expertise
 Integrity
 Malicious Code (Viruses, Worms,
 Coping with Stress
Trojan Horse programs)
 Problem Solving
 Programming Skills
 Time Management
Internal IR Team Coordination &
Communications
• Identify the leader of the incident
• Define responsibilities and roles
• Assign the roles
• Identify who is authorized to make decisions
• Does the system have to come down?
• Whole system need to be scrubbed and reinstalled?
• Initiate immediate plan of action? (black-hole routing)
• Define communication and collaboration plan
• Remote access
• Outside organizations
• Collaboration environments
• Name who will interact with external groups
Dry Run/Practicing
• Test all your IR plans and procedures
• Run through all the scenarios you’re prepared for if possible
• Involve as many of the team members as possible
• Test communication channels (wikis, encryption keys, communication lines)
• Be sure to test disaster recovery procedures
• Test at least once per year
• Better to find issues during a test than a real incident
Log Management Policies and Procedures
Log Management Policies and
Procedures
• Logs are the source of all your investigative efforts.
• It is important to establish site wide log management procedures.
• For large projects and sites this typically requires a dedicated log
management infrastructure.
Log Capture
• Ensure logging is enabled on security systems, routers and devices,
storage, VMs, operating systems and applications
• Ensure that data captured includes all key events such as:
• Sign in and out plus identity, Source and destination, AuthN Versions,
Synchronized Time stamp, root activity
• If it applies record sensitive data access
• Consider privacy issues when setting up user activity logging
• Remember this information captures user behavior
• Universities have strict rules about privacy
• Monitor any changes or access to the logging infrastructure
Log Retention and Storage
• A centrally managed log management infrastructure is needed
• Automated methods to move logs or shadow them
• For efficiency as well as security
• Log management system restricted to security staff.
• You’re going to be adding to logs constantly so devise a strategy
to meet those requirements and take into consideration how you
use the logs (i.e. search them)
• Data retention policy
• Ensure retired logs are disposed of
Log Protection
• Secure the processes that generates log entries
• Limit access to log files – trusted staff only
• Implement secure log transfer mechanisms
• Protect the confidentiality and integrity of log files
• Provide adequate physical protection for logging mechanisms and
stored logs
• Validate log system is working daily
• Logs are valuable because they contain so much information,
information other’s may also find useful
Log Analysis
• Regularly review and analyze logs
• Automation is KEY
• Leverage correlation tools for holistic view & reduce false positives
• Use automated reporting generation tools and review daily
• Log analysis should include real-time monitoring
• Set up an alerting system based on priorities
• Develop a baseline of typical log entries in order to detect
unusual or anomalous events or activities
• And keep updating this over time.
 Understanding Basic Attack
•
Vectors
Stolen accounts & brute force – basically password attacks
• Attacker reused the username and PW captured from another site – likely a system that is
more vulnerable
• Insider attacks – believe it or not students and even professors are an attack
vector. Unfortunately these can be very difficult to detect or protect against
• Software vulnerabilities -
• Misconfiguration of systems
• Poor patch management – unaddressed known vulnerabilities – these are what intruders scan
for and then utilize rootkits to attack for privilege access.
• Zero day attacks
• Malicious Code (Viruses, Worms, Trojan Horses, Ransomware, etc…)
General Topics for this Talk
• Preparation
• Detection & Analysis
• Containment, Recovery, & Eradication
• Case Studies
Containment
• Must determine a strategy
• Reduce damage
• Minimize corruption of evidence
• How does it affect service availability
• How long does it take to implement
• Effectiveness
Containment
Identify:
• Evidence collection
• Methods
• Downtime
• Chain of evidence
• Minimizing corruption
• Attackers/CC
Eradication & Recovery
• Clean viruses/corruption
• Disable accounts
• Restore systems/services to operation
• Remediate vulnerabilities
• Lessons Learned
Lessons Learned
• Critical aspect of IR process
• Closes gaps in processes
• monitoring
• communication
• processes
Basic Incident Response
Incident Response STEPSCycle
Walkthrough
1. Determining if there is an incident
2. Communicate the incident
3. Determine how to handle the incident
4. Perform a detailed investigation
5. Contain the incident
6. Recovery
7. Eradicate
1. Determining if there is an incident
• Head of security or incident response team has been alerted to some kind of
unusual behavior by an individual or service.
• *remember the monitoring services you put into place in the preparation section
• Perhaps notified by some outside source that interesting things are happening.
• Might be a new vulnerability announced
• A detected incident at a different site that points to your site
• Maybe you just feel something does not look right
• Now your security or IR lead is responsible to determine if there is an incident
• Begin to gather information from the alert source
• Then perform an initial investigation
2. Communicate the Incident
• Summarize what you know and using your communication plan
begin to inform the contacts via the reporting procedures you
established as part of your incident response plan.
• Make sure everyone knows you’re on top of the incident
• Identify who is authorized to make decisions
• Identify who will interact with external groups
3. Determining How to Handle the
Incident
• Review alert details to understand what it is telling you
• Look for other alerts that may be related
• Review pertinent logs and services to validate the alert
• This is why you establish log management procedures
• This is when you start to utilize your log analysis tools
• Characterize the impact of the incident
• Has it or is it capable to causing harm to your assets?
• How much impact might it have?
• Is this a recently detected “older” event or something new and rapidly expanding?
• Identify roughly when this incident started (if possible)
• Who needs to know about and what decisions need to be made?
• Who might I need to assist with the investigation?
• Determine the attack vector
3a. Decide how to proceed with incident
response
• Understand what tools and data sources are available to you for further
understanding the scope of the incident
• system logs
• application logs
• firewall logs
• IDS alerts
• File system information
• Are there any mitigation strategies we need to deploy?
• Increase the monitoring of the affected environment to assess whether it’s being
actively used by an attacker
• Firewalling or black hole routing
3b. Additional Questions to Consider

• Decide what the response goal is for this incident

• End the attack and get back online quickly
• investigate with the idea of prosecution
• or?
• Do you already have response procedures?
• If so, are they valid for this incident?
• If the incident is still going on will we perform live analysis?
• What tools can monitor the environment
• Check your backup and restore capabilities
• Have they been followed?
• Are the backups valid?
• Make assignments on who will do what next
3c. Privacy Breaches
• Special note here - be aware if this incident:
• Involves any Personally Identifiable Information (PII) or Personal Health
Information (PHI)?
• Violates International Traffic and Arms Regulations (ITAR) compliance?
• Or other privacy policy infraction
4. Organize Incident Response Team

• Pull team together and layout the incident details

• Assign duties and review procedures
• Team communications & sharing services
• Secure Email
• Secure Wiki
• Schedule team physical collaboration spaces such as a war room
4a. Perform a Detailed Investigation

• Gather up the needed information

• Some of this will be pulling logs together
• Some of it will be analysis of systems and services for additional information
• Has anything been left behind –
• Applications and tools
• Data files

• Have files been modified?

• Have access permissions been modified?
• Did the intruder gain root
• Move forward and backwards through the logs as well as diagonally through
other logs, systems and organizations.
4b. Investigation Checklist
Action Complete

1.0 Find out if any security alerts were generated

1.1 Identify any observations that lead up to incident
1.2 Understand how the problem was initially detected
2.0 Understand the nature of the problem
3.0 Gain specific knowledge
3.1 How was the problem initially detected
4.0 Identify what has been affected
4.1 Equipment
4.2 Devices
4.3 Groups
5.0 Identify affected applications
5.1 Identify affected data
6.0 What has been done so far
6.1 Processes, services stopped/restarted
6.2 Files modified
6.3 Files deleted
6.4 Tools run
7.0 Identify containment steps that may have been taken
8.0 Review logs
9.0 Find the ingress and egress paths
10.0 Investigate if there are any risk to other organizations

*adapted from SANS Incident Response

Handbook
4c. Secure Evidence
• It may be vital to secure evidence for later disciplinary actions
• We have had requests from our university legal for incidents that are over 3
years old
• Most times we know in advance these are likely to be on-going needs for the
evidence
• Preserve evidence in a secure way
• Maintain a chain of custody
5. Containment:
Limit the damage and prevent any further
damage
• Understand what the problem is and decide how to proceed
• Let it run – this allows you to gather more information
• Isolate the service or host that has been compromised
• Bring everything down
• How do you make the decision?
• Do you have to contact people?
• What does your procedure say to do?
• Contact list
5a. Containment Options
• Short-term Containment
• Can the problem be isolated? Taken offline?
• Are all affected systems isolated from non-affected systems?
• System Backup
• Gather information from affected systems for further analysis
• Have all commands and other documentation since the incident occurred been kept up to
date?
• Long-term containment
• If the system can be taken offline, then proceed to the Eradication phase.
• If the system must remain in production, proceed with long-term containment by removing all
malware and other artifacts from affected systems. Harden the affected systems from further
attacks until an ideal circumstance will allow the affected systems to be reimaged.
6. Recovery
• Bring any affected systems, services, and applications back into
production
• Restore data and configurations
• Identify the proper time and date of the recovery media
• Verify images restored correctly and completely
• Ensure the entire incident is cleaned up
• Test all systems, services and applications if possible
• Continue to monitor for any signs of an ongoing issue
• Any more strange behavior
• Access attempts from the same attack IP(s) or account(s)
7. Is the Incident Really
Eradicated?
• How do you know the problem is really gone?
• You need to really understand what happened.
• Attack vector(s)
• Backdoors
• Changes made
• All files that have been changed need to be identified
• Close all the doors that might have been used
• Exhaustive search for any tools that might have been installed
• Add additional monitoring if none is available
Document Lessons Learned
• Document the details so you have a complete record of
everything that happened
• When was the problem was first detected and by whom?
• The scope and impact of the incident?
• What actions were taken to identify and address?
• How did you ensure it was all cleaned up?
• What actions were taken in the recovery process?
• Was the incident response plan effective? And if not what needs
to be addressed
Reassessing Preparation and
Response
Conduct a post mortem
• Delve into the who, what, how, when, and why of the incident
• Review how things could have gone better in order to improve
your processes
• Keep the meeting calm - limit the finger pointing
• Keep the meeting focused - get the information to fix the problem
and improve things
• Identify if the incident could have been identified sooner or even
prevented
The Final Analysis
At the heart of all these issues is one
important question:

Why did this happen and can we

prevent it from happening again?
A reminder about
communication…
• Don’t underestimate the importance of communication
• If you don’t tell the necessary people that you are taking action
they will assume you’re not doing anything
• (I have been bitten by this one numerous time)
• Communication suggestions
• When you have determined there has been a compromise
• When you have concluded the investigation and and planning the eradication
and recovery
• When the service or system is back in production