Human-centric training and assessment for cyber situation awareness
Date
2015
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Cyber attacks have been increasing significantly in both number and complexity, prompting the need for better training of cyber defense analysts. One issue with existing cyber security training is that it relies mostly on lecture-style instructions without much hand- on experience. Thus, need a training solution that provides a realistic, human-in-the-loop environment for cyber analysts to explore, collaborate, and interact for effective learning.
Situation Awareness (SA) means the comprehension and perception of environmental elements with respect to time and space. Cyber Situation Awareness (CSA) is SA extended to the cyber domain. During CSA, the cyber analysts need to understand the meaning of the observations and be able to project the impact of the observations to the system. In this proposal, we describe a Cyber Situation Awareness training and assessment system with the purpose of teaching and measuring individual and team cyber situation awareness within the cyber defense context, as well as incorporating various technologies to enhance the cyber analysts’ learning process.
To conduct effective cyber security training, it becomes essential to design realistic exercise lesson plans. Accurate identification of experts’ cognitive processes through Cognitive Task Analysis can be adapted into training materials to teach novice cyber analysts (or trainees in this proposal) how to think and act like an expert during defense. In order to solve the information overload challenge faced by trainees, we identify and design watch list statistics, which allows trainees to tailor their own watch list statistics and triggering thresh- old conditions in order to recognize cyber attacks faster. The speed with which a trainee can recognize, analyze, and respond to attacks is critical as it will limit the damage and lower the cost of recovery. Therefore, we evaluate trainees’ performance based on their response time comparing with estimated attack ground truth timeline. We also devise scoring algorithms to calculate trainees’ performance scores according to the weighted functions combing all performance metrics.
Then, as training is an iterative process, the assessment component not only assesses the knowledge gained by the cyber analysts, but also adjusts the difficulty of training lessons accordingly based on trainees’ performance. Nevertheless, quantifying difficulty level of training lesson scenario is an important but difficult task. While standard techniques exist for measuring the relative difficulty to exploit individual vulnerability, it is challenging to answer the fundamental question whether one scenario containing several vulnerabilities is more difficult than another one. Based on causal relationships between vulnerabilities in attack graph, we apply Bayesian Reasoning to aggregate individual vulnerabilities into a probabilistic value representing the attackers’ success likelihood to achieve the attack goal. Based on the quantified probability of achieving attack goal, the lesson’s difficulty-level is categorized accordingly.
Furthermore, complex and dynamically changing task such as cyber defense often requires the effective coordination of a team of cyber analysts. Cyber analysts need to work collaboratively as a team at different levels and different parts of system. Each team member collects data, generates its own awareness for the cyber situation, and shares with other team members to get the comprehensive understanding of the overall situation for the decision making purpose. Since each team member may have his/her own personal expertise knowledge, experience, and opinions, it is hard for the entire team to make consensus decision when having conflicting judgments. Considering human cyber analysts tend to use ambiguous linguistic language to express their own cyber situation awareness during the team discussion process, we design a fuzzy set based method to facilitate cyber analysts to quantify their thoughts and make consensus decision that is most acceptable by the entire team.
Finally, we investigate real-time purpose sensors based information fusion for Advanced
Persistent Threats (APTs) detection. As human cyber analysts have to examine huge
amount of data such as system logs, configuration files, traffic logs, IDS log, and audit logs
in order to identify potential threats. Thus, they would be soon overwhelmed by tremendous
xvi
data and forced to ignore potentially significant evidences introducing errors in the detection
process. Furthermore, cyber security attack and anomaly detection techniques suffer from
their reliance on known malicious signatures or unusual conditions that warrant further investigation.
The use of signature-based detection cannot effectively eliminate false negatives
when dealing with Advanced Persistent Threats, since the financial resources and time available
to APTs allows the use of previously unknown ‘zero-day’ attack vectors. The designed
information fusion system reduces an operator burden to handle false positives and reduces
time to detection while using noisy/high false positive inputs. We design and develop hostand
network- based purpose sensors and places them within the network and individual hosts
to provide real-time purpose and correlation inputs and then use this information combined
with network-specific knowledge to create a dynamic set of event threads that, when touched
by a given alert received from traditional intrusion detection systems (IDSs) such as Snort
and OSSEC, allows the immediate identification of the context surrounding the alert and thus
the automatic calculation of the alert’s legitimacy and severity.