Review and Monitoring

Introduction

Why Carry out Monitoring and Evaluation?

The simple answer is that only through monitoring and evaluation can we ensure that the risk has been reduced and the organization can deliver its objectives within acceptable levels of risk. Only with effective monitoring of implementation and evaluation of the results do we know whether the risk management decisions have been genuinely effective and achieved the desired, predicted results.

Review enhances the effectiveness of the SRM process by establishing a link between the past, present, and future. It extracts knowledge of the past and ongoing security risk management information to fine-tune, re-orientate and plan.

Purpose

The purpose of monitoring and evaluation is to improve the effectiveness of security risk management measures and increasingly enable business objectives at an acceptable level of risk.

• Monitoring. Monitoring aims primarily to provide security advisers and the SMT early indications of progress, or lack thereof, in implementation of the security risk management measures. It is an objective process of checking activities. Monitoring is what was previously generally referred to as “compliance”.
• Evaluation. Evaluation is assessing the progress and effectiveness in achieving the desired SRM aim. It is ensuring that the implemented activities are leading to the desired and expected outcomes. Between monitoring and review, the review is the most important if we are to ensure that SRM is fit for purpose.

Within the SRM process, there are inputs, outputs, outcomes, and impacts. The SRM aim is focused on the outcome – enabling the organization project delivery at an acceptable level of risk – but only has full control of the inputs and outputs:

INPUT	OUTPUTS	OUTCOMES	IMPACT
Information SRM Decisions Funds Expertise	Security Plan MOSS Compliance Balance Acceptable Risk Decisions Reduced Risk	Reduced Risk Increased project delivery and achievement of business objectives	Delivery of business Objectives

You Monitor the implementation of the Outputs You Evaluate the Outcomes

Security Risk Management Areas Monitoring and Review

Monitoring Tools.

The organization should have several effective formal tools and forums for monitoring the implementation of agreed SRM decisions. Here is a list of the most common:

• SMT – The security briefings to the SMT allow the routine monitoring of the implementation of the agreed SRM measures. It is within this forum that implementation is monitored in line with the accountabilities and responsibilities of the SMT.
• Security Cell – The security cell is the center for the application of security expertise in a country to ensure that not only the correct advice is given to the SMT but that those decisions are implemented.
• Regional desks and Operational Support Section (OSS) – The regional desks and OSS constantly monitor the SRM process in the countries and projects assigned to them, and where necessary recommend remedial action. Where necessary, their observations can be elevated to a higher level, and ultimately to the board of directors as required.
• Alternative monitoring – Depending on the situation and needs in the designated area, a security adviser, in consultation with the SMT, may implement alternative monitoring. These may include:
  • Spot Checks on staff, residences, offices, vehicles, guard posts, and any other security measure. These spot checks can reveal where security risk management measures have, or have not, been implemented and can indicate which area requires additional focus;
  • Surveys. Surveys of staff, partners or project managers often reveal details of implementation or non-implementation of security risk management measures or gaps in knowledge of policies and procedures that are not apparent in other forms of monitoring.
  • External monitoring. External checks on implementation either internally in the country or the security cell carry out checks and provide supportive feedback. Alternatively, neighboring SRM Areas can check each other.

Apart from the compulsory monitoring through the SMT and Mandatory Self-Assessment, it is the decision of the country senior manager in consultation with the SMT as advised by the Security Advisor in consultation with the Security Cell on what type of monitoring best ensures that security risk management measures are being implemented in a timely and effective manner.

Evaluation Tools.

The principal driver for the evaluation of the effectiveness of the security risk management in the Designated Area is the regular, periodic review of the SRM outcomes. However, restarting the SRM process is not necessarily an active evaluation of effectiveness. Alternatives for evaluation of effectiveness include:

• Assessment of Indicators. Since the aim is to enable acceptably safe business objective delivery, the assessment of security and delivery indicators can be extremely revealing. For example, a comparison of incidents affecting the organization before and after the implementation of the new security risk management measures is a crude indicator of reduction of risk. A simple comparison of the number of activities completed or staff deployed to the field before and after the new security risk management measures can indicate whether organizational delivery is being better enabled. Analysis can be as simple or complex as desired, but it is important to ensure that all factors have been considered. Any system is interconnected and the context of a single indicator often changes over a period of comparison. For example, it may be found that despite the implementation of new security risk management measures, incidents affecting the organization have gone up by 10%. However, if more effective decisions on acceptably safe project delivery have been taken, there may be an increase of 80% in the number of activities and staff deployed. In this case, the increase of 10% in incidents while there has been an 80% increase in activities, in fact, shows that the security risk management measures have been very effective while enabling increased business objective delivery.
• Client Surveys. Surveys of clients, partners, projects, and activities, often reveal details on the effectiveness of security risk management measures or gaps in policies and procedures that are not apparent in other forms of monitoring.
• Exercises. Exercises are useful tools for training and validating security risk management measures. Security staff should conduct exercises to train and also to test plans and capabilities while promoting various roles and responsibilities during a crisis. Successfully conducting an exercise involves considerable coordination among All key stakeholders and managers. There are some key stages essential to conducting successful exercises:
  1. Establish a foundation: Create a base of support from the appropriate entities and senior managers, develop a timeline for training and exercising including milestones, identify an exercise planning team, schedule planning meetings. Training and exercises cost resources so you almost always need support from management and you need to demonstrate utility.
  2. Design and development: Building on the exercise foundation, establish SMART objectives (what are you exercising?), scenario design, documentation, logistics and develop an evaluation (how will you measure success or failure?) and improvement methodology.
  3. Conduct the exercise: This includes the set-up, briefings, facilitation, control, evaluation and wrap-up of activities.
  4. Evaluation: This includes a formal evaluation of the exercise, an integrated analysis, an after-action report, and an improvement plan. Recommendations and need to be implemented and tracked throughout the process.
  5. Improvement Planning: corrective actions identified in the evaluation phase are assigned with due dates to responsible parties, tracked to implementation and validated in the following exercises.

Exercise Cycle

The following diagram illustrates the cyclical nature of process improvement using training and exercising.

Exercise Types

Exercises can be broadly categorized as discussion-based or operation based, and are appropriate at different levels of your planning. Avoid the temptation to jump into operation based exercises too early; people need to learn their roles and gain confidence and knowledge.

• Discussion-based exercises familiarize participants with current plans and procedures, policies, agreements, or may be used to develop new procedures. Discussion-based exercises include:
  • Seminars use different strategies such as lectures, panel discussions, case studies, etc. They are informal discussions based on policies, procedures, protocols, concepts, resources, and ideas.
  • Workshops are more participative and are effective for team building, problem-solving, information sharing and brainstorming.
  • Tabletop exercises (TTXs) consist of informal facilitated discussions of simulated emergencies among key staff. The purpose of a TTX is to test existing plans without incurring costs associated with deploying resources. Most issues involving security risk measures can be resolved using TTXs.
  • Games such as red team exercises (Red Teaming) are a fairly advanced form of testing assessments, plans, and procedures. Red team exercising is normally associated with assessing vulnerabilities and limitations of systems or structures and is well designed for testing the effectiveness and vulnerability reduction of security risk management measures. An independent group challenges an organization to improve its effectiveness by actively looking for vulnerabilities in a system and suggesting methods of exploiting the vulnerabilities. Key in red team exercises the application of realistic capabilities, intents and limiting factors of the threat e.g. by the independent group. There is no point in a Red Team exploiting vulnerabilities to IEDs when no threat actors in the designated area have demonstrated any intent or capability of using IEDs. An unlimited Red Team will always win.
• Operations-based exercises validate plans, policies, agreements, and procedures; clarify roles and responsibilities and identify resource gaps in an operational environment. Operational-based exercises include:
  • Drills are coordinated activities that test a specific operation or function. Drills are for skills development and skill-maintenance on new equipment or procedures.
  • Functional exercises (FEs) or command post exercises examine and validate coordination mechanisms, communication and command, and control between different operational entities. FEs are highly stressful and involve the notional deployment of staff and resources in real-time. FEs are great for testing coordination between Crisis Management Teams (CMT), Security Management Teams (SMT) and HQ.
  • Full-Scale Exercises (FSEs) replicate a real-world response with the actual deployment of resources. FSEs are resource costly but essential in the later stages of your testing plan.

So what exercise and when? The following diagram illustrates points in your planning process when the different types of exercises are most appropriate.

Remember you want to build capacity and confidence, so increase complexity as capacity grows.

Conclusion

The output of monitoring provides oversight of the implementation of the security risk management measures. The output of review and evaluation informs the effectiveness of security risk management measures and therefore informs the vulnerability assessments.