10.5. Restoring and Recovering Resources

During active incident response, there should be considerations toward working on recovery. The actual breach will dictate the course of recovery. This is when having backups or offline, redundant systems will prove invaluable. For recovery, the response team should be planning to bring back online any downed systems or applications, such as authentication servers, database servers, and any other production resources.

Having production backup hardware ready for use is highly recommended, such as extra hard drives, hot-spare servers, and the like. Ready-made systems should have all production software loaded and ready for immediate use. Perhaps only the most recent and pertinent data would need to be imported. This ready-made system should be kept isolated from the rest of the potentially affected network. If a compromise occurs and the backup system is a part of the network, then the purpose of having a backup system is defeated.

System recovery can be a tedious process. In many instances there are two courses of action from which to choose. Administrators can perform a clean reinstallation of the operating system followed by restoration of all applications and data. Alternatively, administrators can patch the system of the offending vulnerability and bring the affected system(s) back into production.

10.5.1. Reinstalling the System

Performing a clean reinstallation ensures that the affected system will be cleansed of any trojans, backdoors, or malicious processes. Reinstallation also ensures that any data (if restored from a trusted backup source) is cleared of any malicious modification. The drawback to total system recovery is the time involved in rebuilding systems from scratch. However, if there is a hot backup system available for use where the only action to take is to dump the most recent data, then system downtime is greatly reduced.

10.5.2. Patching the System

The alternate course to recovery is to patch the affected system(s). This method of recovery is more dangerous to perform and should be undertaken with great caution. The danger with patching a system instead of reinstalling is determining whether or not you have sufficiently cleansed the system of trojans, holes, and corrupted data. If using a modular kernel, then patching a breached system can be even more difficult. Most rootkits (programs or packages that a cracker leaves to gain root access to your system), trojan system commands, and shell environments are designed to hide malicious activities from cursory audits. If the patch approach is taken, only trusted binaries should be used (for example, from a mounted, read-only CD-ROM).