The capacity of a system to keep functioning in the face of hardware or software failure is called

11.2.7 Identify failure conditions

Every functional transaction must be evaluated to identify situations or conditions that may cause failure conditions. Identified failure states must be resolved by stipulating the data integrity criterion that must be interrogated to determine a failure state, and the actions to be taken when a certain state arises to complete the data processing transaction.

Some failure conditions may result in a state that cannot be resolved via automation and requires human intervention. The functional analysis effort must then address the manner by which the software will continue to operate in a degraded mode, if possible. For example, if an ATM’s supply of money has been depleted, then the withdrawal function must be temporarily suspended until the money supply if restocked. The software functions for detecting failure conditions and operating in a degraded mode must be included within the functional architecture. This includes identifying how the software product can be “informed” of the current state of data processing or system resources and how the state indicators are managed.

Potential data processing failure modes and effects must be analyzed to determine how the software product should behave in response to each failure condition. Failure modes and effects analysis (FMEA) is an engineering procedure that enables the design team to classify potential failure modes by the severity (consequences) and likelihood of the failures resulting with improved product quality and dependability. Dependability is a term that is better suited for software products than reliability. Throughout the engineering community, reliability deals with predicting the mean time between failure (MTBF) of hardware components during normal operation and provides an estimate of the expected duration life expectancy for the component. Software does not breakdown or wear out over time with use. Therefore, dependability refers to a software component’s ability to perform its function as expected under all circumstances. Dependability is a more suitable term to be used for software products due to the nature of the material of which it is comprised. If a software component fails, it is due to the software design inability to be resilient to unexpected circumstances or situations.

Software FMEA should be used to identify potential failure modes, determine their effects on the operation of the system or business process, and design response mechanisms that prevent the failure from occurring or mitigate the impact of the failure on operational performance. While anticipating every failure mode may not be possible, the development team should formulate an extensive list of potential failure modes in the following manner:

Develop software product requirements that minimize the likelihood of potential operational failures from arising.

Evaluate the requirements obtained from stakeholders in the software performance and post-development processes to ensure that those requirements do not introduce complicated failure conditions or situations.

Identify design characteristics that contribute to failure detection and minimize failure propagation throughout a data processing transaction.

Develop software test scenarios and procedures designed to exercise the software behaviors associated with failure detection, isolation, and recovery.

Identify, track, and manage potential design risks to ensure that product dependability is predictable and substantiated via the software test effort

Ensure that any failures that could occur will not result in personal injury or seriously impact the operation of the system or operational processes.

Properly used, the software FMEA provides the development team several benefits, including:

Improved software dependability and quality.

Increased customer and stakeholder satisfaction.

Identifies and eliminates potential software failure modes early in the development process when such design challenges can be cost-effectively regulated.

Emphasized failure detection and preventive measures.

Provides a focus for improved software test coverage.

Minimizes late design changes and their associated cost and schedule impacts.

Improves teamwork and idea exchange among development team members.

A complete FMEA for a software product should contend with failures arising from the computing environment hardware, external systems, and data processing transactions, and their effects on the final system or operational processes. The software FMEA procedures should adhere to the following steps, adapted from IEC 608121:

Define the software boundaries for analysis (accomplished during computational requirements analysis).

Understand the software requirements, functionality, and performance.

Develop the functional architecture representations (hierarchical decomposition and behavioral views).

Identify functional failure modes and summarize failure effects.

Develop criteria for successful failure detection, isolation, and recovery.

Report findings.

Software

This brings us to software failures. The most serious type of software failure is an operating system crash, since it stops the entire computer system. Since many software problems are transient, a reboot often repairs the problem. This involves rebooting the operating system, running software that repairs disk state that might have become inconsistent due to the failure, recovering communications sessions with other systems in a distributed system, and restarting all the application programs. These steps all increase the MTTR and therefore reduce availability. So they should be made as fast as possible. The requirement for faster recovery inspired operating systems vendors in the 1990s to incorporate fast file system recovery procedures, which was a major component of operating system boot time. Some operating systems are carefully engineered for fast boot. For example, highly available communication systems have operating systems that reboot in under a minute, worst case. Taking this goal to the extreme, if the repair time were zero, then failures wouldn’t matter, since the system would recover instantaneously, and the user would never know the difference. Clearly reducing the repair time can have a big impact on availability.

Some software failures only degrade a system’s capabilities, not cause it to fail. For example, consider an application that offers functions that require access to a remote service. When the remote service is unavailable, those functions stop working. However, through careful application design, other application functions can still be operational. That is, the system degrades gracefully when parts of it stop working. A real example we know of is an application that used a TP database and a data warehouse, where the latter was nice to have but not mission-critical. The application was not designed to degrade gracefully, so when the data warehouse failed, the entire application became unavailable, which caused a large and unnecessary loss of revenue.

When an application process or database system does fail, the failure must be detected and the application or database system process must be recovered. This is where TP-specific techniques become relevant.

2.5.3 Commit Processing

To guard against any software failures or power failures, it is essential to backup database. This ensures the durability and consistency guarantees. As the backup is done regularly, it is prudent to keep a log of transaction activity. Owing to volatility of the memory, the log must be stored on stable storage. Logging of data to persistent storage impacts the response times. Each transaction has to wait till the log is committed to storage. And the throughput is also affected if log becomes bottleneck. This is the only disk operation required by the IMDBs [34,35]. The following are few solutions that have been suggested and some of them are used by major IMDBs:

Use stable storage for logging: A stable storage is used to hold portion of the log. As a transaction commits, the log information is written to stable storage. A dedicated processor asynchronous writes the data to log disks. Response time improves even though log bottleneck is still present. A modern variation of this approach is to use nonvolatile RAM storage. This device offers lesser read/write than disk storages.

Precommitting transactions: Transaction locks are released as soon as its log record is placed in the log. This reduces the blocking delays for other concurrent transactions.

Group commits: A log record is not sent to disk as soon as transaction commits. Instead the log records of several transactions are accumulated in memory. All these records are flushed to log disk in a single-disk operations. This reduces the number of disk operations. The log bottleneck and blocking wait are also eliminated. The downside is that in case of software failures some transactions might be lost.

Asynchronous commits: The log records are flushed to database asynchronously. The transactions need not wait for log to be committed to disk. As with the group commit, certain transactions might be lost in case of failures.

7.12 Over-the-Air Network Reprogramming

After a field deployment, the network functionality may need improvement or fix new software failures. Thus, it is important to support remote software upgrades. Deluge [16] is the de facto network reprogramming protocol that provides an efficient method for disseminating code update over the wireless network and having each node program itself with the new image. Deluge originally does not support the iMote2 platform, and it is not trivial to port it to support the iMote2 platform. Deluge was also improved to ensure that it could handle some adverse situations. Following are such important features. (1) Image integrity verification: If a node reboots during the download phase, it has to be ensured that it correctly resumes the download. To address this issue, a mechanism is implemented where the image integrity during startup is verified. If the image has been completely downloaded, then the system continue with the normal operations; otherwise it erases the entire downloaded image and reset the meta data to enable a fresh re-download. (2) Image version consistency: The original Deluge is based on sequence number. However, if the gateway lost track of the sequence number and did not use a higher sequence number, then Deluge will not respond to new request of code update. This problem was fixed by using the compilation timestamp to differentiate new image from old image.

20.4 Summary and Outcomes of Chapter 20

•

Applications with long execution times run a significant risk of encountering a hardware or software failure before completion.

Long execution times also frequently violate supercomputer usage policies where a maximum wallclock limit for a simulation is established.

The consequences of a hardware or software failure can be very significant and costly in terms of time lost and computing resources wasted for long-running jobs.

At designated points during the execution of an application on a supercomputer, the data necessary to allow later resumption of the application at that point in the execution can be output and saved. This data is called a checkpoint.

Checkpoint files help mitigate the risk of a hardware or software failure in a long-running job.

Checkpoint files also provide snapshots of the application at different simulation epochs, help in debugging, aid in performance monitoring and analysis, and can help improve load-balancing decisions for better distributed-memory usage.

In HPC applications, two common strategies for checkpoint/restart are employed: system-level checkpoint and application-level checkpointing.

System-level checkpointing requires no modifications to the user code but may require loading a specific system-level library.

System-level checkpointing strategies center on full memory dumps and may result in very large checkpoint files.

Application-level checkpointing requires modifications to the user code. Libraries exist to assist this process.

Application-level checkpoint files tend to be more efficient, since they only output the most relevant data needed for restart.

18.10 Conclusions

In this chapter, we have presented a root cause analysis framework that allows the identification of possible root causes of software failures that stem either from internal faults or from the actions of external agents. The framework uses goal models to associate system behavior with specific goals and actions that need be achieved in order for the system to deliver its intended functionality or meet its quality requirements. Similarly, the framework uses antigoal models to denote the negative impact the actions of an external agent may have on specific system goals. In this context, goal and antigoal models represent cause and effect relations and provide the means for denoting diagnostic knowledge for the system being examined. Goal and antigoal model nodes are annotated with pattern expressions. The purpose of the annotation expressions is twofold: first, to be used as queries in an information retrieval process that is based on LSI to reduce the volume of the logs to be considered for the satisfaction or denial of each node; and second, to be used as patterns for inferring the satisfaction of each node by satisfying the corresponding precondition, occurrence, and postcondition predicates attached to each node. Furthermore, the framework allows a transformation process to be applied so that a rule base can be generated from the specified goal and antigoal models.

The transformation process also allows the generation of observation facts from the log data obtained. The rule base and the fact base form a complete, for a particular session, diagnostic knowledge base. Finally, the framework uses a probabilistic reasoning engine that is based on the concept of Markov logic and MLNs. In this context, the use of a probabilistic reasoning engine is important as it allows inference to commence even in the presence of incomplete or partial log data, and ranking the root causes obtained by their probability or likelihood values.

The proposed framework provides some interesting new points for root cause analysis systems. First, it introduces the concept of log reduction to increase performance and is tractable. Second, it argues for the use of semantically rich and expressive models for denoting diagnostic knowledge such as goal and antigoal models. Finally, it argues that in real-life scenarios root cause analysis should be able to commence even with partial, incomplete, or missing log data information.

New areas and future directions that can be considered include new techniques for log reduction, new techniques for hypothesis selection, and tractable techniques for goal satisfaction and root cause analysis determination. More specifically, techniques that are based on complex event processing, information theory, or PLSI could serve as a starting point for log reduction. SAT solvers, such as Max-SAT and Weighted Max-SAT solvers, can also be considered as a starting point for generating and ranking root cause hypotheses on the basis of severity, occurrence probability, or system domain properties. Finally, the investigation of advanced fuzzy reasoning, probabilistic reasoning, and processing distribution techniques such as map-reduce algorithms may shed new light on the tractable identification of root causes for large-scale systems.

2.4.2 Inadequacies in Networks Today

In Chapter 1 we discussed the evolution of networks that allowed them to survive catastrophic events such as outages and hardware or software failures. In large part, networks and networking devices have been designed to overcome these rare but severe challenges. However, with the advent of data centers, there is a growing need for networks to not only recover from these types of events but also to be able to respond quickly to frequent and immediate changes.

Although the tasks of creating a new network, moving a new network, and removing a network are similar to those performed for servers and storage, doing so requires work orders, coordination between server and networking administrators, physical or logical coordination of links, network interface cards (NICs), and ToR switches, to name a few. Figure 2.4 illustrates the elapsed time in creating a new instance of a VM, which is on the order of minutes, compared to the multiple days that it may take to create a new instance of a network. This is because the servers are virtualized, yet the network is still purely a physical network. Even when we are configuring something virtual for the network, such as a VLAN, making changes is more cumbersome than in their server counterparts. In Chapter 1 we explained that although the control plane of legacy networks had sophisticated ways of autonomously and dynamically distributing layer two and layer three states, no corresponding protocols exist for distributing the policies that are used in policy-based routing. Thus, configuring security policy, such as ACLs or virtualization policy such as to which VLAN a host belongs, remains static and manual in traditional networks. Therefore, the task of reconfiguring a network in a modern data center does not take minutes but rather days. Such inflexible networks are hindering IT administrators in their attempts to automate and streamline their virtualized data center environments. SDN holds the promise that the time required for such network reconfiguration be reduced to the order of minutes, such as is already the case for reconfiguration of VMs.

Data cycle

Data, like most other business assets, has a life cycle (see Figure 5.1) governing its transition in terms of substance and use. Its stage of evolution also determines its value at any given time.

5.1 How to Maximize Resource Availability in the Cloud?

One of the main issues of cloud computing is the availability of resources in the cloud. Several factors should be addressed to ensure the high availability of the application on the cloud, including hardware and software failures, specifically single points of failures, network vulnerabilities, power outages, conservations, or denial of service invasions. An efficient deprovisioning of resource reduces the low availability of the services (eg, application) on the cloud. We summarize some of the solutions that address resource availability problem.

Armbrust et al. [1] discussed how availability is one of the top obstacles in cloud computing. Although the authors have analyzed the high-availability approaches used by cloud providers, they do not discuss any existing solutions that enable a cross-cloud resource provisioning model. Cloud providers (eg, GCP, Amazon Web Services, Microsoft Azure, GoGrid, and Rackspace) lack a common platform for cross-cloud provisioning.

Galante et al. [13] pointed out that the use of multiple clouds is one of the solutions to the resources availability issue. The CCIF [44] attempted to overcome the limitations of interoperability standards among various cloud platforms by introducing an open and standardized cloud interface to unify different cloud platform APIs. IEEE also has a similar project P2301 (Guide for Cloud Portability and Interoperability) [45], where the purpose of the project is to guide cloud users to develop and use cloud services using a common standard to increase portability and interoperability among different providers.

Buyya et al. [51] proposed architecture for cloud federation to integrate distributed clouds to meet business requirements. A federated cloud enables cloud providers to manage and deploy several external and internal cloud computing services. For example, it allows fulfilling the exceeding demands of a cloud by renting resources from other cloud service providers.

Pawluk et al. [52] developed an initial step toward the idea of a cloud of clouds [53] to enable an automated cross-cloud resource provisioning platform. They proposed a broker service that enables cross-cloud to facilitate the construction of application topology platform and runtime modification according to the objectives of an application deployer. In most cases, the assumption of acquired resources should be homogeneous. However, the authors eliminated this assumption to support an actual intercloud platform. An open project [54] for cross-cloud acquirement and VM management enables a developer to select which of the available clouds to use, whereas Pawluk et al. [52] select the available clouds to use for the developer.

An attempt to define unified access to multiple clouds via a unified API has been advanced by Refs. [44,45,54], whereas Refs. [51,52] introduced a methodology for the federation of cloud computing platforms. Both works do not discuss any implementation to automate the resource acquirement procedure via unified access to multiple clouds. References [44,45] presented limited support for interoperability and intercloud interfaces, but they did not provide any mechanism (eg, implementation) to automate the resource acquirement procedure via unified access to multiple clouds through APIs.

These studies have proposed solutions to the problems associated with resource availability of cloud systems. These solutions are very valuable to ensure the high availability of the services on the cloud systems to maintain elasticity.

17.8 Summary

In this chapter, we have discussed various aspects of improving the availability of the data warehouse. Oracle Database 10g provides features such as RAC to provide fault-tolerant operation in the face of hardware and software failures and allows logical and physical reorganization of data without requiring downtime. We discussed the role of disaster recovery in a data warehouse and also how the data warehouse fits into an enterprise disaster recovery strategy using Data Guard. Finally, we also touched upon the subject of information life-cycle management, which ensures that the data warehouse will continue to be cost effective even as data sizes grow.