> Consider a mobile app that you make use of on your personal phone. List and discuss 3 to 5 security risks that developers should consider when developing apps like this one.
Outline your plan addressing these issues and other issues. Need a 7-9 page APA-formatted paper with minimum of 9 peer-reviewed citations. Need proper introduction and conclusion included.
NO AI WORK!
310
C H A P T E R
What is it? The answer isn’t as easy as you might think. You know quality when you see it, and yet, it can be an elusive thing to define. But for computer software, quality is some- thing that we must define, and that’s what we’ll do in this chapter.
Who does it? Everyone—software engineers, managers, all stakeholders—involved in the software process is responsible for quality.
Why is it important? You can do it right, or you can do it over again. If a software team stresses quality in all software engineering ac- tivities, it reduces the amount of rework that it must do. That results in lower costs, and more importantly, improved time to market.
What are the steps? To achieve high-quality software, four activities must occur: proven software engineering process and practice, solid project management, comprehensive quality control, and the presence of a quality assurance infrastructure.
What is the work product? Software that meets its customer’s needs, performs accu- rately and reliably, and provides value to all who use it.
How do I ensure that I’ve done it right? Track quality by examining the results of all quality control activities, and measure quality by examining errors before delivery and defects released to the field.
Q u i c k L o o k
cost of quality . . . . . . . . . . . . . . . . . . . . . . . . . 317 good enough . . . . . . . . . . . . . . . . . . . . . . . . . . 316 liability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .320 machine learning . . . . . . . . . . . . . . . . . . . . . . .322 management actions . . . . . . . . . . . . . . . . . . . 321 quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
quality dilemma . . . . . . . . . . . . . . . . . . . . . . . . 315 quality dimensions . . . . . . . . . . . . . . . . . . . . . 314 quality factors . . . . . . . . . . . . . . . . . . . . . . . . . 312 quantitative quality assessment . . . . . . . . . . . 315 risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .320
k e y c o n c e p t s
15 Quality Concepts
The drumbeat for improved software quality began in earnest as software became increasingly integrated in every facet of our lives. By the 1990s, major corpora- tions recognized that billions of dollars each year were being wasted on software that didn’t deliver the features and functionality that were promised. Worse, both government and industry became increasingly concerned that a major software fault might cripple important infrastructure, costing tens of billions more. By the turn of the century, CIO Magazine trumpeted the headline, “Let’s Stop Wasting $78 Billion a Year,” lamenting the fact that “American businesses spend billions for software that doesn’t do what it’s supposed to do” [Lev01]. Sadly, at least one survey of the state of software quality practices done in 2014 suggests that maintenance and software evolution activities make up as much as 90 percent of the total software development costs [Nan14]. Poor software quality caused by a rush to release products without adequate testing continues to plague the soft- ware industry.
CHAPTER 15 QUALITY CONCEPTS 311
Today, software quality remains an issue, but who is to blame? Customers blame developers, arguing that sloppy practices lead to low-quality software. Developers blame customers (and other stakeholders), arguing that irrational delivery dates and a continuing stream of changes force them to deliver software before it has been fully validated. Who’s right? Both—and that’s the problem. In this chapter, we consider software quality as a concept and examine why it’s worthy of serious consideration whenever software engineering practices are applied.
15.1 Wh at is Qua L i t y? In his mystical book Zen and the Art of Motorcycle Maintenance, Robert Persig [Per74] commented on the thing we call quality:
Quality . . . you know what it is, yet you don’t know what it is. But that’s self-contradictory. But some things are better than others; that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There’s nothing to talk about . . . Obviously some things are better than others . . . but what’s the betterness? . . . So round and round you go, spinning mental wheels and nowhere finding anyplace to get traction. What the hell is Quality? What is it?
Indeed—what is it? At a somewhat more pragmatic level, David Garvin [Gar84] of the Harvard Busi-
ness School suggests that “quality is a complex and multifaceted concept” that can be described from five different points of view. The transcendental view argues (like Persig) that quality is something you immediately recognize but cannot explicitly define. The user view sees quality in terms of an end user’s specific goals. If a prod- uct meets those goals, it exhibits quality. The manufacturer’s view defines quality in terms of the original specification of the product. If the product conforms to the spec, it exhibits quality. The product view suggests that quality can be tied to inherent characteristics (e.g., functions and features) of a product. Finally, the value-based view measures quality based on how much a customer is willing to pay for a product. In reality, quality encompasses all these views and more.
Quality of design refers to the characteristics that designers specify for a product. The grade of materials, tolerances, and performance specifications all contribute to the quality of design. As higher-grade materials are used, tighter tolerances and greater levels of performance are specified, and the design quality of a product increases if the product is manufactured according to specifications.
In software development, quality of design encompasses the degree to which the design meets the functions and features specified in the requirements model. Quality of conformance focuses on the degree to which the implementation follows the design and the resulting system meets its requirements and performance goals.
But are quality of design and quality of conformance the only issues that software engineers must consider? Robert Glass [Gla98] argues that a more “intuitive” relation- ship is in order:
user satisfaction = compliant product + good quality + delivery within budget and schedule
312 PART THREE QUALITY AND SECURITY
At the bottom line, Glass contends that quality is important, but if the user isn’t satisfied, nothing else really matters. DeMarco [DeM98] reinforces this view when he states: “A product’s quality is a function of how much it changes the world for the better.” This view of quality contends that if a software product provides substantial benefit to its end users, they may be willing to tolerate occasional reliability or per- formance problems. A modern view of software quality requires attention to customer satisfaction as well as conformance to the product requirements [Max16].
15.2 so f t Wa r e Qua L i t y
Even the most jaded software developers will agree that high-quality software is an important goal. But how do we define software quality? In the most general sense, software quality can be defined as: An effective software process applied in a manner that creates a useful product that provides measurable value for those who produce it and those who use it.
There is little question that the preceding definition could be modified or extended and debated endlessly. For the purposes of this book, the definition serves to empha- size three important points:
1. An effective software process establishes the infrastructure that supports any effort at building a high-quality software product. The management aspects of process create the checks and balances that help avoid project chaos—a key contributor to poor quality. Software engineering practices allow the developer to analyze the problem and design a solid solution—both critical to building high-quality software. Finally, umbrella activities such as change management and technical reviews have as much to do with quality as any other part of software engineering practice.
2. A useful product delivers the content, functions, and features that the end user desires, but as important, it delivers these assets in a reliable, error-free way. A useful product always satisfies those requirements that have been explicitly stated by stakeholders. In addition, it satisfies a set of implicit requirements (e.g., ease of use) that are expected of all high-quality software.
3. By adding value for both the producer and user of a software product, high- quality software provides benefits for the software organization and the end- user community. The software organization gains added value because high-quality software requires less maintenance effort, fewer bug fixes, and reduced customer support. This enables software engineers to spend more time creating new applications and less on rework. The user community gains added value because the application provides a useful capability in a way that expe- dites some business process. The end result is (1) greater software product rev- enue, (2) better profitability when an application supports a business process, and/or (3) improved availability of information that is crucial for the business.
15.2.1 Quality Factors Several software quality models and standards have been proposed in the software engineering literature. David Garvin [Gar84] writes that quality is a multifaceted
CHAPTER 15 QUALITY CONCEPTS 313
phenomena and requires the use of multiple perspectives to assess it. McCall and Walters [McC77] proposed a useful way to think about and organize factors affecting software quality. Their software quality factors (shown in Figure 15.1) focus on three software product aspects: its operation characteristics, its ability to undergo change, and its adaptability to new environments. McCall’s quality factors provide a basis for engineering software that provides high levels of user satisfaction by focusing on the overall user experience delivered by the software product. This cannot be done unless developers ensure that the requirements specification is correct and that defects are removed early in the software development process [Max16].
The ISO 25010 quality model is the newest standard (created in 2011 and revised in 2017).1 This standard defines two quality models. The quality in use model describes five characteristics that are appropriate when considering using the product in a particular context (e.g., using the product on a specific platform by a human). The product quality model describes eight characteristics that focus on both the static and dynamic nature of computer systems.
∙ Quality in Use Model ∙ Effectiveness. Accuracy and completeness with which users achieve goals ∙ Efficiency. Resources expended to achieve user goals completely with
desired accuracy ∙ Satisfaction. Usefulness, trust, pleasure, comfort ∙ Freedom from risk. Mitigation of economic, health, safety, and environ-
mental risks ∙ Context coverage. Completeness, flexibility
∙ Product Quality ∙ Functional suitability. Complete, correct, appropriate ∙ Performance efficiency. Timing, resource utilization, capacity ∙ Compatibility. Coexistence, interoperability
Figure 15.1 McCall’s software quality factors
Product Operation
Correctness Reliability Usability Integrity Efficiency
Te sta
bilit y
Flexib
ilit y
Te
sta bilit
y
Product Revisi
on
Portability Reusability Interoperability
Product Transition
1 The ISO 25010 can be found at https://www.iso.org/standard/35733.html.
314 PART THREE QUALITY AND SECURITY
∙ Usability. Appropriateness, learnability, operability, error protection, aesthetics, accessibility
∙ Reliability. Maturity, availability, fault tolerance, recoverability ∙ Security. Confidentiality, integrity, accountability, authenticity ∙ Maintainability. Modularity, reusability, modifiability, testability ∙ Portability. Adaptability, installability, replaceability
The addition of the quality in use model helps to emphasize the importance of customer satisfaction in the assessment of software quality. The product quality model points out the importance of assessing both the functional and nonfunctional require- ments for the software product [Max16].
15.2.2 Qualitative Quality Assessment The quality dimensions and factors presented in Section 15.2.1 focus on the complete software product and can be used as a generic indication of the quality of an applica- tion. Your software team can develop a set of quality characteristics and associated questions that would probe the degree to which each factor has been satisfied.2 For example, ISO 25010 identifies usability as an important quality factor. If you were asked to evaluate a user interface and assess its usability, how would you proceed?
Although it’s tempting to develop quantitative measures for the quality factors noted in Section 15.2.1, you can also create a simple checklist of attributes that provide a solid indication that the factor is present. You might start with the sub-characteristics suggested for usability in ISO 25010: appropriateness, learnability, operability, error protection, aesthetics, and accessibility. You and your team might decide to create a user questionnaire and a set of structured tasks for users to perform. You might observe the users while they perform these tasks and have them complete the question- naire when they finish. We will discuss usability testing in more detail in Chapter 21.
To conduct your assessment, you and your team will need to address specific, measurable (or at least, recognizable) attributes of the interface. Your tasks might be focused on answering the following questions:
∙ How quickly can users determine whether the software product can be used to help them complete their task or not? (appropriateness)
∙ How long does it take users to learn how to use the system functions needed to complete their task? (learnability)
∙ Is the user able to recall how to use system functions in subsequent testing sessions without having to relearn them? (learnability)
∙ How long does it take users to complete tasks using the system? (operability) ∙ Does the system try to prevent users from making errors? (error protection) ∙ Does the system allow users to undo operations that may have resulted in
errors? (error protection) ∙ Do answers give favorable responses to questions about the appearance of the
user interface? (aesthetics)
2 These characteristics and questions would be addressed as part of a software review (Chapter 16).
CHAPTER 15 QUALITY CONCEPTS 315
∙ Does the interface conform to the expectations set forth by the golden rules from Chapter 12? (accessibility)
∙ Does the user interface conform to the accessibility checklist items required for the intended users? (accessibility)
As the interface design is developed, the software team would review the design prototype and ask the questions noted. If the answer to most of these questions is yes, it is likely that the user interface exhibits high quality. A collection of questions similar to these would be developed for each quality factor to be assessed. In the case of usability, it is always important to observe representative users interact with the system. For some other quality factors it may be important to test the software in the wild (or at least in the production environment).
15.2.3 Quantitative Quality Assessment In the preceding subsections, we have presented a set of qualitative factors for the “measurement” of software quality. The software engineering community strives to develop precise measures for software quality and is sometimes frustrated by the subjective nature of the activity. Cavano and McCall [Cav78] discuss this situation:
Subjectivity and specialization . . . apply to determining software quality. To help solve this problem, a more precise definition of software quality is needed as well as some way to derive quantitative measurements of software quality for objective analysis . . . Since there is no such thing as absolute knowledge, one should not expect to measure software quality exactly, for every measurement is partially imperfect.
Several software design defects can be detected using software metrics. The process consists of finding code fragments that suggest the presence of things like high cou- pling or unnecessary levels of complexity. Internal code attributes can be described quantitatively using software metrics. Any time software metric values computed for a code fragment fall outside the range of acceptable values, it signals the existence of a quality problem that should be investigated [Max16].
In Chapter 23, we’ll present a set of software metrics that can be applied to the quantitative assessment of software quality. In all cases, the metrics represent indirect measures; that is, we never really measure quality but rather some manifestation of quality. The complicating factor is the precise relationship between the variable that is measured and the quality of software.
15.3 th e so f t Wa r e Qua L i t y Di L e m m a
In an interview [Ven03] published on the Web, Bertrand Meyer discusses what we call the quality dilemma:
If you produce a software system that has terrible quality, you lose because no one will want to buy it. If on the other hand you spend infinite time, extremely large effort, and huge sums of money to build the absolutely perfect piece of software, then it’s going to take so long to complete and it will be so expensive to produce that you’ll be out of business any- way. Either you missed the market window, or you simply exhausted all your resources. So people in industry try to get to that magical middle ground where the product is good enough not to be rejected right away, such as during evaluation, but also not the object of so much perfectionism and so much work that it would take too long or cost too much to complete.
316 PART THREE QUALITY AND SECURITY
It’s fine to state that software engineers should strive to produce high-quality sys- tems. It’s even better to apply good practices in your attempt to do so. But the situ- ation discussed by Meyer is real life and represents a dilemma for even the best software engineering organizations. When you’re faced with the quality dilemma (and everyone is faced with it at one time or another), try to achieve balance—enough effort to produce acceptable quality without burying the project.
15.3.1 “Good Enough” Software Stated bluntly, if we are to accept the argument made by Meyer, is it acceptable to produce “good enough” software? The answer to this question must be yes, because software companies do it every day [Rod17]. They create software with known bugs and deliver it to a broad population of end users. They recognize that some of the functions and features delivered in version 1.0 may not be of the highest quality and plan for improvements in version 2.0. They do this knowing that some customers will complain, but they recognize that time to market may trump better quality as long as the delivered product is “good enough.”
Exactly what is “good enough”? Good enough software delivers high-quality func- tions and features that end users desire, but at the same time it delivers other more obscure or specialized functions and features that contain known bugs. The software vendor hopes that the vast majority of end users will overlook the bugs because they are so happy with other application functionality.
This idea may resonate with many readers. If you’re one of them, we can only ask you to consider some of the arguments against “good enough.” It is true that “good enough” may work in some application domains and for a few major software com- panies. After all, if a company has a large marketing budget and can convince enough people to buy version 1.0, it has succeeded in locking them in. As we noted earlier, it can argue that it will improve quality in subsequent versions. By delivering a good enough version 1.0, it has cornered the market.
If you work for a small company, be wary of this philosophy. When you deliver a good enough (buggy) product, you risk permanent damage to your company’s reputa- tion. You may never get a chance to deliver version 2.0 because bad buzz may cause your sales to plummet and your company to fold.
If you work in certain application domains (e.g., real-time embedded software) or build application software that is integrated with hardware (e.g., automotive software, telecommunications software), delivering software with known bugs can be negligent and open your company to expensive litigation. In some cases, it can even be criminal. No one wants good enough aircraft avionics software! Ebert writes that the software process model should provide clear criteria to guide devel- opers to determine what is really “good enough” for the intended application domain [Ebe14].
So, proceed with caution if you believe that “good enough” is a shortcut that can solve your software quality problems. It can work, but only for a few and only in a limited set of application domains.3
3 A worthwhile discussion of the pros and cons of “good enough” software can be found in [Bre02].
CHAPTER 15 QUALITY CONCEPTS 317
15.3.2 The Cost of Quality The argument goes something like this: We know that quality is important, but it costs us time and money—too much time and money to get the level of software quality we really want. On its face, this argument seems reasonable (see Meyer’s comments earlier in this section). There is no question that quality has a cost, but lack of qual- ity also has a cost—not only to end users who must live with buggy software, but also to the software organization that has built and must maintain it. The real question is this: Which cost should we be worried about? To answer this question, you must understand both the cost of achieving quality and the cost of low-quality software.
The cost of quality includes all costs incurred in the pursuit of quality or in per- forming quality-related activities and the downstream costs of lack of quality. To understand these costs, an organization should collect metrics to provide a baseline for the current cost of quality, identify opportunities for reducing these costs, and provide a normalized basis of comparison. The cost of quality can be divided into costs associated with prevention, appraisal, and failure.
Prevention costs include (1) the cost of management activities required to plan and coordinate all quality control and quality assurance activities, (2) the cost of added technical activities to develop complete requirements and design models, (3) test plan- ning costs, and (4) the cost of all training associated with these activities. Don’t be afraid to incur significant prevention costs. Rest assured that your investment will provide an excellent return.
Appraisal costs include activities to gain insight into product condition the “first time through” each process. Examples of appraisal costs include: (1) the cost of conducting technical reviews (Chapter 16) for software engineering work products, (2) the cost of data collection and metrics evaluation (Chapter 23), and (3) the cost of testing and debugging (Chapters 19 through 21).
Failure costs are those that would disappear if no errors appeared before shipping a product to customers. Failure costs may be subdivided into internal failure costs and external failure costs. Internal failure costs are incurred when you detect an error in a product prior to shipment. Internal failure costs include: (1) the cost required to perform rework (repair) to correct an error, (2) the cost that occurs when rework inadvertently generates side effects that must be mitigated, and (3) the costs associated with the collection of quality metrics that allow an organization to assess the modes of failure. External failure costs are associated with defects found after the product has been shipped to the customer. Examples of external failure costs are complaint resolution, product return and replacement, help line support, and labor costs associ- ated with warranty work. A poor reputation and the resulting loss of business is another external failure cost that is difficult to quantify but nonetheless very real. Bad things happen when low-quality software is produced.
In an indictment of software developers who refuse to consider external failure costs, Cem Kaner [Kan95] states:
Many of the external failure costs, such as goodwill, are difficult to quantify, and many companies therefore ignore them when calculating their cost-benefit tradeoffs. Other external failure costs can be reduced (e.g. by providing cheaper, lower-quality, post-sale support, or by charging customers for support) without increasing customer satisfaction. By ignoring the costs to our customers of bad products, quality engineers encourage quality-related decision-making that victimizes our customers, rather than delighting them.
318 PART THREE QUALITY AND SECURITY
As expected, the relative costs to find and repair an error or defect increase dra- matically as we go from prevention to detection to internal failure to external failure costs. Figure 15.2, based on data collected by Boehm and Basili [Boe01b] and illus- trated by Cigital Inc. [Cig07], illustrates this phenomenon.
The industry average cost to correct a defect during code generation is approxi- mately $977 per error. The industry average cost to correct the same error if it is discovered during system testing is $7,136 per error. Cigital Inc. [Cig07] considers a large application that has 200 errors introduced during coding.
According to industry average data, the cost of finding and correcting defects during the coding phase is $977 per defect. Thus, the total cost for correcting the 200 “critical” defects during this phase (200 × $977) is approximately $195,400.
Industry average data shows that the cost of finding and correcting defects during the system testing phase is $7,136 per defect. In this case, assuming that the system testing phase revealed approximately 50 critical defects (or only 25% of those found by Cigital in the coding phase), the cost of finding and fixing those defects (50 × $7,136) would have been approximately $356,800. This would also have resulted in 150 critical errors going undetected and uncorrected. The cost of finding and fixing these remaining 150 defects in the maintenance phase (150 × $14,102) would have been $2,115,300. Thus, the total cost of finding and fixing the 200 defects after the coding phase would have been $2,472,100 ($2,115,300 + $356,800).
Even if your software organization has costs that are half of the industry average (most have no idea what their costs are!), the cost savings associated with early qual- ity control and assurance activities (conducted during requirements analysis and design) are compelling.
Figure 15.2 Relative cost of correcting errors and defects Source: Boehm, Barry and Basili, Victor R., “Soft- ware Defect Reduction Top 10 List,” IEEE Computer, vol. 34, no. 1, January 2001.
$139
CodingDesignRequirements MaintenanceTesting
$16,000
$14,000
$12,000
$10,000
$8,000
$-
$6,000
$4,000
$2,000
$14,102
$7,136
$977 $455
CHAPTER 15 QUALITY CONCEPTS 319
15.3.3 Risks In Chapter 1 of this book, we wrote, “people bet their jobs, their comforts, their safety, their entertainment, their decisions, and their very lives on computer software. It bet- ter be right.” The implication is that low-quality software increases risks for both the developer and the end user.4 In the preceding subsection, we discussed one of these risks (cost). But the downside of poorly designed and implemented applications does not always stop with dollars and time. An extreme example [Gag04] might serve to illustrate.
Throughout the month of November 2000 at a hospital in Panama, 28 patients received massive overdoses of gamma rays during treatment for a variety of cancers. In the months that followed, 5 of these patients died from radiation poisoning and 15 others developed serious complications. What caused this tragedy? A software pack- age, developed by a U.S. company, was modified by hospital technicians to compute modified doses of radiation for each patient.
Quality Issues
The scene: Doug Miller’s office as the SafeHome software project begins.
The players: Doug Miller, manager of the SafeHome software engineering team, and other members of the product software engi- neering team.
The conversation: Doug: I was looking at an industry report on the costs of repairing software defects. It’s pretty sobering.
Jamie: We’re already working on developing test cases for each functional requirement.
Doug: That’s good, but I was noticing that it costs eight times as much to repair a defect that is discovered in testing than it does if the defect is caught and repaired during coding.
Vinod: We’re using pair programming, so we should be able to catch most of the defects during coding.
Doug: I think you’re missing the point. Quality is more than simply removing coding errors. We need to look at the project quality goals and ensure that the evolving software prod- ucts are meeting them.
Jamie: Do you mean things like usability, security, and reliability?
Doug: Yes, I do. We need to build checks into the software process to monitor our progress toward meeting our quality goals.
Vinod: Can’t we finish the first prototype and then check it for quality?
Doug: I’m afraid not. We must establish a culture of quality early in the project.
Vinod: What do you want us to do, Doug?
Doug: I think we will need to find a technique that will allow us to monitor the quality of the SafeHome products. Let’s think about this and revisit this again tomorrow.
safehome
4 In an article titled “And the ‘Most Shocking Software Failure’ Award Goes To . . .” Chelsea Frischnecht provides a few brief examples of what can go wrong. The article can be found at https://www.tricentis.com/blog/2017/03/01/software-fail-awards/.
320 PART THREE QUALITY AND SECURITY
The three Panamanian medical physicists, who tweaked the software to provide additional capability, were charged with second-degree murder. The U.S. company was faced with serious litigation in two countries. Gage and McCormick comment:
This is not a cautionary tale for medical technicians, even though they can find them- selves fighting to stay out of jail if they misunderstand or misuse technology. This also is not a tale of how human beings can be injured or worse by poorly designed or poorly explained software, although there are plenty of examples to make the point. This is a warning for any creator of computer programs: that software quality matters, that appli- cations must be foolproof, and that—whether embedded in the engine of a car, a robotic arm in a factory or a healing device in a hospital—poorly deployed code can kill.
Poor quality leads to risks, some of them very serious.5
15.3.4 Negligence and Liability The story is all too common. A governmental or corporate entity hires a major soft- ware developer or consulting company to analyze requirements and then design and construct a software-based “system” to support some major activity. The system might support a major corporate function (e.g., pension management) or some governmental function (e.g., health care administration or homeland security).
Work begins with the best of intentions on both sides, but by the time the system is delivered, things have gone bad. The system is late, fails to deliver desired features and functions, is error-prone, and does not meet with customer approval. Litigation ensues.
In most cases, the customer claims that the developer has been negligent (in the manner in which it has applied software practices) and is therefore not entitled to payment. The developer often claims that the customer has repeatedly changed its requirements and has subverted the development partnership in other ways. In every case, the quality of the delivered system comes into question.
15.3.5 Quality and Security As the criticality of Web-based and mobile systems grows, application security has become increasingly important. Stated simply, software that does not exhibit high quality is easier to hack, and as a consequence, low-quality software can indirectly increase the security risk with all its attendant costs and problems.
In an interview in ComputerWorld, author and security expert Gary McGraw comments [Wil05]:
Software security relates entirely and completely to quality. You must think about secu- rity, reliability, availability, dependability—at the beginning, in the design, architecture, test, and coding phases, all through the software life cycle [process]. Even people aware of the software security problem have focused on late life-cycle stuff. The earlier you find the software problem, the better. And there are two kinds of software problems. One is bugs, which are implementation problems. The other is software flaws—architectural problems in the design. People pay too much attention to bugs and not enough on flaws.
To build a secure system, you must focus on quality, and that focus must begin during design. The concepts and methods discussed in Part Two of this book lead to
5 In early 2019, an error in the flight control software produced by a major aircraft manufac- turer was directly linked to two crashes and the deaths of 346 people.
CHAPTER 15 QUALITY CONCEPTS 321
a software architecture that reduces “flaws.” A more detailed discussion of security engineering is presented in Chapter 18.
15.3.6 The Impact of Management Actions Software quality is often influenced as much by management decisions as it is by technology decisions. Even the best software engineering practices can be subverted by poor business decisions and questionable project management actions.
In Part Four of this book we discuss project management within the context of the software process. As each project task is initiated, a project leader will make decisions that can have a significant impact on product quality.
Estimation Decisions. A software team is rarely given the luxury of providing an estimate for a project before delivery dates are established and an overall budget is specified. Instead, the team conducts a “sanity check” to ensure that delivery dates and milestones are rational. In many cases there is enormous time-to-market pressure that forces a team to accept unrealistic delivery dates. As a consequence, shortcuts are taken, activities that lead to higher-quality software may be skipped, and product quality suffers. If a delivery date is irrational, it is important to hold your ground. Explain why you need more time, or alternatively, suggest a subset of functionality that can be delivered (with high quality) in the time allotted.
Scheduling Decisions. When a software project schedule is established (Chapter 25), tasks are sequenced based on dependencies. For example, because component A depends on processing that occurs within components B, C, and D, component A cannot be scheduled for testing until components B, C, and D are fully tested. A project schedule would reflect this. But if time is very short, and A must be available for further critical testing, you might decide to test A without its subordinate components (which are run- ning slightly behind schedule), so that you can make it available for other testing that must be done before delivery. After all, the deadline looms. As a consequence, A may have defects that are hidden, only to be discovered much later. Quality suffers.
Risk-Oriented Decisions. Risk management (Chapter 26) is one of the key attributes of a successful software project. You really do need to know what might go wrong and establish a contingency plan if it does. Too many software teams prefer blind optimism, establishing a development schedule under the assumption that nothing will go wrong. Worse, they don’t have a way of handling things that do go wrong. As a consequence, when a risk becomes a reality, chaos reigns, and as the degree of crazi- ness rises, the level of quality invariably falls.
The software quality dilemma can best be summarized by stating Meskimen’s law: There’s never time to do it right, but always time to do it over again. Our advice: Taking the time to do it right is almost never the wrong decision.
15.4 ac h i e v i ng so f t Wa r e Qua L i t y
Software quality doesn’t just appear. It is the result of good project management and solid software engineering practice. Management and practice are applied within the context of four broad activities that help a software team achieve high software quality: software engineering methods, project management techniques, quality control actions, and software quality assurance.
322 PART THREE QUALITY AND SECURITY
15.4.1 Software Engineering Methods If you expect to build high-quality software, you must understand the problem to be solved. You must also be capable of creating a design that conforms to the problem while at the same time exhibiting characteristics that lead to software that exhibits the quality dimensions and factors discussed in Section 15.2.
In Part Two of this book, we presented a wide array of concepts and methods that can lead to a reasonably complete understanding of the problem and a comprehensive design that establishes a solid foundation for the construction activity. If you apply those concepts and adopt appropriate analysis and design methods, the likelihood of creating high-quality software will increase substantially.
15.4.2 Project Management Techniques The impact of poor management decisions on software quality has been discussed in Section 15.3.6. The implications are clear: If (1) a project manager uses estimation to verify that delivery dates are achievable, (2) schedule dependencies are understood and the team resists the temptation to use shortcuts, (3) risk planning is conducted so problems do not breed chaos, software quality will be affected in a positive way.
In addition, the project plan should include explicit techniques for quality and change management. Techniques that lead to good project management practices are discussed in Part Four of this book.
15.4.3 Machine Learning and Defect Prediction Defect prediction [Mun17] is an important part of identifying software components that may have quality concerns. Defect prediction models use statistical techniques to examine the relationships among combinations of software metrics and software com- ponents containing known software defects. They can be an efficient and effective way for software developers to quickly identify defect-prone classes. This can reduce costs and development times [Mal16].
Machine learning is an application of artificial intelligence (AI) techniques that provide systems with the ability to learn and improve from experience without being explicitly programmed. Stated another way, machine learning focuses on the develop- ment of computer programs that can access data and use the data to learn for them- selves. Machine learning techniques can be used to automate the process of discovering predictive relationships between software metrics and defective compo- nents [Ort17], [Li16], [Mal16].
Machine learning systems process large data sets containing representative combi- nations of metrics for defective and nondefective software components. These data are used to tune classification algorithms. Once the system has built a prediction model though this type of training, it can be used for quality assessment and defect prediction on data associated with future software products. Building these types of classifiers is a big part of what modern data scientists do. More discussion on the use of data science and software engineering appears in Appendix 2 of this book.
15.4.4 Quality Control Quality control encompasses a set of software engineering actions that help to ensure that each work product meets its quality goals. Models are reviewed to ensure that
CHAPTER 15 QUALITY CONCEPTS 323
they are complete and consistent. Code may be inspected to uncover and correct errors before testing commences. A series of testing steps is applied to uncover errors in processing logic, data manipulation, and interface communication. A combination of measurement and feedback allows a software team to tune the process when any of these work products fails to meet quality goals. Quality control activities are discussed in detail throughout the remainder of Part Three of this book.
15.4.5 Quality Assurance Quality assurance establishes the infrastructure that supports solid software engineering methods, rational project management, and quality control actions—all pivotal if you intend to build high-quality software. In addition, quality assurance consists of a set of auditing and reporting functions that assess the effectiveness and completeness of qual- ity control actions. The goal of quality assurance is to provide management and tech- nical staff with the data necessary to be informed about product quality, thereby gaining insight and confidence that actions to achieve product quality are working. Of course, if the data provided through quality assurance identifies problems, it is manage- ment’s responsibility to address the problems and apply the necessary resources to resolve quality issues. Software quality assurance is discussed in detail in Chapter 17.
15.5 su m m a ry
Concern for the quality of the software-based systems has grown as software becomes integrated into every aspect of our daily lives. But it is difficult to develop a compre- hensive description of software quality. In this chapter, quality has been defined as an effective software process applied in a manner that creates a useful product that pro- vides measurable value for those who produce it and those who use it.
A wide variety of software quality dimensions and factors has been proposed over the years. All try to define a set of characteristics that, if achieved, will lead to high software quality. McCall’s and the ISO 25010 quality factors establish characteristics such as reliability, usability, maintainability, functionality, and portability as indicators that quality exists.
Every software organization is faced with the software quality dilemma. In essence, everyone wants to build high-quality systems, but the time and effort required to produce “perfect” software are simply unavailable in a market-driven world. The ques- tion becomes, Should we build software that is “good enough”? Although many com- panies do just that, there is a significant downside that must be considered.
Regardless of the approach that is chosen, quality does have a cost that can be discussed in terms of prevention, appraisal, and failure. Prevention costs include all software engineering actions that are designed to prevent defects in the first place. Appraisal costs are associated with those actions that assess software work products to determine their quality. Failure costs encompass the internal price of failure and the external effects that poor quality precipitates.
Software quality is achieved through the application of software engineering meth- ods, solid management practices, and comprehensive quality control—all supported by a software quality assurance infrastructure. In the chapters that follow, quality control and assurance are discussed in some detail.
324 PART THREE QUALITY AND SECURITY
Pro b l e m s a n d Po i n t s to Po n d e r
15.1. Describe how you would assess the quality of a university before applying to it. What factors would be important? Which would be critical?
15.2. Using the definition of software quality proposed in Section 15.2, do you think it’s pos- sible to create a useful product that provides measurable value without using an effective process? Explain your answer.
15.3. Using the subattributes noted for the ISO 25010 quality factor “maintainability” in Sec- tion 15.2.1, develop a set of questions that explore whether or not these attributes are present. Follow the example shown in Section 15.2.2.
15.4. Describe the software quality dilemma in your own words.
15.5. What is “good enough” software? Name a specific company and specific products that you believe were developed using the good enough philosophy.
15.6. Considering each of the four aspects of the cost of quality, which do you think is the most expensive and why?
15.7. Do a Web search and find three other examples of “risks” to the public that can be directly traced to poor software quality. Consider beginning your search at http://catless.ncl .ac.uk/risks.
15.8. Are quality and security the same thing? Explain.
15.9. Explain why it is that many of us continue to live by Meskimen’s law. What is it about the software business that causes this?
Design element: Quick Look icon magnifying glass: © Roger Pressman
325
C H A P T E R
16
What is it? You’ll make mistakes as you de- velop software engineering work products. There’s no shame in that—as long as you try hard, very hard, to find and correct the mis- takes before they are delivered to end users. Technical reviews are the most effective mechanism for finding mistakes early in the software process.
Who does it? Software engineers perform technical reviews, also called peer reviews, with their colleagues. As we discussed in Chapters 3 and 4, sometimes it is wise to include other stakeholders in these reviews.
Why is it important? If you find an error early in the process, it is less expensive to correct. In addition, errors have a way of amplifying as the process proceeds. So a relatively minor error left untreated early in the process can be amplified into a major set of errors later in the project. Finally, reviews save time by reducing
the amount of rework that will be required late in the project.
What are the steps? Your approach to reviews will vary depending on the type of review you select. In general, six steps are employed, although not all are used for every type of review: planning, preparation, structuring the meeting, noting errors, making corrections (done outside the review), and verifying that corrections have been performed properly.
What is the work product? The output of a review is a list of issues and/or errors that have been uncovered. In addition, the techni- cal status of the work product is also indicated.
How do I ensure that I’ve done it right? First, select the type of review that is appropriate for your development culture. Follow the guide- lines that lead to successful reviews. If the re- views that you conduct lead to higher-quality software, you’ve done it right.
Q u i c k L o o k
Reviews—A Recommended Approach
bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326 cost effectiveness . . . . . . . . . . . . . . . . . . . . . .329 defect amplification . . . . . . . . . . . . . . . . . . . .327 defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326 error density . . . . . . . . . . . . . . . . . . . . . . . . . .328
errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326 informal reviews . . . . . . . . . . . . . . . . . . . . . . . 331 record keeping . . . . . . . . . . . . . . . . . . . . . . . .333 review reporting . . . . . . . . . . . . . . . . . . . . . . .333 technical reviews . . . . . . . . . . . . . . . . . . . . . .326
k e y c o n c e p t s
Software reviews are a “filter” for the software process work flow. Too few, and the flow is “dirty.” Too many, and the flow slows to a trickle. Reviews are applied at various points during software engineering and serve to uncover errors and defects. Software reviews “purify” software engineering work products, including requirements and design models, code, and testing data. Using metrics, you can determine which reviews work and emphasize them and at the same time remove ineffective reviews from the flow to accelerate the process.
326 PART THREE QUALITY AND SECURITY
Many different types of reviews can be conducted as part of software engineering. Each has its place. An informal meeting around the coffee machine is a form of review, if technical problems are discussed. A formal presentation of software archi- tecture to an audience of customers, management, and technical staff is also a form of review. In this book, however, we focus on technical or peer reviews, exemplified by casual reviews, walkthroughs, and inspections. A technical review (TR) is the most effective filter from a quality control standpoint. Conducted by software engineers and other stakeholders for all project team members, the TR is an effective means for uncovering errors and improving software quality.
16.1 co st im pac t o f so f t wa r e De f e c t s
Within the context of the software process, the terms defect and fault are synonymous. Both imply a quality problem that is discovered after the software has been released to end users (or to another framework activity in the software process). In earlier chapters, we used the term error to depict a quality problem that is discovered by software engineers (or others) before the software is released to the end user (or to another framework activity in the software process).
Bugs, Errors, and Defects The goal of software quality control, and in a broader sense, quality management
in general, is to remove quality problems in the software. These problems are referred to by vari- ous names—bugs, faults, errors, or defects, to name a few. Are these terms synonymous, or are there subtle differences between them?
In this book we make a clear distinction be- tween an error (a quality problem found before the software is released to other stakeholders or end users) and a defect (a quality problem found only after the software has been re- leased to end users or other stakeholders).1 We make this distinction because errors and defects have very different economic, business, psychological, and human impacts. As software engineers, we want to find and correct as many errors as possible before the customer and/or end user encounter them. We want to avoid
defects—because defects ( justifiably) make software people look bad.
It is important to note, however, that the tempo- ral distinction made between errors and defects in this book is not mainstream thinking. The general consensus within the software engineering com- munity is that defects and errors, faults, and bugs are synonymous. That is, the point in time that the problem was encountered has no bearing on the term used to describe the problem. Part of the ar- gument in favor of this view is that it is sometimes difficult to make a clear distinction between pre- and postrelease (e.g., consider an incremental process used in agile development).
Regardless of how you choose to interpret these terms, recognize that the point in time at which a problem is discovered does matter and that software engineers should try hard—very hard—to find problems before their customers and end users encounter them.
info
1 If software process improvement is considered, a quality problem that is propagated from one process framework activity (e.g., modeling) to another (e.g., construction) can also be called a “defect” because the problem should have been found before a work product (e.g., a design model) was “released” to the next activity.
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 327
The primary objective of a formal technical review (FTR) is to find errors before they are passed on to another software engineering activity or released to the end user. The obvious benefit of technical reviews is the early discovery of errors so that they do not propagate to the next step in the software process.
A number of industry studies indicate that design activities introduce between 50 and 65 percent of all errors (and ultimately, all defects) during the software process. However, review techniques have been shown to be up to 75 percent effective [Jon86] in uncovering design flaws. By detecting and removing a large percentage of these errors, the review process substantially reduces the cost of subsequent activities in the software process. We have known this for decades, and yet there are still many devel- opers who do not believe the time spent on reviews is almost always less than the time required to rewrite bad code [Yad17].
16.2 De f e c t am p L i f i cat i o n a n D re m ova L
Defect amplification is a concept originally proposed almost four decades ago [IBM81]. It helps to justify the effort expended on software reviews. In essence, defect ampli- fication makes the following argument—an error introduced early in the software engineering work flow (e.g., during requirement modeling) and undetected, can and often will be amplified into multiple errors during design. If those errors are not uncovered (using effective reviews), they themselves may be further amplified into still more errors during coding. A single error introduced early and not uncovered and corrected can amplify into multiple errors later in the process. Defect propagation is a term used to describe the impact an undiscovered error has on future development activities or product behavior [Vit17].
As a development team moves deeper into the software process, the cost of find- ing and fixing an error grows. This simple reality is exacerbated by defect amplifi- cation and propagation because a single error may become multiple errors downstream. The cost of finding and fixing a single error can be signficant, but the cost to find and fix multiple errors propagated by a single earlier error is substan- tially more significant.
To conduct reviews, you must expend time and effort, and your development orga- nization must spend money. However, the reality of defect amplification and propaga- tion leaves little doubt that you can pay now or pay much more later. This is what technical debt (Chapter 11) is all about [Xia16] [Vit17].
16.3 re v i e w me t r i c s a n D th e i r us e
Technical reviews are one of many actions that are required as part of good software engineering practice. Each action requires dedicated human effort. Because available project effort is finite, it is important for a software engineering organization to under- stand the effectiveness of each action by defining a set of metrics (Chapter 23) that can be used to assess their efficacy.
328 PART THREE QUALITY AND SECURITY
Although many metrics can be defined for technical reviews, a relatively small subset can provide useful insight. The following review metrics can be collected for each review that is conducted:
∙ Preparation effort, Ep. The effort (in person-hours) required to review a work product prior to the actual review meeting
∙ Assessment effort, Ea. The effort (in person-hours) that is expended during the actual review
∙ Rework effort, Er. The effort (in person-hours) that is dedicated to the correction of those errors uncovered during the review
∙ Review effort, Ereview. Represents the sum of effort measures for reviews:
Ereview = Ep + Ea + Er
∙ Work product size (WPS). A measure of the size of the work product that has been reviewed (e.g., the number of UML models, the number of docu- ment pages, or the number of lines of code)
∙ Minor errors found, Errminor. The number of errors found that can be catego- rized as minor (requiring less than some prespecified effort to correct)
∙ Major errors found, Errmajor. The number of errors found that can be catego- rized as major (requiring more than some prespecified effort to correct)
∙ Total errors found, Errtot. Represents the sum of the errors found:
Errtot = Errminor + Errmajor
∙ Error density. Represents the errors found per unit of work product reviewed:
Error density = Errtot
WPS
How might these metrics be used? As an example, consider a requirements model that is reviewed to uncover errors, inconsistencies, and omissions. It would be pos- sible to compute the error density in several different ways. Assume the requirements model contains 18 UML diagrams as part of 32 overall pages of descriptive materials. The review uncovers 18 minor errors and 4 major errors. Therefore, Errtot = 22. Error density is 1.2 errors per UML diagram, or 0.68 errors per requirements model page.
If reviews are conducted for a number of different types of work products (e.g., requirements model, design model, code, test cases), the percentage of errors uncov- ered for each review can be computed against the total number of errors found for all reviews. In addition, the error density for each work product can be computed.
Once data are collected for many reviews conducted across many projects, average values for error density enable you to estimate the number of errors to be found in a new document before it is reviewed. For example, if the average error density for a requirements model is 0.68 errors per page, and a new requirements model is 40 pages long, a rough estimate suggests that your software team will find around 27 errors during the review of the document. If you find only 9 errors, you’ve either done an extremely good job in developing the requirements model or your review approach was not thorough enough.
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 329
It is difficult to measure the cost effectiveness of any technical review in real time. A software engineering organization can assess the effectiveness of reviews and their cost benefit only after reviews have been completed, review metrics have been col- lected, average data have been computed, and then the downstream quality of the software is measured (via testing).
Returning to the previous example, the average error density for requirements mod- els was determined to be 0.68 per page. The effort required to correct a minor model error (immediately after the review) has been found to require 4 person-hours. The effort required for a major requirement error has been found to be 18 person-hours. Examining the review data collected, you find that minor errors occur about six times more frequently than major errors. Therefore, you can estimate that the average effort to find and correct a requirements error during review is about 6 person-hours.
Requirements-related errors uncovered during testing require an average of 45 person-hours to find and correct (no data are available on the relative severity of the error). Using the averages noted, we get:
Effort saved per error = Etesting − Ereviews
= 45 − 6 = 39 person-hours/error
Because 22 errors were found during the review of the requirements model, a savings of about 858 person-hours of testing effort would be achieved. And that’s just for requirements-related errors. Errors associated with design and code would add to the overall benefit.
The bottom line—effort saved leads to shorter delivery cycles and improved time to market. The example presented in this section suggests this may be true. More importantly, industry data for software reviews has been collected for more than three decades and is summarized qualitatively using the graphs shown in Figure 16.1.
Referring to the figure, the effort expended when reviews are used does increase early in the development of a software increment, but this early investment for reviews pays dividends because testing and corrective effort is reduced. It is important to note
Planning Requirements Design Code Test Deployment
Time
E� or
t
Without inspection
With inspection
Figure 16.1 Effort expended with and without reviews Source: Fagan, Michael E., “Advances in Soft- ware Inspections,” IEEE Transactions on Software Engineering, vol. SE-12, no. 7, July 1986, 744–751.
330 PART THREE QUALITY AND SECURITY
that the deployment date for development with reviews is sooner than the deployment date without reviews. Reviews don’t take time; they save it!
16.4 Cr i t e r i a f o r ty p e s o f re v i e ws
Technical reviews can be classified as either formal or informal or somewhere in between these two extremes. The level of formality is chosen to match the type of product to be built, the project time line, and the people who are doing the work. Figure 16.2 depicts a reference model for technical reviews [Lai02] that identifies four characteristics that contribute to the formality with which a review is conducted.
Each of the reference model characteristics helps to define the level of review formality. The formality of a review increases when (1) distinct roles are explicitly defined for the reviewers, (2) there is a sufficient amount of planning and preparation for the review, (3) a distinct structure for the review (including tasks and internal work products) is defined, and (4) follow-up by the reviewers occurs for any corrections that are made.
An element that is not presented in this model is the frequency of the reviews them- selves. If you are using an agile prototyping model (Chapter 4) that contains relatively short sprints, your team may opt for less formal reviews because the reviews are hap- pening fairly often. This usually means that defects are caught sooner and more often.
To understand the reference model, let’s assume that you’ve decided to review the interface design for SafeHomeAssured.com. You can do this in a variety of different ways that range from relatively casual to extremely rigorous. If you decide that the casual approach is most appropriate, you ask a few colleagues (peers) to examine the interface prototype in an effort to uncover potential problems. All of you decide that there will be no advance preparation, but that you will evaluate the prototype in a reasonably structured way—looking at layout first, aesthetics next, navigation options after that, and so on. As the designer, you decide to take a few notes, but nothing formal.
Review
Planning & Preparation
Correction & Verification
Meeting Structure
Roles Individuals
Play
Figure 16.2 Reference model for technical reviews
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 331
But what if the interface is pivotal to the success of the entire project? What if human lives depended on an interface that was ergonomically sound? You might decide that a more rigorous approach was necessary. A review team would be formed. Each person on the team would have a specific role to play—leading the team, record- ing findings, presenting the material, and so on. Each reviewer would be given access to the work product (in this case, the interface prototype) before the review and would spend time looking for errors, inconsistencies, and omissions. A set of specific tasks would be conducted based on an agenda that was developed before the review occurred. The results of the review would be formally recorded, and the team would decide on the status of the work product based on the outcome of the review. Members of the review team might also verify that the corrections made were done properly.
In this book we consider two broad categories of technical reviews: informal reviews and more formal technical reviews. Within each category, a number of dif- ferent approaches can be chosen. These are presented in the sections that follow.
16.5 in f o r m a L re v i e ws
Informal reviews include a simple desk check of a software engineering work product with a colleague, a casual meeting (involving more than two people) for the purpose of reviewing a work product, or the review-oriented aspects of pair programming (Chapter 3).
A simple desk check or a casual meeting conducted with a colleague is a review. However, because there is no advance planning or preparation, no agenda or meeting structure, and no follow-up on the errors that are uncovered, the effectiveness of such reviews is considerably lower than more formal approaches. But a simple desk check can and does uncover errors that might otherwise propagate further into the software process.
One way to improve the efficacy of a desk check review is to develop a set of simple review checklists2 for each major work product produced by the software team. The questions posed within the checklist are generic, but they will serve to guide the reviewers as they check the work product. For example, let’s reexamine a desk check of the interface prototype for SafeHomeAssured.com. Rather than simply playing with the prototype at the designer’s workstation, the designer and a colleague examine the prototype using a checklist for interfaces:
∙ Is the layout designed using standard conventions? Left to right? Top to bottom? ∙ Does the presentation need to be scrolled? ∙ Are color and placement, typeface, and size used effectively? ∙ Are all navigation options or functions represented at the same level of
abstraction? ∙ Are all navigation choices clearly labeled?
2 Literally hundreds of technical review checklists can be found via a web search. For exam- ple, a useful code review checklist can be downloaded from https://courses.cs.washington. edu/courses/cse403/12wi/sections/12wi_code_review_checklist.pdf.
332 PART THREE QUALITY AND SECURITY
and so on. Any errors or issues noted by the reviewers are recorded by the designer for resolution at a later time. Desk checks may be scheduled in an ad hoc manner, or they may be mandated as part of good software engineering practice. In general, the amount of material to be reviewed is relatively small and the overall time spent on a desk check spans little more than 1 or 2 hours.
In Chapter 3, we described pair programming in the following manner: XP recom- mends that two people work together at one computer workstation to create code for a story. This provides a mechanism for real-time problem solving (two heads are often better than one) and real-time quality assurance.
Pair programming (Section 3.5.1) can be characterized as a continuous desk check. Rather than scheduling a review at some point in time, pair programming encourages continuous review as a work product (design or code) is created. The benefit is imme- diate discovery of errors and better work product quality.
Some software engineers argue that the inherent redundancy built into pair pro- gramming is wasteful of resources. After all, why assign two people to a job that one person can accomplish? The answer to this question can be found in Section 16.3. If the quality of the work product produced as a consequence of pair programming is significantly better than the work of an individual, the quality-related savings can more than justify the “redundancy” implied by pair programming.
16.6 fo r m a L te c h n i ca L re v i e ws
A formal technical review (FTR) is a software quality control activity performed by software engineers (and others). The objectives of an FTR are: (1) to uncover errors in function, logic, or implementation for any representation of the software; (2) to verify that the software under review meets its requirements; (3) to ensure that the software has been represented according to predefined standards; (4) to achieve soft- ware that is developed in a uniform manner; and (5) to make projects more manage- able. In addition, the FTR serves as a training ground, enabling junior engineers to observe different approaches to software analysis, design, and implementation. The FTR also serves to promote backup and continuity because a number of people become familiar with parts of the software that they may not have otherwise seen.
The FTR is actually a class of reviews that includes walkthroughs and inspections. Each FTR is conducted as a meeting and will be successful only if it is properly planned, controlled, and attended. In the sections that follow, guidelines similar to those for a walkthrough are presented as a representative formal technical review. If you have interest in software inspections, as well as additional information on walk- throughs, see [Rad02], [Wie02], or [Fre90].
16.6.1 The Review Meeting Regardless of the FTR format that is chosen, every review meeting should abide by the following constraints:
∙ Between three and five people (typically) should be involved in the review. ∙ Advance preparation should occur but should require no more than 2 hours of
work for each person. ∙ The duration of the review meeting should be less than 2 hours.
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 333
Given these constraints, it should be obvious that an FTR focuses on a specific (and small) part of the overall software. For example, rather than attempting to review an entire design, walkthroughs are conducted for each component or small group of components. By narrowing the focus, the FTR has a higher likelihood of uncovering errors.
The focus of the FTR is on a work product (e.g., a self-contained portion of a requirements model, a detailed component design, or source code for a component). The individual who has developed the work product—the producer—informs the proj- ect leader that the work product is complete and that a review is required. The project leader contacts a review leader, who evaluates the product for readiness, generates copies of product materials, and distributes them to two or three reviewers for advance preparation. Each reviewer is expected to spend between 1 and 2 hours reviewing the product, making notes, and otherwise becoming familiar with the work. Concurrently, the review leader also reviews the product and establishes an agenda for the review meeting, which is typically scheduled for the next day.
The review meeting is attended by the review leader, all reviewers, and the pro- ducer. One of the reviewers takes on the role of a recorder, that is, the individual who records (in writing) all important issues raised during the review. The FTR begins with an introduction of the agenda and a brief introduction by the producer. The producer then proceeds to “walk through” the work product, explaining the material, while reviewers raise issues based on their advance preparation. When valid problems or errors are discovered, the recorder notes each. Don’t point out errors too harshly. One way to be gentle is to ask a question that enables the producer to discover the error.
At the end of the review, all attendees of the FTR must decide whether to: (1) accept the product without further modification, (2) reject the product due to severe errors (once corrected, another review must be performed), or (3) accept the product provisionally (minor errors have been encountered and must be corrected, but no additional review will be required). After the decision is made, all FTR attendees complete a sign-off, indicating their participation in the review and their concurrence with the review team’s findings.
16.6.2 Review Reporting and Record Keeping During the FTR, a reviewer (the recorder) actively records all issues that have been raised. These are summarized at the end of the review meeting, and a review issues list is produced. In addition, a formal technical review summary report is completed. A review summary report answers three questions:
1. What was reviewed? 2. Who reviewed it? 3. What were the findings and conclusions?
The review summary report is a single-page form (with possible attachments). It becomes part of the project historical record and may be distributed to the project leader and other interested parties.
The review issues list serves two purposes: (1) to identify problem areas within the product and (2) to serve as an action item checklist that guides the producer as corrections are made. An issues list is normally attached to the summary report.
334 PART THREE QUALITY AND SECURITY
You should establish a follow-up procedure to ensure that items on the issues list have been properly corrected. Unless this is done, it is possible that issues raised can “fall between the cracks.” One approach is to assign the responsibility for follow-up to the review leader.
16.6.3 Review Guidelines Guidelines for conducting formal technical reviews must be established in advance, distributed to all reviewers, agreed upon, and then followed. A review that is uncon- trolled can often be worse than no review at all. The following represents a minimum set of guidelines for formal technical reviews:
1. Review the product, not the producer. An FTR involves people and egos. Conducted properly, the FTR should leave all participants with a warm feel- ing of accomplishment. Conducted improperly, the FTR can take on the aura of an inquisition. Errors should be pointed out gently; the tone of the meeting should be loose and constructive; the intent should not be to embarrass or belittle. The review leader should conduct the review meeting to ensure that the proper tone and attitude are maintained and should immediately halt a review that has gotten out of control.
2. Set an agenda and maintain it. One of the key maladies of meetings of all types is drift. An FTR must be kept on track and on schedule. The review leader is chartered with the responsibility for maintaining the meeting sched- ule and should not be afraid to nudge people when drift sets in.
3. Limit debate and rebuttal. When an issue is raised by a reviewer, there may not be universal agreement on its impact. Rather than spending time debating the question, the issue should be recorded for further discussion off-line.
4. Enunciate problem areas, but don’t attempt to solve every problem noted. A review is not a problem-solving session. The solution of a problem can often be accomplished by the producer alone or with the help of only one other indi- vidual. Problem solving should be postponed until after the review meeting.
5. Take written notes. It is sometimes a good idea for the recorder to make notes on a wall board, so that wording and priorities can be assessed by other reviewers as information is recorded.
6. Limit the number of participants and insist upon advance preparation. Two heads are better than one, but 14 are not necessarily better than 4. Keep the number of people involved to the necessary minimum. However, all review team members must prepare in advance.
7. Develop a checklist for each product that is likely to be reviewed. A checklist helps the review leader to structure the FTR meeting and helps each reviewer to focus on important issues. Checklists should be developed for analysis, design, code, and even testing work products.
8. Allocate resources and schedule time for FTRs. For reviews to be effective, they should be scheduled as a task during the software process. In addition, time should be scheduled for the inevitable modifications that will occur as the result of an FTR.
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 335
9. Conduct meaningful training for all reviewers. The training should stress both process-related issues and the human psychological side of reviews.
10. Review your early reviews. Debriefing can be beneficial in uncovering prob- lems with the review process itself. The very first product to be reviewed should be the review guidelines themselves.
Because many variables (e.g., number of participants, type of work products, timing and length, specific review approach) have an impact on a successful review, a software organization should experiment to determine what approach works best in a local context.
In an ideal setting, every software-engineering work product would undergo a for- mal technical review. In the real world of software projects, resources are limited and time is short. As a consequence, reviews are often skipped, even though their value as a quality control mechanism is recognized. However, full FTR resources should be used on those work products that are likely to be error prone.
Quality Issues
The scene: Doug Miller’s office as the SafeHome software project begins.
The players: Doug Miller, manager of the SafeHome software engineering team, and other members of the product software engineering team.
The conversation: Doug: I know we didn’t spend time develop- ing a quality plan for this project, but we’re already into it and we have to consider quality . . . right?
Jamie: Sure. We’ve already decided that as we develop the requirements model [Chapter 8], Ed has committed to develop a testing procedure for each requirement.
Doug: That’s really good, but we’re not going to wait until testing to evaluate quality, are we?
Vinod: No! Of course not. We’ve got reviews scheduled into the project plan for this soft- ware increment. We’ll begin quality control with the reviews.
Jamie: I’m a bit concerned that we won’t have enough time to conduct all the reviews. In fact, I know we won’t.
Doug: Hmmm. So what do you propose?
Jamie: I say we select those elements of the requirements and design model that are most critical to SafeHome and review them.
Vinod: But what if we miss something in a part of the model we don’t review?
Jamie: Maybe . . . but I’m not sure we even have time to review every element of the models.
Vinod: What do you want us to do, Doug?
Doug: Let’s steal something from Extreme Programming [Chapter 3]. We’ll develop the elements of each model in pairs—two people—and conduct an informal review of each as we go. We’ll then target “critical” elements for a more formal team review, but keep those reviews to a minimum. That way, everything gets looked at by more than one set of eyes, but we still maintain our delivery dates.
Jamie: That means we’re going to have to revise the schedule.
Doug: So be it. Quality trumps schedule on this project.
safehome
336 PART THREE QUALITY AND SECURITY
16.7 po st m o rt e m eva Luat i o n s
Many lessons can be learned if a software team takes the time to evaluate the results of a software project after the software has been delivered to end users. Baaz and his colleagues [Baa10] suggest the use of a postmortem evaluation (PME) as a mechanism to determine what went right and what went wrong when software engineering process and practice is applied in a specific project.
Unlike an FTR that focuses on a specific work product, it is more like a Scrum retrospective (Section 3.4.5). A PME examines the entire software project, focusing on both “excellences (that is, achievements and positive experiences) and challenges (problems and negative experiences)” [Baa10]. Often conducted in a workshop format, a PME is attended by members of the software team and stakeholders. The intent is to identify excellences and challenges and to extract lessons learned from both. The objective is to suggest improvements to both process and practice going forward. Many software engineers regard PME documents as some of the most valuable docu- ments to save in the project archive.
16.8 ag i L e re v i e ws
It’s not surprising that some software engineers balk at the thought of including reviews of any kind in agile development processes. Yet failing to catch defects early can be costly in terms of time and resources. Ignoring technical debt does not make it go away. Agile developers have the same need to find defects early (and often) that all software developers have. Admittedly, it may be somewhat more difficult to get agile developers to make use of metrics, but many of them can be collected unobtrusively.
If we take a closer look at the Scrum framework (Section 3.4), there are several places where informal and formal reviews take place. During the sprint planning meeting, user stories are reviewed and ordered according to priority, before selecting the user stories to include in the next sprint. The daily Scrum meeting is an informal way to ensure that the team members are all working on the same priorities and to catch any defects that may prevent completing the sprint on time. Agile developers often use pair programming, another informal review technique. The sprint review meeting is often conducted using guidelines similar to those discussed for a formal technical review. The code producers walk through the user stories selected for the sprint and demonstrate to the product owner that all functionality is present. Unlike the FTR, the product owner has the final word on whether to accept the sprint proto- type or not.
If we look at the evaluate prototype portion of our recommended process model (Section 4.5), this task is also likely to be conducted as a formal technical review with development risk assessment added to it. We mentioned previously (Section 16.7) that the sprint retrospective meeting is really similar to the project postmortem meeting in that the development team is trying to capture its lessons learned. A major aspect of software quality assurance is being able to repeat your successes and avoid repeating your mistakes.
CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 337
16.9 su m m a ry
The intent of every technical review is to find errors and uncover issues that would have a negative impact on the software to be deployed. The sooner an error is uncovered and corrected, the less likely that error will propagate to other software engineering work products and amplify itself, resulting in significantly more effort to correct it.
To determine whether quality control activities are working, a set of metrics should be collected. Review metrics focus on the effort required to conduct the review and the types and severity of errors uncovered during the review. Once metrics data are collected, they can be used to assess the efficacy of the reviews you do conduct. Industry data indicate that reviews provide a significant return on investment.
A reference model for review formality identifies the roles people play, planning and preparation, meeting structure, correction approach, and verification as the char- acteristics that indicate the degree of formality with which a review is conducted. Informal reviews are casual in nature but can still be used effectively to uncover errors. Formal reviews are more structured and have the highest probability of leading to high-quality software.
Informal reviews are characterized by minimal planning and preparation and little record keeping. Desk checks and pair programming fall into the informal review category.
A formal technical review is a stylized meeting that has been shown to be extremely effective in uncovering errors. Formal technical reviews establish defined roles for each reviewer, encourage planning and advance preparation, require the application of defined review guidelines, and mandate record keeping and status reporting.
pro b L e m s a n D po i n t s to po n D e r
16.1. Explain the difference between an error and a defect.
16.2. Why can’t we just wait until testing to find and correct all software errors?
16.3. Assume that 10 errors have been introduced in the requirements model and that each error will be amplified by a factor of 2:1 into design and an additional 20 design errors are introduced and then amplified 1.5:1 into code where an additional 30 errors are introduced. Assume further that all unit testing will find 30 percent of all errors, integration will find 30 percent of the remaining errors, and validation tests will find 50 percent of the remaining errors. No reviews are conducted. How many errors will be released to the end users?
16.4. Reconsider the situation described in Problem 16.3, but now assume that requirements, design, and code reviews are conducted and are 60 percent effective in uncovering all errors at that step. How many errors will be released to the field?
16.5. Reconsider the situation described in Problems 16.3 and 16.4. If each of the errors released to the field costs $4800 to find and correct and each error found in review costs $240 to find and correct, how much money is saved by conducting reviews?
16.6. Describe the meaning of Figure 16.1 in your own words.
16.7. Can you think of a few instances in which a desk check might create problems rather than provide benefits?
338 PART THREE QUALITY AND SECURITY
16.8. A formal technical review is effective only if everyone has prepared in advance. How do you recognize a review participant who has not prepared? What do you do if you’re the review leader?
16.9. How is technical debt addressed in agile process models?
16.10. Consider the review guidelines presented in Section 16.6.3. Which do you think is most important and why?
Design element: Quick Look icon magnifying glass: © Roger Pressman
339
C H A P T E R
17
What is it? It’s not enough to talk the talk by saying that software quality is important. You have to (1) explicitly define what is meant when you say “software quality,” (2) create a set of activities that will help ensure that every soft- ware engineering work product exhibits high quality, (3) perform quality control and assur- ance activities on every software project, (4) use metrics to develop strategies for improving your software process and, as a consequence, the quality of the end product.
Who does it? Everyone involved in the software engineering process is responsible for quality.
Why is it important? You can do it right, or you can do it over again. If a software team stresses quality in all software engineering ac- tivities, it reduces the amount of rework that it must do. That results in lower costs, and more importantly, improved time to market.
What are the steps? Before software quality assurance (SQA) activities can be initiated, it is
important to define software quality at several different levels of abstraction. Once you un- derstand what quality is, a software team must identify a set of SQA activities that will filter errors out of work products before they are passed on.
What is the work product? A Software Qual- ity Assurance Plan is created to define a soft- ware team’s SQA strategy. During modeling and coding, the primary SQA work product is the output of technical reviews (Chapter 16). During testing (Chapters 19 through 21), test plans and procedures are produced. Other work products associated with process improvement may also be generated.
How do I ensure that I’ve done it right? Find errors before they become defects! That is, work to improve your defect removal effi- ciency (Chapter 23), thereby reducing the amount of rework that your software team must perform.
Q u i c k L o o k
Software Quality Assurance
Bayesian inference . . . . . . . . . . . . . . . . . . . . . 351 elements of software quality assurance . . . . 341 formal approaches . . . . . . . . . . . . . . . . . . . . . 347 genetic algorithms . . . . . . . . . . . . . . . . . . . . .352 goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .346 ISO 9001:2015 standard . . . . . . . . . . . . . . . . .354
Six Sigma . . . . . . . . . . . . . . . . . . . . . . . . . . . . .349 software reliability . . . . . . . . . . . . . . . . . . . . .350 software safety . . . . . . . . . . . . . . . . . . . . . . . .352 SQA plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . .354 SQA tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .343 statistical software quality assurance . . . . . . 347
k e y c o n c e p t s
The software engineering approach described in this book works toward a single goal: to produce on-time, high-quality software. Yet many readers will be challenged by the question: “What is software quality?”
340 PART 3 QUALITY AND SECURITY
Philip Crosby [Cro79], in his landmark book on quality, provides a wry answer to this question:
The problem of quality management is not what people don’t know about it. The problem is what they think they do know . . .
In this regard, quality has much in common with sex. Everybody is for it. (Under certain conditions, of course.) Everyone feels they understand it. (Even though they wouldn’t want to explain it.) Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.) And, of course, most people feel that problems in these areas are caused by other people. (If only they would take the time to do things right.)
Indeed, quality is a challenging concept—one that we addressed in some detail in Chapter 15.1
Some software developers continue to believe that software quality is something you begin to worry about after code has been generated. Nothing could be further from the truth! Software quality assurance (often called quality management) is an umbrella activity (Chapter 2) that is applied throughout the software process.
Software quality assurance (SQA) encompasses (Figure 17.1): (1) an SQA process, (2) specific quality assurance and quality control tasks (including technical reviews and a multitiered testing strategy), (3) effective software engineering practice (meth- ods and tools), (4) control of all software work products and the changes made to them (Chapter 22), (5) a procedure to ensure compliance with software development standards (when applicable), and (6) measurement and reporting mechanisms.
In this chapter, we focus on the management issues and the process-specific activities that enable a software organization to ensure that it does “the right things at the right time in the right way.”
1 If you have not read Chapter 15, you should do so now.
Figure 17.1 Software quality assurance
Software Quality
Assurance
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 341
17.1 Bac kg ro u n d is s u e s
Quality control and assurance are essential activities for any business that produces products to be used by others. Prior to the twentieth century, quality control was the sole responsibility of the craftsperson who built a product. As time passed and mass production techniques became commonplace, quality control became an activity performed by people other than the ones who built the product.
The first formal quality assurance and control function was introduced at Bell Labs in 1916 and spread rapidly throughout the manufacturing world. During the 1940s, more formal approaches to quality control were suggested. These relied on measure- ment and continuous process improvement [Dem86] as key elements of quality management.
The history of quality assurance in software development parallels the history of quality in hardware manufacturing. During the early days of computing (1950s and 1960s), quality was the sole responsibility of the programmer. Standards for quality assurance for software were introduced in military contract software development during the 1970s and have spread rapidly into software development in the commer- cial world [IEE17]. Extending the definition presented earlier, software quality assur- ance is a “planned and systematic pattern of actions” [Sch01] that are required to ensure high quality in software. The scope of quality assurance responsibility might best be characterized by paraphrasing a once-popular automobile commercial: “Quality Is Job #1.” The implication for software is that many different constituencies have software quality assurance responsibility—software engineers, project managers, customers, salespeople, and the individuals who serve within an SQA group.
The SQA group serves as the customer’s in-house representative. That is, the peo- ple who perform SQA must look at the software from the customer’s point of view. Does the software adequately meet the quality factors noted in Chapter 15? Have software engineering practices been conducted according to preestablished standards? Have technical disciplines properly performed their roles as part of the SQA activity? The SQA group attempts to answer these and other questions to ensure that software quality is maintained.
17.2 eL e m e n t s o f so f t wa r e Qua L i t y as s u r a nc e
Software quality assurance encompasses a broad range of concerns and activities that focus on the management of software quality. These can be summarized in the following manner [Hor03]:
Standards. The IEEE, ISO, and other standards organizations have pro- duced a broad array of software engineering standards and related documents. Standards may be adopted voluntarily by a software engineering organization or imposed by the customer or other stakeholders. The job of SQA is to ensure that standards that have been adopted are followed and that all work products conform to them. Reviews and audits. Technical reviews are a quality control activity per- formed by software engineers for software engineers (Chapter 16). Their
342 PART 3 QUALITY AND SECURITY
intent is to uncover errors. Audits are a type of review performed by SQA personnel with the intent of ensuring that quality guidelines are being fol- lowed for software engineering work. For example, an audit of the review process might be conducted to ensure that reviews are being performed in a manner that will lead to the highest likelihood of uncovering errors. Testing. Software testing (Chapters 19 through 21) is a quality control func- tion that has one primary goal—to find errors. The job of SQA is to ensure that testing is properly planned and efficiently conducted so that it has the highest likelihood of achieving its primary goal. Error/defect collection and analysis. The only way to improve is to mea- sure how you’re doing. SQA collects and analyzes error and defect data to better understand how errors are introduced and what software engineering activities are best suited to eliminating them. Change management. Change is one of the most disruptive aspects of any software project. If it is not properly managed, change can lead to confusion, and confusion almost always leads to poor quality. SQA ensures that adequate change management practices (Chapter 22) have been instituted. Education. Every software organization wants to improve its software engi- neering practices. A key contributor to improvement is education of software engineers, their managers, and other stakeholders. The SQA organization takes the lead in software process improvement (Chapter 28) and is a key proponent and sponsor of educational programs. Vendor management. Three categories of software are acquired from exter- nal software vendors—shrink-wrapped packages (e.g., Microsoft Office), a tailored shell [Hor03] that provides a basic skeletal structure that is custom tailored to the needs of a purchaser, and contracted software that is custom designed and constructed from specifications provided by the customer orga- nization. The job of the SQA organization is to ensure that high-quality soft- ware results by suggesting specific quality practices that the vendor should follow (when possible) and incorporating quality mandates as part of any contract with an external vendor. Security management. With the increase in cyber crime and new govern- ment regulations regarding privacy, every software organization should insti- tute policies that protect data at all levels, establish firewall protection for mobile apps, and ensure that software has not been tampered with internally. SQA ensures that appropriate process and technology are used to achieve software security (Chapter 18). Safety. Because software is almost always a pivotal component of human- rated systems (e.g., automotive or aircraft applications), the impact of hidden defects can be catastrophic. SQA may be responsible for assessing the impact of software failure and for initiating those steps required to reduce risk. Risk management. Although the analysis and mitigation of risk (Chapter 26) is the concern of software engineers, the SQA organization ensures that risk management activities are properly conducted and that risk-related contingency plans have been established.
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 343
In addition to each of these concerns and activities, SQA works to ensure that software support activities (e.g., maintenance, help lines, documentation, and manuals) are conducted or produced with quality as a dominant concern.
17.3 sQa pro c e s s e s a n d pro d u c t ch a r ac t e r i st i c s
As we begin a discussion of software quality assurance, it’s important to note that SQA procedures and approaches that work in one software environment may not work as well in another. Even within a company that adopts a consistent approach2 to software engineering, different software products may exhibit different levels of quality [Par11].
The solution to this dilemma is to understand the specific quality requirements for a software product and then select the process and specific SQA actions and tasks that will be used to meet those requirements. The Software Engineering Institute’s CMMI and ISO 9000 standards are the most commonly used process frameworks. Each proposes “a syntax and semantics” [Par11] that will lead to the implementation of software engineering practices that improve product quality. Rather than instantiat- ing either framework in its entirety, a software organization can “harmonize” the two models by selecting elements of both frameworks and matching them to the quality requirements of an individual product.
17.4 sQa ta s k s, goa L s, a n d me t r i c s
Software quality assurance is composed of a variety of tasks associated with two dif- ferent constituencies—the software engineers who do technical work and an SQA group that has responsibility for quality assurance planning, oversight, record keeping, analysis, and reporting.
Modern software quality assurance is often data driven, as shown in Figure 17.2. The product stakeholders define goals and quality measures, problem areas are identified, indicators are measured, and a determination is made as to whether or not process changes are needed. Software engineers address quality (and perform quality control activities) by applying solid technical methods and measures, conducting tech- nical reviews, and performing well-planned software testing.
17.4.1 SQA Tasks The charter of the SQA group is to assist the software team in achieving a high- quality end product. The Software Engineering Institute recommends a set of SQA activities that address quality assurance planning, oversight, record keeping, analysis, and reporting. These activities are performed (or facilitated) by an independent SQA group that:
Prepares an SQA plan for a project. The plan is developed as part of project planning and is reviewed by all stakeholders. Quality assurance activi- ties performed by the software engineering team and the SQA group are gov- erned by the plan. The plan identifies evaluations to be performed, audits and
2 For example, CMMI-defined process and practices (Chapter 28).
344 PART 3 QUALITY AND SECURITY
reviews to be conducted, standards that are applicable to the project, proce- dures for error reporting and tracking, work products that are produced by the SQA group, and feedback that will be provided to the software team. Participates in the development of the project’s software process description. The software team selects a process for the work to be performed. The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan. Reviews software engineering activities to verify compliance with the defined software process. The SQA group identifies, documents, and tracks deviations from the process and verifies that corrections have been made. Audits designated software work products to verify compliance with those defined as part of the software process. The SQA group reviews selected work products; identifies, documents, and tracks deviations; verifies that cor- rections have been made; and periodically reports the results of its work to the project manager. Ensures that deviations in software work and work products are docu- mented and handled according to a documented procedure. Deviations may be encountered in the project plan, process description, applicable standards, or software engineering work products. Records any noncompliance, and reports to senior management. Noncompliance items are tracked until they are resolved.
In addition to these activities, the SQA group coordinates the control and manage- ment of change (Chapter 22) and helps to collect and analyze software metrics.
Figure 17.2 Software quality assurance
Analyze the Assessment Software
Quality Assurance
Process
Compare Results
Prepare Criteria
Identify Indicators
Identify a Problem Area
Take Measures to Improve
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 345
17.4.2 Goals, Attributes, and Metrics The SQA activities described in the preceding section are performed to achieve a set of pragmatic goals:
Requirements quality. The correctness, completeness, and consistency of the requirements model will have a strong influence on the quality of all work products that follow. SQA must ensure that the software team has properly reviewed the requirements model to achieve a high level of quality. Design quality. Every element of the design model should be assessed by the software team to ensure that it exhibits high quality and that the design itself conforms to requirements. SQA looks for attributes of the design that are indicators of quality. Code quality. Source code and related work products (e.g., other descriptive information) must conform to local coding standards and exhibit characteris- tics that will facilitate maintainability. SQA should isolate those attributes that allow a reasonable analysis of the quality of code.
Software Quality Assurance
The scene: Doug Miller’s office as the SafeHome software project begins.
The players: Doug Miller, manager of the SafeHome software engineering team, and other members of the product software engineering team.
The conversation: Doug: How are things going with the informal reviews?
Jamie: We’re conducting informal reviews of the critical project elements in pairs as we code but before testing. It’s going faster than I thought.
Doug: That’s good, but I want to have Bridget Thorton’s SQA group conduct audits of our work products to ensure that we’re following our processes and meeting our quality goals.
Venod: Aren’t they already doing the bulk of the testing?
Doug: Yes, they are. But QA is more than test- ing. We need to be sure that our documents are evolving along with our code and that
we’re making sure we don’t introduce errors as we integrate new components. Jamie: I really don’t want to be evaluated based on their findings. Doug: No worries. The audits are focused on conformance of our work products to the re- quirements and the process activities we’ve defined. We’ll only be using audit results to try to improve our processes as well as our software products. Vinod: I must believe it’s going to take more of our time. Doug: In the long run it will save us time when we find defects earlier. It also costs less to fix defects if they’re caught early. Jamie: That sounds like a good thing then. Doug: It’s also important to identify the activi- ties where defects were introduced and add review tasks to catch them in the future. Vinod: That’ll help us determine if we’re sam- pling carefully enough with our review activities. Doug: I think SQA activities will make us a better team in the long run.
safehome
346 PART 3 QUALITY AND SECURITY
Quality control effectiveness. A software team should apply limited resources in a way that has the highest likelihood of achieving a high-quality result. SQA analyzes the allocation of resources for reviews and testing to assess whether they are being allocated in the most effective manner.
Table 17.1 (adapted from [Hya96]) identifies the attributes that are indicators for the existence of quality for each of the goals discussed. Metrics that can be used to indicate the relative strength of an attribute are also shown.
Table 17.1 Software quality goals, attributes, and metrics Source: Adapted from Hyatt, L., and Rosenberg, L., “A Software Quality Model and Metrics for Identifying Project Risks and As- sessing Software Quality,” NASA SATC, 1996.
Goal Attribute Metric
Requirement quality
Ambiguity Number of ambiguous modifiers (e.g., many, large, human-friendly)
Completeness Number of TBA, TBD
Understandability Number of sections/subsections
Volatility Number of changes per requirement
Time (by activity) when change is requested
Traceability Number of requirements not traceable to design/code
Model clarity Number of UML models
Number of descriptive pages per model
Number of UML errors
Design quality Architectural integrity Existence of architectural model
Component completeness Number of components that trace to architectural model
Complexity of procedural design
Interface complexity Average number of pick to get to a typical function or content
Layout appropriateness
Patterns Number of patterns used
Code quality Complexity Cyclomatic complexity
Maintainability Design factors
Understandability Percent internal comments
Variable naming conventions
Percent reused components
Reusability Percent reused component
Documentation Readability index
QC effectiveness Resource allocation Staff hour percentage per activity
Completion rate Actual vs. budgeted completion time
Review effectiveness See review metrics
Testing effectiveness Number of errors found and criticality
Effort required to correct an error
Origin of error
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 347
17.5 fo r m a L ap p roac h e s to sQa In the preceding sections, we have argued that software quality is everyone’s job and that it can be achieved through competent software engineering practice as well as through the application of technical reviews, a multitiered testing strategy, better con- trol of software work products and the changes made to them, and the application of accepted software engineering standards and process frameworks. In addition, quality can be defined in terms of a broad array of quality attributes and measured (indirectly) using a variety of indices and metrics.
Over the past three decades, a small, but vocal, segment of the software engineer- ing community has argued that a more formal approach to software quality assurance is required. It can be argued that a computer program is a mathematical object. A rigorous syntax and semantics can be defined for every programming language, and a rigorous approach to the specification of software requirements is available. If the requirements model (specification) and the programming language can be represented in a rigorous manner, it should be possible to apply mathematic proof of correctness to demonstrate that a program conforms exactly to its specifications.
Attempts to prove programs correct are not new. Dijkstra [Dij76a] and Linger, Mills, and Witt [Lin79], among others, advocated proofs of program correctness and tied these to the use of structured programming concepts. Although formal methods are interesting to some software engineering researchers, commercial developers rarely make use of formal methods in 2018.
17.6 stat i st i ca L so f t wa r e Qua L i t y as s u r a nc e
Statistical quality assurance reflects a growing trend throughout the industry to become more quantitative about quality. For software, statistical quality assurance implies the following steps:
1. Information about software errors and defects is collected and categorized. 2. An attempt is made to trace each error and defect to its underlying cause
(e.g., nonconformance to specifications, design error, violation of stan- dards, poor communication with the customer).
3. Using the Pareto principle (80 percent of the defects can be traced to 20 percent of all possible causes), isolate the 20 percent (the vital few).
4. Once the vital few causes have been identified, move to correct the problems that have caused the errors and defects.
This relatively simple concept represents an important step toward the creation of an adaptive software process in which changes are made to improve those elements of the process that introduce error.
17.6.1 A Generic Example To illustrate the use of statistical methods for software engineering work, assume that a software engineering organization collects information on errors and defects for a period of 1 year. Some of the errors are uncovered as software is being developed. Other defects are encountered after the software has been released to its end users.
348 PART 3 QUALITY AND SECURITY
Although hundreds of different problems are uncovered, all can be tracked to one (or more) of the following causes:
∙ Incomplete or erroneous specifications (IES) ∙ Misinterpretation of customer communication (MCC) ∙ Intentional deviation from specifications (IDS) ∙ Violation of programming standards (VPS) ∙ Error in data representation (EDR) ∙ Inconsistent component interface (ICI) ∙ Error in design logic (EDL) ∙ Incomplete or erroneous testing (IET) ∙ Inaccurate or incomplete documentation (IID) ∙ Error in programming language translation of design (PLT) ∙ Ambiguous or inconsistent human/computer interface (HCI) ∙ Miscellaneous (MIS)
To apply statistical SQA, a table is built (see Table 17.2). The table indicates that IES, MCC, and EDR are the vital few causes that account for 53 percent of all errors. It should be noted, however, that IES, EDR, PLT, and EDL would be selected as the vital few causes if only serious errors are considered. Once the vital few causes are determined, the software engineering organization can begin corrective action. For example, to correct MCC, you might implement requirements gathering techniques (Chapter 7) to improve the quality of customer communication and specifications. To improve EDR, you might acquire tools for data modeling and perform more stringent data design reviews.
Table 17.2 Data collection for statistical SQA
Total Serious Moderate Minor
Error No. % No. % No. % No. %
IES 205 22% 34 27% 68 18% 103 24%
MCC 156 17% 12 9% 68 18% 76 17%
IDS 48 5% 1 1% 24 6% 23 5%
VPS 25 3% 0 0% 15 4% 10 2%
EDR 130 14% 26 20% 68 18% 36 8%
ICI 58 6% 9 7% 18 5% 31 7%
EDL 45 5% 14 11% 12 3% 19 4%
IET 95 10% 12 9% 35 9% 48 11%
IID 36 4% 2 2% 20 5% 14 3%
PLT 60 6% 15 12% 19 5% 26 6%
HCI 28 3% 3 2% 17 4% 8 2%
MIS 56 6% 0 0% 15 4% 41 9%
Totals 942 100% 128 100% 379 100% 435 100%
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 349
It is important to note that corrective action focuses primarily on the vital few. As the vital few causes are corrected, new candidates pop to the top of the stack.
Statistical quality assurance techniques for software have been shown to provide substantial quality improvement (e.g., [Rya11], [Art97]). In some cases, software orga- nizations have achieved a 50 percent reduction per year in defects after applying these techniques.
The application of the statistical SQA and the Pareto principle can be summarized in a single sentence: Spend your time focusing on things that really matter, but first be sure that you understand what really matters!
17.6.2 Six Sigma for Software Engineering Six Sigma is the most widely used strategy for statistical quality assurance in industry today. Originally popularized by Motorola in the 1980s, the Six Sigma strategy “is a business-management strategy designed to improve the quality of process outputs by minimizing variation and causes of defects in processes. It is a subset of the Total Quality Management (TQM) methodology with a heavy focus on statistical applica- tions used to reduce costs and improve quality” [Voe18]. The term Six Sigma is derived from six standard deviations—3.4 instances (defects) per million occurrences— implying an extremely high-quality standard. The Six Sigma methodology defines three core steps:
∙ Define customer requirements and deliverables and project goals via well-defined methods of customer communication.
∙ Measure the existing process and its output to determine current quality performance (collect defect metrics).
∙ Analyze defect metrics and determine the vital few causes.
If an existing software process is in place, but improvement is required, Six Sigma suggests two additional steps:
∙ Improve the process by eliminating the root causes of defects. ∙ Control the process to ensure that future work does not reintroduce the causes
of defects.
These core and additional steps are sometimes referred to as the DMAIC (define, measure, analyze, improve, and control) method.
If an organization is developing a software process (rather than improving an existing process), the core steps are augmented as follows:
∙ Design the process to (1) avoid the root causes of defects and (2) to meet customer requirements.
∙ Verify that the process model will, in fact, avoid defects and meet customer requirements.
This variation is sometimes called the DMADV (define, measure, analyze, design, and verify) method.
A comprehensive discussion of Six Sigma is best left to resources dedicated to the subject. If you have further interest, see [Voe18], [Pyz14], and [Sne18].
350 PART 3 QUALITY AND SECURITY
17.7 so f t wa r e re L i a B i L i t y
There is no doubt that the reliability of a computer program is an important element of its overall quality. If a program repeatedly and frequently fails to perform, it mat- ters little whether other software quality factors are acceptable.
Software reliability, unlike many other quality factors, can be measured directly and estimated using historical and developmental data. Software reliability is defined in statistical terms as “the probability of failure-free operation of a computer program in a specified environment for a specified time” [Mus87]. To illustrate, program X is estimated to have a reliability of 0.999 over eight elapsed processing hours. In other words, if program X were to be executed 1000 times and require a total of 8 hours of elapsed processing time (execution time), it is likely to operate correctly (without failure) 999 times.
Whenever software reliability is discussed, a pivotal question arises: What is meant by the term failure? In the context of any discussion of software quality and reliability, failure is nonconformance to software requirements. Yet, even within this definition, there are gradations. Failures can be only annoying or catastrophic. One failure can be corrected within seconds, while another requires weeks or even months to correct. Complicating the issue even further, the correction of one failure may in fact result in the introduction of other errors that ultimately result in other failures.
17.7.1 Measures of Reliability and Availability Early work in software reliability attempted to extrapolate the mathematics of hard- ware reliability theory to the prediction of software reliability. Most hardware-related reliability models are predicated on failure due to wear rather than failure due to design defects. In hardware, failures due to physical wear (e.g., the effects of tem- perature, corrosion, shock) are more likely than a design-related failure. Unfortunately, the opposite is true for software. In fact, all software failures can be traced to design or implementation problems; wear (see Chapter 1) is not a factor.
There has been an ongoing debate over the relationship between key concepts in hardware reliability and their applicability to software. Although an irrefutable link has yet to be established, it is worthwhile to consider a few simple concepts that apply to both system elements.
If we consider a computer-based system, a simple measure of reliability is mean time between failure (MTBF):3
MTBF = MTTF + MTTR
where the acronyms MTTF and MTTR are mean time to failure and mean time to repair,4 respectively.
3 It is important to note that MTBF and related measures are based on CPU time, not wall clock time.
4 Although debugging (and related corrections) may be required following a failure, in many cases the software will work properly after a restart with no other change.
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 351
Many researchers argue that MTBF is a far more useful measure than other quality-related software metrics discussed in Chapter 23. Stated simply, an end user is concerned with failures, not with the total defect count. Because each defect contained within a program does not have the same failure rate, the total defect count provides little indication of the reliability of a system. For example, consider a program that has been in operation for 3000 processor hours without failure. Many defects in this pro- gram may remain undetected for tens of thousands of hours before they are discovered. The MTBF of such obscure errors might be 30,000 or even 60,000 processor hours. Other defects, not yet discovered, might have a failure rate of 4000 or 5000 hours. Even if every one of the first category of errors (those with long MTBF) is removed, the impact on software reliability is negligible.
However, MTBF can be problematic for two reasons: (1) it projects a time span between failures but does not provide us with a projected failure rate, and (2) MTBF can be misinterpreted to mean average life span even though this is not what it implies.
An alternative measure of reliability is failures in time (FIT)—a statistical measure of how many failures a component will have over 1 billion hours of operation. There- fore, 1 FIT is equivalent to one failure in every billion hours of operation.
In addition to a reliability measure, you should also develop a measure of avail- ability. Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as
Availability = MTTF
MTTF + MTTR × 100%
The MTBF reliability measure is equally sensitive to MTTF and MTTR. The avail- ability measure is somewhat more sensitive to MTTR, an indirect measure of the maintainability of software. Of course, some aspects of availability have nothing to do with failure. For example, scheduling downtime (for support functions) causes the software to be unavailable. For a comprehensive discussion of software reliability measures, see [Laz11].
17.7.2 Use of AI to Model Reliability Some software engineers view data science as the application of artificial intelli- gence techniques to solve software engineering problems. One of the things artificial intelligence methods attempt to do is provide reasonable solutions to problems where the needed data may be incomplete. Software reliability is defined as the probability of failure-free software operation for a specified time period in a spec- ified environment. This means that we can never know the exact moment when a software product will fail because we will never have the complete data needed to calculate the probability.
Software engineers have used statistical techniques based on Bayes’ theorem5 in quantitative decision-making situations for several years. Bayesian inference is a method of statistical inference in which Bayes’ theorem is used to update the probability
5 Bayes’ theorem for conditional probabilities is P(A|B) = (P(B|A) * P(A)) / P(B). For more details, see http://www.statisticshowto.com/bayes-theorem-problems/.
352 PART 3 QUALITY AND SECURITY
for a hypothesis (such as system reliability) as more evidence or information becomes available. Bayesian inference can be used to estimate probabilistic quantities using historic data even when some information is missing. Using Bayesian techniques has allowed for real-time solutions to probability estimation problems that are beyond human reasoning [Tos17].
Proactive failure prediction using machine learning was discussed briefly in Section 15.4.3. It would be nice to be able to predict system failures in subsequent sprints before you deliver the prototype being developed in the current sprint. Mak- ing use of predictive data analytics such as a regression model involving MTBF has been used to estimate where and what types of defects might occur in future prototypes [Bat18].
A genetic algorithm is a heuristic search method used in artificial intelligence and computing. It is used for finding near-optimal solutions to search problems based on the theory of natural selection and evolutionary biology. Genetic algorithms can be used to grow reliability models by discovering relationships in historic system data. These models have been used to identify software components that may fail in the future. Sometimes these models have been created based on metrics estimated from UML models before any codes has been written [Pad17]. This type of work is very important to developers interested in refactoring software products or reusing software components in other products.
17.7.3 Software Safety Software safety is a software quality assurance activity that focuses on the identifica- tion and assessment of potential hazards that may affect software negatively and cause an entire system to fail. If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards.
A modeling and analysis process is conducted as part of software safety. Ini- tially, hazards are identified and categorized by criticality and risk. For example, some of the hazards associated with a computer-based cruise control for an auto- mobile might be: (1) causes uncontrolled acceleration that cannot be stopped, (2) does not respond to depression of brake pedal (by turning off), (3) does not engage when switch is activated, and (4) slowly loses or gains speed. Once these system-level hazards are identified, analysis techniques are used to assign severity and probability of occurrence.6 To be effective, software must be analyzed in the context of the entire system. For example, a subtle user input error (people are system components) may be magnified by a software fault to produce control data that improperly positions a mechanical device. If and only if a set of external environmental conditions is met, the improper position of the mechanical device will cause a disastrous failure. Analysis techniques [Eri15] such as fault tree anal- ysis, real-time logic, and Petri net models can be used to predict the chain of events that can cause hazards and the probability that each of the events will occur to create the chain.
6 This approach is similar to the risk analysis methods described in Chapter 26. The primary difference is the emphasis on technology issues rather than project-related topics.
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 353
Once hazards are identified and analyzed, safety-related requirements can be spec- ified for the software. That is, the specification can contain a list of undesirable events and the desired system responses to these events. The role of software in managing undesirable events is then indicated.
Although software reliability and software safety are related to one another, it is important to understand the subtle difference between them. Software reliability uses statistical analysis to determine the likelihood that a software failure will occur. How- ever, the occurrence of a failure does not necessarily result in a hazard or mishap. Software safety examines the ways in which failures result in conditions that can lead to a mishap. That is, failures are not considered in a vacuum but are evaluated in the context of an entire computer-based system and its environment.
A comprehensive discussion of software safety is beyond the scope of this book. If you have further interest in software safety and related system issues, see [Fir13], [Har12a], and [Lev12].
17.8 th e iso 9000 Qua L i t y sta n da r d s 7
A quality assurance system may be defined as the organizational structure, responsi- bilities, procedures, processes, and resources for implementing quality management [ANS87]. Quality assurance systems are created to help organizations ensure their products and services satisfy customer expectations by meeting their specifications. These systems cover a wide variety of activities encompassing a product’s entire life cycle including planning, controlling, measuring, testing, and reporting, and improv- ing quality levels throughout the development and manufacturing process. ISO 9000 describes quality assurance elements in generic terms that can be applied to any busi- ness regardless of the products or services offered.
To become registered to one of the quality assurance system models contained in ISO 9000, a company’s quality system and operations are scrutinized by third-party auditors for compliance to the standard and for effective operation. Upon successful registration, a company is issued a certificate from a registration body represented by the auditors. Semiannual surveillance audits ensure continued compliance to the standard.
The requirements delineated by ISO 9001:2015 address topics such as management responsibility, quality system, contract review, design control, document and data con- trol, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques. For a software organization to become registered to ISO 9001:2015, it must establish policies and procedures to address each of the requirements just noted (and others) and then be able to demonstrate that these policies and procedures are being followed. If you desire further information on ISO 9001:2015, see [ISO14].
7 This section, written by Michael Stovsky, has been adapted from Fundamentals of ISO 9000, a workbook developed for Essential Software Engineering, a video curriculum developed by R. S. Pressman & Associates, Inc. Reprinted with permission.
354 PART 3 QUALITY AND SECURITY
17.9 th e sQa pL a n
The SQA plan provides a road map for instituting software quality assurance. Devel- oped by the SQA group (or by the software team if an SQA group does not exist), the plan serves as a template for SQA activities that are instituted for each software project.
A standard for SQA plans has been published by the IEEE [IEE17]. The standard recommends a structure that identifies: (1) the purpose and scope of the plan, (2) a description of all software engineering work products (e.g., models, documents, source code) that fall within the purview of SQA, (3) all applicable standards and practices that are applied during the software process, (4) SQA actions and tasks (including reviews and audits) and their placement throughout the software process, (5) the tools and methods that support SQA actions and tasks, (6) software configuration manage- ment (Chapter 22) procedures, (7) methods for assembling, safeguarding, and main- taining all SQA-related records, and (8) organizational roles and responsibilities relative to product quality.
The ISO 9001:2015 Standard The following outline defines the basic elements of the ISO 9001:2015 standard.
Comprehensive information on the standard can be obtained from the International Organization for Standardization (www.iso.ch) and other Internet sources (e.g., www.praxiom.com).
Establish the elements of a quality management system.
Develop, implement, and improve the system.
Define a policy that emphasizes the importance of the system.
Document the quality system.
Describe the process.
Produce an operational manual.
Develop methods for controlling (updating) documents.
Establish methods for record keeping.
Support quality control and assurance.
Promote the importance of quality among all stakeholders.
Focus on customer satisfaction.
Define a quality plan that addresses objectives, responsibilities, and authority.
Define communication mechanisms among stakeholders.
Establish review mechanisms for the quality management system.
Identify review methods and feedback mechanisms.
Define follow-up procedures.
Identify quality resources including personnel, training, and infrastructure elements.
Establish control mechanisms.
For planning.
For customer requirements.
For technical activities (e.g., analysis, design, testing).
For project monitoring and management.
Define methods for remediation.
Assess quality data and metrics.
Define approach for continuous process and quality improvement.
info
CHAPTER 17 SOFTWARE QUALITY ASSURANCE 355
17.10 Su m m a ry
Software quality assurance is a software engineering umbrella activity that is applied at each step in the software process. SQA encompasses procedures for the effective application of methods and tools, oversight of quality control activities such as tech- nical reviews and software testing, procedures for change management, procedures for assuring compliance to standards, and measurement and reporting mechanisms.
To properly conduct software quality assurance, data about the software engineer- ing process should be collected, evaluated, and disseminated. Statistical SQA helps to improve the quality of the product and the software process itself. Software reli- ability models extend measurements, enabling collected defect data to be extrapolated into projected failure rates and reliability predictions.
In summary, you should note the words of Dunn and Ullman [Dun82]: “Software quality assurance is the mapping of the managerial precepts and design disciplines of quality assurance onto the applicable managerial and technological space of software engineering.” The ability to ensure quality is the measure of a mature engineering discipline. When the mapping is successfully accomplished, mature software engineer- ing is the result.
Pro b l e m S a n d Po i n t S to Po n d e r
17.1. Some people say that “variation control is the heart of quality control.” Because every program that is created is different from every other program, what are the variations that we look for and how do we control them?
17.2. Is it possible to assess the quality of software if the customer keeps changing what it is supposed to do?
17.3. Quality and reliability are related concepts but are fundamentally different in a number of ways. Discuss the differences.
17.4. Can a program be correct and still not be reliable? Explain.
17.5. Can a program be correct and still not exhibit good quality? Explain.
17.6. Why is there often tension between a software engineering group and an independent software quality assurance group? Is this healthy?
17.7. You have been given the responsibility for improving the quality of software across your organization. What is the first thing that you should do? What’s next?
17.8. Besides counting errors and defects, are there other countable characteristics of software that imply quality? What are they and can they be measured directly?
17.9. The MTBF concept for reliability is open to criticism. Explain why.
17.10. Consider two safety-critical systems that are controlled by computer. List at least three hazards for each that can be directly linked to software failures.
Design element: Quick Look icon magnifying glass: © Roger Pressman
356
Software Security Engineering Contributed by: Nancy Mead Carnegie Mellon University Software Engineering Institute
18 C H A P T E R
What is it? Software security engineering en- compasses a set of techniques that can im- prove the security of software while it is under development.
Who does it? Although software engineers do not need to become security experts, they need to collaborate with security experts. The security experts may be members of the soft- ware team, on a separate specialized team, or they may be outside consultants.
Why is it important? The media continually re- ports instances of hacking—whether by gang- sters, corporate competitors, hostile nations, or any other bad actor. The consequences for critical infrastructure, financial institutions, health care, and all aspects of modern life are significant.
What are the steps? There are a number of steps that can be taken to ensure that software is secure. We will discuss some of them here and provide pointers to resources for further exploration.
What is the work product? As you will see, there are many work products that are devel- oped in the process of secure software engi- neering. The ultimate work product, or course, is the software that you have developed using secure software engineering practices.
How do I ensure that I’ve done it right? Everything that we will discuss as a method to improve software security, whether at the or- ganizational or project level, can and should be reviewed by the interested parties. In addi- tion, secure development processes can be improved, if they are found to be lacking.
Q u i c k L o o k
attack patterns . . . . . . . . . . . . . . . . . . . . . . . .363 attack surface . . . . . . . . . . . . . . . . . . . . . . . . .366 maturity models . . . . . . . . . . . . . . . . . . . . . . .370 measurement . . . . . . . . . . . . . . . . . . . . . . . . .368 misuse and abuse cases . . . . . . . . . . . . . . . . .363 requirements engineering . . . . . . . . . . . . . . .360 secure coding . . . . . . . . . . . . . . . . . . . . . . . . .367 secure development life-cycle activities . . . .359
security life-cycle models . . . . . . . . . . . . . . . .357 security process improvement . . . . . . . . . . . .370 security risk analysis . . . . . . . . . . . . . . . . . . . .364 software security engineering, importance of . . . . . . . . . . . . . . . . . . . . . . . . .357 threat modeling, prioritization, and mitigation . . . . . . . . . . . . . . . . . . . . . . . . .365
k e y c o n c e p t s
Stop and take a look around. Where do you see software being deployed? Sure, it’s in your laptop, tablet, and cell phone. What about your appliances—refrigera- tor, dishwasher, and so forth? How about your car? Financial transactions—ATM, online banking, financial software, tax software? Your supplier of electricity? Definitely using software. Do you have any wearable devices on? A fit-bit? Maybe you have medical devices, like a pacemaker. The bottom line is: Software is all around us, on us, and sometimes in us. Every software product has the potential to be hacked, sometimes with dire consequences. This is the reason that we, as software engineers, need to be concerned about software security.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 357
18.1 Wh y so f t Wa r e se c u r i t y eng i n e e r i ng is im p o rta n t
Software security is about more than just securing operational software with firewalls, strong passwords, and encryption. It’s also about developing software in such a way that it is more secure from the outset. Techniques are now available that will help us develop software that is significantly more secure than it would be otherwise.
In this chapter, we’re going to look as some of the models and techniques that can help us achieve better software security. We’ll start by looking at security process models. Then we’ll examine specific process activities, including such activities as requirements engineering, misuse or abuse cases, security risk analysis, threat model- ing, attack surface, secure coding, and measurement. We’ll also consider security process improvement models. Finally, we’ll summarize and provide you with a list of references so that you can dive into any of these topics in more depth.
Software security engineering research is a very active area. In this book we’re only providing an overview of methods and tools to support actual practice. There are many books (for example, [Mea16], [Shu13], and [Hel18]) and other resources devoted exclusively to software security engineering, and we will point you to some of them.
18.2 se c u r i t y Li f e-cyc L e mo d e L s
The Microsoft Security Development Lifecycle (SDL) [Mea16] [Mic18] is an industry- leading software security process. A Microsoft-wide initiative and a mandatory policy since 2004, the SDL enabled Microsoft to embed security and privacy in its software and culture. The SDL introduces security and privacy early and throughout all phases of the development process and is without question the most widely known and used security development life-cycle model.
Microsoft defined a collection of principles it calls Secure by Design, Secure by Default, Secure in Deployment, and Communications (SD3+C) to help determine where security efforts are needed. These are as follows [Mic10]:
Secure by Design Secure architecture, design, and structure. Developers consider security issues part of
the basic architectural design of software development. They review detailed designs for possible security issues, and they design and develop mitigations for all threats.
Threat modeling and mitigation. Threat models are created, and threat mitigations are present in all design and functional specifications.
Elimination of vulnerabilities. No known security vulnerabilities that would present a sig- nificant risk to the anticipated use of the software remain in the code after review. This review includes the use of analysis and testing tools to eliminate classes of vulnerabilities.
Improvements in security. Less secure legacy protocols and code are deprecated, and, where possible, users are provided with secure alternatives that are consistent with industry standards.
Secure by Default Least privilege. All components run with the fewest possible permissions. Defense in depth. Components do not rely on a single threat mitigation solution that leaves
users exposed if it fails.
358 PART THREE QUALITY AND SECURITY
Conservative default settings. The development team is aware of the attack surface for the product and minimizes it in the default configuration.
Avoidance of risky default changes. Applications do not make any default changes to the operating system or security settings that reduce security for the host com- puter. In some cases, such as for security products, it is acceptable for a software program to strengthen (increase) security settings for the host computer. The most common violations of this principle are games that either open firewall ports without informing the user or instruct users to open firewall ports without informing users of possible risks.
Less commonly used services off by default. If fewer than 80 percent of a program’s users use a feature, that feature should not be activated by default. Measuring 80 percent usage in a product is often difficult because programs are designed for many different personas. It can be useful to consider whether a feature addresses a core/primary use scenario for all personas. If it does, the feature is sometimes referred to as a P1 feature.
Secure in Deployment Deployment guides. Prescriptive deployment guides outline how to deploy each feature
of a program securely, including providing users with information that enables them to assess the security risk of activating non-default options (and thereby increasing the attack surface).
Analysis and management tools. Security analysis and management tools enable admin- istrators to determine and configure the optimal security level for a software release.
Patch deployment tools. Deployment tools aid in patch deployment.
Communications Security response. Development teams respond promptly to reports of security vulner-
abilities and communicate information about security updates. Community engagement. Development teams proactively engage with users to answer
questions about security vulnerabilities, security updates, or changes in the security landscape.
The secure software development process model looks like the one shown in Figure 18.1.
The Microsoft SDL documentation describes what architects, designers, develop- ers, and testers are required to do for each of the 16 recommended practices. The data that Microsoft collected after implementing the SDL shows a significant reduction in
Figure 18.1 Secure Software Development Process Model at Microsoft Adapted from Shunn, A., et al. Strengths in Security Solutions, Software Engineering Institute, Carnegie Mellon University, 2013. Available at http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=77878.
Training Design Implementation VerificationRequirements Release Response
Core Security Training
Execute Incident
Response Plan
Established Security
Requirements
Create Quality Gates/Bug Bars
Security & Privacy Risk Assessment
Established Design
Requirements
Analyze Attack Surface
Threat Modeling
Use Approved Tools
Deprecate Unsafe
Functions
Static Analysis
Dynamic Analysis
Fuzz Testing
Attack Surface Review
Incident Response Plan
Final Security Review
Release Archive
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 359
vulnerabilities, which led to a need for fewer patches, and thus a significant cost savings. We recommend that you browse the SDL website to learn more about these practices. Since the SDL was developed, there have been numerous papers, books, training, and so on, to go with the SDL model.1
18.3 se c u r e de v e L o p m e n t Li f e-cyc L e ac t i v i t i e s
A different approach that is independent of a life-cycle model is the touchpoints for software security [McG06], which argues that the activities (touchpoints) are what matter, not the model. The activities can be incorporated into any life-cycle model, and thus are considered to be process agnostic. The touchpoints later formed the basis for BSIMM, a maturity model that will be discussed later in this chapter. Some orga- nizations consider the touchpoints to be the minimum set of activities that should be performed in secure software development. A pictorial version of the touchpoints is shown in Figure 18.2. In this diagram, the recommended security activities appear above the corresponding software development activity, or life-cycle phase:
With the SDL and touchpoints in mind, we’ll look at some of the important secure software development activities that are associated with them.
1 More recently, Microsoft has shown how the SDL activities can be integrated with an agile development approach: https://www.microsoft.com/en-us/SDL/Discover/sdlagile.aspx.
Figure 18.2 Software security touchpoints
Investor Feedback
Tests and Test Results
Code
Test Plans
Architecture and Design
Requirements and Use Cases
External Review
Abuse Cases
Security Requirements
Risk-Based Security Tests
Security Operations
Penetration Testing
Code Review (Tools)
Risk Analysis
360 PART THREE QUALITY AND SECURITY
18.4 se c u r i t y re Q u i r e m e n t s eng i n e e r i ng
Although security requirements are an important part of secure software develop- ment, in practice they are often neglected. When they exist, they are often an add- on, copied from a generic list of security features. The requirements engineering that is needed to get a better set of security requirements seldom takes place [All08].
Requirements engineering practice typically addresses desired user features. There- fore, attention is given to the functionality of the system from the user’s perspective, but little attention is given to what the system should not do [Bis02]. Users expect systems to be secure, and these assumptions need to make their way into security requirements for software systems before they are developed, not after the fact. Often the users’ assumptions about security are overlooked because system features are the primary focus.
In addition to security life-cycle models, there are many process models that are specific to security requirements. These include: core security requirements artifacts [Mof04], Software Cost Reduction (SCR) [Hei02], SQUARE (Security QUAlity Requirements Engineering) [Mea05], and Security Requirements Engineering Process (SREP) [Mel06]. For the remainder of this section, we’ll consider SQUARE as a representative example of security life-cycle models.
18.4.1 SQUARE SQUARE is a representative security requirements engineering process model, but it’s important to keep in mind that if you already have a development process model, such as the one presented in Chapter 4, you can just pick up some of the SQUARE steps to enhance your existing model. There’s no need to develop a whole new process to address security in your software development activities. We suggest that you add security definitions to your glossary; perform risk analysis, including iden- tification of potential attacks via misuse cases or threat modeling; develop mitigation strategies; and perform categorization and prioritization of candidate security requirements.
The SQUARE process model provides for eliciting, categorizing, and prioritizing security requirements for software-intensive systems. Its focus is to build security concepts into the early stages of the development life cycle. It can also be used for fielded systems and those undergoing improvements and modifications. The process is shown in Table 18.1, followed by brief descriptions of each step.
18.4.2 The SQUARE Process The SQUARE process is best applied by the project’s requirements engineers and security experts with supportive executive management and stakeholders. Let’s take a look at the steps.2
Step 1. Agree on definitions. So that there is not semantic confusion, this step is needed as a prerequisite to security requirements engineering. On a
2 To dig deeper, see the resources available at the SEPA 9/e website.
361
N um
be r
St ep
In pu
t Te
ch ni
qu es
Pa rt
ic ip
an ts
O ut
pu t
1 A
gr ee
o n
de fin
iti on
s. C
an di
da te
d efi
ni tio
ns f
ro m
IE
EE a
nd o
th er
s ta
nd ar
ds St
ru ct
ur ed
in te
rv ie
w s,
f oc
us g
ro up
St ak
eh ol
de rs
, re
qu ire
m en
ts t
ea m
A gr
ee d-
to d
efi ni
tio ns
2 Id
en tif
y as
se ts
a nd
se
cu rit
y go
al s.
D efi
ni tio
ns , c
an di
da te
a ss
et s
an d
go al
s, b
us in
es s
dr iv
er s,
po
lic ie
s an
d pr
oc ed
ur es
, ex
am pl
es
Fa ci
lit at
ed w
or k
se ss
io n,
s ur
ve ys
, in
te rv
ie w
s St
ak eh
ol de
rs ,
re qu
ire m
en ts
en
gi ne
er
A ss
et s
an d
se cu
rit y
go al
s
3 D
ev el
op a
rti fa
ct s
to
su pp
or t
se cu
rit y
re qu
ire m
en ts
d efi
ni tio
n.
Po te
nt ia
l a rti
fa ct
s (e
.g .,
sc en
ar io
s, m
is us
e ca
se s,
te
m pl
at es
, f or
m s)
W or
k se
ss io
n Re
qu ire
m en
ts
en gi
ne er
N ee
de d
ar tif
ac ts
: s ce
na rio
s,
m is
us e
ca se
s, m
od el
s,
te m
pl at
es , f
or m
s
4 Pe
rfo rm
(s ec
ur ity
) r is
k as
se ss
m en
t. M
is us
e ca
se s,
s ce
na rio
s,
se cu
rit y
go al
s Ri
sk a
ss es
sm en
t m
et ho
d, a
na ly
si s
of
an tic
ip at
ed r
is k
ag ai
ns t
or ga
ni za
tio na
l ris
k to
le ra
nc e,
in cl
ud in
g th
re at
a na
ly si
s
Re qu
ire m
en ts
en
gi ne
er , r
is k
ex pe
rt,
st ak
eh ol
de rs
Ri sk
a ss
es sm
en t
re su
lts
5 Se
le ct
e lic
ita tio
n te
ch ni
qu es
. G
oa ls
, d efi
ni tio
ns , c
an di
da te
te
ch ni
qu es
, e xp
er tis
e of
st
ak eh
ol de
rs , o
rg an
iz at
io na
l st
yl e,
c ul
tu re
, l ev
el o
f se
cu rit
y ne
ed ed
, c os
t-b en
efi t
an al
ys is
, et
c.
W or
k se
ss io
n Re
qu ire
m en
ts
en gi
ne er
Se le
ct ed
e lic
ita tio
n te
ch ni
qu es
6 El
ic it
se cu
rit y
re
qu ire
m en
ts .
A rti
fa ct
s, r
is k
as se
ss m
en t
re su
lts , s
el ec
te d
te ch
ni qu
es A
cc el
er at
ed R
eq ui
re m
en ts
M et
ho d,
Jo
in t
A pp
lic at
io n
D ev
el op
m en
t, in
te rv
ie w
s, s
ur ve
ys , m
od el
-b as
ed
an al
ys is
, c he
ck lis
ts , l
is ts
o f r
eu sa
bl e
re qu
ire m
en ts
t yp
es , d
oc um
en t
re vi
ew s
St ak
eh ol
de rs
fa
ci lit
at ed
b y
re qu
ire m
en ts
en
gi ne
er
In iti
al c
ut a
t se
cu rit
y re
qu ire
m en
ts
7 C
at eg
or iz
e re
qu ire
m en
ts
as t
o le
ve l (
sy st
em ,
so ftw
ar e,
e tc
.) an
d w
he th
er
th ey
a re
r eq
ui re
m en
ts o
r ot
he r
ki nd
s of
c on
st ra
in ts
.
In iti
al r
eq ui
re m
en ts
, ar
ch ite
ct ur
e W
or k
se ss
io n
us in
g a
st an
da rd
s et
o f
ca te
go rie
s Re
qu ire
m en
ts
en gi
ne er
, o th
er
sp ec
ia lis
ts a
s ne
ed ed
C at
eg or
iz ed
r eq
ui re
m en
ts
8 Pr
io rit
iz e
re qu
ire m
en ts
. C
at eg
or iz
ed r
eq ui
re m
en ts
an
d ris
k as
se ss
m en
t re
su lts
Pr io
rit iz
at io
n m
et ho
ds s
uc h
as
A na
ly tic
al H
ie ra
rc hy
P ro
ce ss
(A H
P) ,
Tr ia
ge , W
in -W
in , e
tc .
St ak
eh ol
de rs
fa
ci lit
at ed
b y
re qu
ire m
en ts
e ng
in ee
r
Pr io
rit iz
ed r
eq ui
re m
en ts
9 In
sp ec
t re
qu ire
m en
ts .
Pr io
rit iz
ed r
eq ui
re m
en ts
, ca
nd id
at e
fo rm
al in
sp ec
tio n
te ch
ni qu
e
In sp
ec tio
n m
et ho
d su
ch a
s Fa
ga n
an d
pe er
r ev
ie w
s In
sp ec
tio n
te am
In iti
al s
el ec
te d
re qu
ire m
en ts
, do
cu m
en ta
tio n
of d
ec is
io n-
m ak
in g
pr oc
es s
an d
ra tio
na le
Ta bl
e 18
.1 Th
e SQ
U A
R E
pr oc
es s
362 PART THREE QUALITY AND SECURITY
given project, team members tend to have definitions in mind, based on their prior experience, but those definitions are often different from one another [Woo05]. Sources such as the Institute for Electrical and Electron- ics Engineers (IEEE) and the Software Engineering Body of Knowledge (SWEBOK) provide a range of definitions to select from or tailor [SWE14].
Step 2. Identify assets and security goals. This step occurs at the project’s organizational level and is needed to support software development. Different stakeholders usually have concerns about different assets, and thus have dif- ferent goals. For example, a stakeholder in human resources may be con- cerned about maintaining the confidentiality of personnel records, whereas a stakeholder in a research area may be concerned with ensuring that research project information is not accessed, modified, or stolen.
Step 3. Develop artifacts. This step is necessary to support all subsequent security requirements engineering activities. Often, organizations do not have key documents needed to support requirements definition, or they may not be up to date. This means that a lot of time may be spent backtracking to try to obtain documents, or the team will have to bring them up to date before going further.
Step 4. Perform risk assessment. This step requires an expert in risk assess- ment methods, the support of the stakeholders, and the support of a security requirements engineer. There are a number of risk assessment methods, but regardless of the one that you choose, the outcomes of risk assessment can help in identifying the high-priority security exposures.
Step 5. Select elicitation technique. This step becomes important when there are diverse stakeholders. A more formal elicitation technique, such as the Accelerated Requirements Method [Hub99], Joint Application Design [Woo89], or structured interviews, can be effective in overcoming communica- tion issues when there are stakeholders with different cultural backgrounds. In other cases, elicitation may simply consist of sitting down with a primary stakeholder to try to understand that stakeholder’s security requirements needs.
Step 6. Elicit security requirements. This step encompasses the actual elicitation process using the selected technique. Most elicitation techniques provide detailed guidance on how to perform elicitation. This builds on the artifacts that were developed in earlier steps.
Step 7. Categorize requirements. This step allows the security requirements engineer to distinguish among essential requirements, goals (desired require- ments), and architectural constraints that may be present. This categorization also helps in the prioritization activity that follows.
Step 8. Prioritize requirements. This step depends on the prior step and may also involve performing a cost-benefit analysis to determine which security requirements have a high payoff relative to their cost. Of course prioritization may also depend on other consequences of security breaches, such as loss of life, loss of reputation, and loss of consumer confidence.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 363
Step 9. Requirements inspection. This review activity can be accomplished at varying levels of formality, discussed in Chapter 16. Once inspection is complete, the project team should have an initial set of prioritized security requirements that can be revisited as needed later in the project.
18.5 mi s u s e a n d ab u s e ca s e s a n d at tac k pat t e r n s
Misuse (or abuse) cases can help you view your software in the same way that attackers do. By thinking about negative events, you can better understand how to develop secure software. A misuse case can be thought of as a use case that the attacker initiates.
One of the goals of misuse cases [Sin00] is to decide up front how software should react to potential attacks. You can also use misuse and normal use cases together to conduct threat and hazard analysis [Ale03].
We suggest creating misuse cases through brainstorming. Teaming security experts with subject matter experts (SMEs) covers a lot of ground quickly. During brainstorm- ing, software security experts ask many questions of developers to help identify the places where the system is likely to have weaknesses. This involves a careful look at all user interfaces and considers events that developers assume can’t happen, but that attackers can actually cause to happen.
Here are some questions that need to be considered: How can the system distin- guish between valid and invalid input data? Can it tell whether a request is coming from a legitimate application or a rogue application? Can an insider cause a system to malfunction? Trying to answer such questions helps developers to analyze their assumptions and allows them to fix problems up front.
Misuse cases can be in table or diagram form. Figure 18.3 provides an example misuse case that shows how DroidCleaner malware can successfully attack a cell phone using an open-source e-mail application called K-9. This is extracted from a much larger report that you may wish to study [Ali14].
In this misuse case, the user keeps e-mail on the phone’s external storage area. The attacker gains access to the phone’s storage by compromising the operating system. A common way for the attacker to gain access to the phone is by tricking the user into installing a Trojan, to which the user unwittingly grants access to the drive dur- ing the install process. The attacker is then able to use the Trojan to download files, including the e-mail contents file.
Attack patterns can provide some help by providing a blueprint for creating an attack. For example, buffer overflow is one type of security exploitation. Attackers trying to capitalize on a buffer overflow make use of similar steps [OWA16]. Attack patterns can document these steps (e.g., timing, resources, techniques) as well as practices software developers can use to prevent or mitigate their success [Hog04]. When you’re trying to develop misuse and abuse cases, attack patterns can help.
Misuse cases need to be prioritized as they are generated. In addition, they need to strike the right balance between cost and benefit. The project budget may not allow
364 PART THREE QUALITY AND SECURITY
a software team to implement all defined mitigation strategies at once. In such cases, the strategies can be prioritized and implemented incrementally. The team can also exclude certain cases as being extremely unlikely.
Templates for misuse and abuse cases appear in a number of references. They can be text or diagrams and may be supported by tools. Good sources for templates are in materials by Sindre and Opdahl [Sin01] and Alexander [Ale02].
18.6 se c u r i t y ri s k ana Lys i s
A wide variety of security risk assessment methods have been proposed. Typical examples include SEI CERT’s Security Engineering Risk Analysis (SERA) method3 and the NIST Risk Management Framework (RMF).4
RMF has emerged as an approach that is widely used, providing guidelines for the users. The RMF steps for security are:
∙ Categorize the information system and the information processed, stored, and transmitted by that system based on an impact analysis.
∙ Select an initial set of baseline security controls for the information system based on the security categorization; using an organizational assessment of risk and local conditions, tailor and supplement the security control baseline as needed.
∙ Implement the security controls, and describe how the controls are employed within the information system and its operational environment.
∙ Assess the security controls using appropriate assessment procedures to deter- mine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.
Figure 18.3 Misuse case (exploited by DroidCleaner): Data in an e-mail stored on the smartphone is stolen
User Attacker
Android
Save E-mail Contents
Download E-mail Contents
Grant Access to File
Gain Access to E-mail Contents
Access E-mail Access E-mail<<extend>>
Manage Access <<extend>>
<<Include>> <<Include>> Compromise Phone Security
3 See https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=485410. 4 See https://csrc.nist.gov/publications/detail/sp/800-37/rev-1/final.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 365
∙ Authorize the information system operation based on a determination of the risk to organizational operations and assets, individuals, or other organizations (including national defense), from the operation of the information system that this risk is acceptable.
∙ Monitor the security controls in the information system on an ongoing basis including assessing control effectiveness, documenting changes to the system or its environment of operation, conducting security impact analyses of the associated changes, and reporting the security state of the system to desig- nated organizational officials.
It’s important to note that NIST also provides a set of security controls to select from, thus simplifying the risk assessment work. Recently, the RMF has been modi- fied to include privacy concerns.
18.7 th r e at mo d e L i ng, pr i o r i t i z at i o n, a n d mi t i gat i o n
A threat modeling method (TMM) is an approach for creating an abstraction of a soft- ware system, aimed at identifying attackers’ abilities and goals, and using that abstrac- tion to generate and catalog possible threats that the system must mitigate [Shu16].
STRIDE (an acronym for six threat categories) is representative of a number of threat modeling methods [Mea18] and is the most well-established TMM, representing the state of the practice. At its core, STRIDE requires breaking down a system into its various elements, assessing each of these elements for their vulnerability to threats, and then mitigating these threats [Her06]. In practice, a typical STRIDE implementa- tion includes modeling a system with data flow diagrams (DFDs),5 mapping the DFD elements to the six threat categories, determining the specific threats via checklists or threat trees, and documenting the threats and steps for their prevention [Sca15]. STRIDE can be implemented manually; however, a free Microsoft Secure Develop- ment Lifecycle (SDL) Threat Modeling Tool [Mic17] can also be used. Table 18.2 identifies the security property associated with each of the six threat categories.
Table 18.2 Threat categories and security properties
Threat Security Property
Spoofing Authentication
Tampering Integrity
Repudiation Nonrepudiation
Information disclosure Confidentiality
Denial of service Availability
Elevation of privilege Authorization
5 A brief tutorial on data flow diagrams can be downloaded from https://ratandon.mysite.syr .edu/cis453/notes/DFD_over_Flowcharts.pdf.
366 PART THREE QUALITY AND SECURITY
DFDs are designed to show how a system works by using standard symbols to graphically represent the interaction between data stores (e.g., databases, files, regis- tries), processes (e.g., DLLs, Web services), data flows (e.g., function calls, remote procedure calls), and external entities (e.g., people, other systems) [Sho14]. Once complete, each of these system elements in turn can be associated with one or more relevant threat categories, as depicted in Table 18.3.
In the next stage, the typical STRIDE user works through a checklist (that may be in the form of a threat tree) of specific threats that are associated with each match between a DFD element and threat category. Such checklists are accessible through STRIDE reference books or tools.
Once the threats have been identified, mitigation strategies can be developed and prioritized. Typically, prioritization is based on cost and value considerations. Consid- ering the cost of implementing the mitigation strategy is important, but it’s equally important to also consider the cost of not implementing it, which is reflected in value. Remember that risks that are realized result in costs that are not only expressed in terms of dollars, but could also reflect loss of reputation, loss of trust, and even loss of life.
18.8 at tac k su r fac e
An attack surface can be defined6 in the following manner:
The attack surface describes all of the different points where an attacker could get into a system, and where they could get data out.
The attack surface of an application is:
1. the sum of all paths for data/commands into and out of the application, and 2. the code that protects these paths (including resource connection and authentica-
tion, authorization, activity logging, data validation and encoding), and 3. all valuable data used in the application, including secrets and keys, intellectual
property, critical business data, personal data and PII, and 4. the code that protects these data (including encryption and checksums, access
auditing, and data integrity and operational security controls). [OWA18]
Table 18.3 Threat categories of DFD system elements
Element Spoofing Tampering Repudiation Information Disclosure
Denial of Service
Elevation of Privilege
Data flows X X X
Data stores X X X
Processes X X X X X X
External entity X X
6 See https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Attack_Surface_ Analysis_Cheat_Sheet.md.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 367
The OWASP Foundation [OWA18] states that attack surface analysis is:
. . . targeted to be used by developers to understand and manage application security risks as they design and change an application, as well as by application security specialists doing a security risk assessment. The focus here is on protecting an application from external attack—it does not take into account attacks on the users or operators of the system (e.g., malware injection, social engineering attacks), and there is less focus on insider threats, although the principles remain the same. The internal attack surface is likely to be different to the external attack surface and some users may have a lot of access.
Attack Surface Analysis is about mapping out what parts of a system need to be reviewed and tested for security vulnerabilities. The point of attack surface analysis is to understand the risk areas in an application, to make developers and security specialists aware of what parts of the application are open to attack, to find ways of minimizing this, and to notice when and how the Attack Surface changes and what this means from a risk perspective.
18.9 se c u r e co d i ng
Secure coding is just what the name implies—coding in such a way that vulnerabili- ties are not introduced as a result of coding errors. It’s not surprising that most soft- ware vulnerabilities occur because of sloppy and erroneous coding practices, many of which can be easily avoided.
For example, a condition known as buffer overflow results from one of the most well-known and common coding errors. OWASP7 describes it as follows:
A buffer overflow condition exists when a program attempts to put more data in a buf- fer than it can hold or when a program attempts to put data in a memory area past a buffer. In this case, a buffer is a sequential section of memory allocated to contain anything from a character string to an array of integers. Writing outside the bounds of a block of allocated memory can corrupt data, crash the program, or cause the execution of malicious code.
Buffer overflow is just one example of coding errors that can result in vulnerabil- ities. Fortunately, a number of coding standards now exist to provide guidance on secure coding. The SEI/CERT website8 provides a list of the top-10 secure coding practices:
1. Validate input. Validate input from all untrusted data sources. 2. Heed compiler warnings. Compile code using the highest warning level
available for your compiler and eliminate warnings by modifying the code. 3. Architect and design for security policies. Create a software architecture
and design your software to implement and enforce security policies. 4. Keep it simple. Keep the design as simple and as small as possible. 5. Default deny. Base access decisions on permission rather than exclusion.
7 See https://www.owasp.org/index.php/Buffer_overflow_attack. 8 See https://wiki.sei.cmu.edu/confluence/display/seccode/Top+10+Secure+Coding+Practices.
368 PART THREE QUALITY AND SECURITY
6. Adhere to the principle of least privilege. Every process should execute with the least set of privileges necessary to complete the job.
7. Sanitize data sent to other systems. Sanitize all data passed to complex subsystems such as command shells, relational databases, and commercial off-the-shelf (COTS) components.
8. Practice defense in depth. Manage risk with multiple defensive strategies. 9. Use effective quality assurance techniques. 10. Adopt a secure coding standard.
SEI CERT and others also provide secure coding standards.9 In addition to using a secure coding standard, you should inspect for coding errors that lead to vulnera- bilities. This is just a natural add-on to your normal code inspection and review process (Chapter 16). Static analysis tools10 are used to automatically analyze code and are another mechanism for detecting vulnerabilities due to coding errors.
18.10 me a s u r e m e n t
Developing adequate measures of software security is a difficult problem, and one for which there are differing viewpoints. On the one hand, you can look at the develop- ment processes followed and assess whether the resultant software is likely to be secure. On the other hand, you can look at the incidence of vulnerabilities and suc- cessful break-ins and measure those as a way of assessing software security. However, neither of these measurement approaches will allow you to say with 100 percent cer- tainty that our software is secure. When you add supporting software such as operating systems and external interoperable systems, the measurement of software security becomes even more difficult. Nevertheless some progress has been made.
Measures of software quality can go a long way toward measuring software secu- rity. Specifically, vulnerabilities invariably point to software defects. Although not all software defects are security problems, vulnerabilities in software generally result from a defect of some kind, whether it is in the requirements, architecture, or code. Therefore, measures such as defect and vulnerability counts [Woo14] are useful. Microsoft uses measures such as attack surface analysis and tries to keep the attack surface (places where software can be compromised) to a minimum.
Just as use of maturity models such as CMMI (Chapter 28) suggests that higher- quality software will result, mature security development processes, such as those emphasized by BSIMM,11 will result in more secure software. In some cases, organi- zations are encouraged to identify the unique set of security metrics that are relevant to them. BSIMM makes reference to this, as does SAMM.12
9 See https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards. 10 A list of commercially available tools can be found at https://en.wikipedia.org/wiki/List_of_
tools_for_static_code_analysis. 11 See https://www.bsimm.com/. 12 See https://www.owasp.org/index.php/OWASP_SAMM_Project.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 369
It’s important to note that none of the measurement characteristics implied by various maturity models is perfect. If you follow good secure software development processes, does that guarantee that the software is secure? No! If you find a lot of vulnerabilities, does that mean that most of them have been found or that there are still more to be found because this is a particularly poor piece of software? There are no simple answers. However, to assess vulnerabilities and associated software security, we have to collect data so that patterns can be analyzed over time. If we don’t collect data about software security, we will never be able to measure its improvement.
Tables 18.4 and 18.5 provide examples of how to assess software security during each life-cycle phase. The full tables and discussion can be found in [Mea17] and [Alb10].
Table 18.4 Examples of life-cycle- phase measures
Life-Cycle Phase Example Software Security Measures
Requirements engineering Percentage of relevant software security principles reflected in requirements-specific actions (assuming security principles essential for a given development project have been selected)
Percentage of security requirements that have been subject to analysis (risk, feasibility, cost-benefit, performance trade-offs) prior to being included in the specification
Percentage of security requirements covered by attack patterns, misuse and abuse cases, and other specified means of threat modeling and analysis
Architecture and design Percentage of architectural and design components subject to attack surface analysis and measurement
Percentage of architectural and design components subject to architectural risk analysis
Percentage of high-value security controls covered by a security design pattern
Table 18.5 Example measures based on the seven principles of evidence
Principle Description
Risk Number of active and latent threats, categorized
Incidents reported by category of threat
Likelihood of occurrence for each threat category
Financial and/or human safety estimate of impact for each threat category
Trusted dependencies Number of levels of subcontracting in the supply chain (in other words, have the subcontractors, in turn, executed subcontracts, and what is the depth of this activity?)
Number of suppliers by level
Hierarchical and peer dependencies between suppliers by level
Number of (vetted) trusted suppliers in the supply chain by level
370 PART THREE QUALITY AND SECURITY
18.11 se c u r i t y pro c e s s im p rov e m e n t a n d mat u r i t y mo d e L s
A number of process improvement and maturity models are available for software development in general, such as the Capability Maturity Model Integration (CMMI).13 For cyber security maturity, the CMMI Institute offers a newer product, the Cyber Capability Maturity Management platform.14 OWASP offers the Software Assurance Maturity Model (SAMM).15 SAMM is an open framework to help organizations for- mulate and implement a strategy for software security that is tailored to the specific risks facing the organization.
A comprehensive discussion of these models is beyond the scope of this book. To provide a simple overview, consider the overall objective of SAMM:
∙ Evaluate an organization’s existing software security practices. ∙ Build a balanced software security assurance program in well-defined iterations. ∙ Demonstrate concrete improvements to a security assurance program. ∙ Define and measure security-related activities throughout an organization.
Perhaps the most well-known maturity model that is specifically for software secu- rity is the Building Security in Maturity Model (BSIMM). BSIMM has periodic releases, typically every year or two. The BSIMM model and its recent summarized assessment results can be downloaded from the BSIMM website.16 According to the BSIMM developers, BSIMM is meant to be used by those who create and execute a software security initiative.
All the maturity models mentioned here (and others) have benefits, and the essen- tial elements of the models are freely available. However, sometimes the assessment is done by external entities. It’s possible to do an internal self-assessment and define the associated improvement program, but it requires dedicated resources and effort. Alternatively, some of these organizations offer assessment programs, thus providing an external view of the strengths and areas for improvement within a software organization.
18.12 su m m a ry
All software engineers should have an understanding of what it takes to develop secure software. The steps that are needed to improve the security of your software products are relevant in each activity in the software process, regardless of which process model is used.
Although there are still many open questions, and technologies that need additional research, there are many resources available today to assist with this challenge. For every activity that normally takes place in the software process, try to incorporate
13 See https://cmmiinstitute.com/. 14 See https://cmmiinstitute.com/products/cybermaturity. 15 See https://www.owasp.org/index.php/OWASP_SAMM_Project. 16 See https://www.bsimm.com/.
CHAPTER 18 SOFTWARE SECURITY ENGINEERING 371
security aspects. Models such as Microsoft SDL and the SQUARE model can be assessed to determine which steps you can incorporate into your development process.
Add security to a risk analysis activity, especially using detailed guidance available from NIST. Given the number of secure coding standards already available, it is cer- tainly possible for anyone to learn how to code securely. Inspect your code for remain- ing vulnerabilities. Learn how to identify security holes and develop and prioritize mitigation strategies. Perform static analysis testing on your code. Visit the OWASP and BSIMM websites, among others, to learn about maturity in software security engineering.
As software becomes ever more ubiquitous, the number of vulnerabilities and suc- cessful hacks grow as well. It will take all our efforts to stem the tide, but many of the tools already exist to tackle this problem. The consequences of failing to address software security are high, and the benefits of developing secure software are huge.
Pro b l e m s a n d Po i n t s to Po n d e r
18.1. What is the most important thing that a software team can do to improve software security?
18.2. If you were recommending one activity for your organization to improve software secu- rity, what would it be? If you were recommending multiple activities, what are they, and what would be the priorities, considering that it’s not likely that all of them will be implemented at once?
18.3. How could you incorporate software security into your existing process model or into a new process model?
18.4. Sit down with a colleague and identify security risks on a software project that is in development. Come up with mitigation strategies and prioritize them.
18.5. Are you collecting measurement data that could be used or repurposed to help measure software security? If not, is there data that could easily be collected for this purpose?
18.6. Use the Internet to find out the details needed to create a phishing attack pattern.
18.7. Explain some of the problems that might be encountered if you wait until after the system is completed to address security risks.
18.8. Use the Internet to determine the average cost to the consumer of a single incidence of identity theft.
18.9. Consider a MobileApp that you make use of on your personal phone. List three to five security risks that developers should consider when developing apps like this one.
18.10. Determine the security requirements of a bill-paying wallet-type MobileApp.
Design element: Quick Look icon magnifying glass: © Roger Pressman
