Q 670. Compare White Box / Black Box / Gray Box software testing approaches against each other. Which of the these is most suitable to do with tech enabled solutions in DMAIC projects? Justify your answers with examples.

 

Note for website visitors -

• This platform hosts two weekly questions, one on Tuesday and the other on Friday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be approved. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honorable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error prone as our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://crossplag.com/ai-content-detector/. Only answers with less than 15-20% AI-generated content will be approved.

Software testing:

• In every industry Quality testing is the most important phase, once the product cleared from this phase it will be sent for customer. In most of the companies, quality function is considered as the customer face in the business. In software development, testing is also one of the crucial phases, before releasing the developed product to users.  In manufacturing, products are checking using some sampling method, which represent the population. In any software product the life cycle is very high, hence it has been rigorously tested against the user requirement, the quality of code and the external access (penetration test) etc. Here the testing done for each requirement in multiple ways. Here sampling cannot be considered as the client might have use the products in various ways in their day-to day business.
• White box, Black box and grey box three kind of testing techniques used in software testing.             All these types of tests are important to verify the functionality & integrity of the software.
• Black box testing: This test is performed by the software testers. The code internal structure is unknown to the tester. The focus more about the user requirement, functionalities are fulfilled or not. This test can be carried out easily(quick) and it can be performed by testers with no knowledge of the code. This is sometimes performed by the independent teams (out sourced). It is also called as behavioral testing. In this test we only focus on input and Output. This test doesn’t consider about the how the it is processing.
• The tests performed are broadly categories into function & non-functional. The details are as below.
  • Function testing – Smoke testing, sanity testing, Integration testing, System testing, Regression testing, user acceptance testing
  • Non-functional testing – Usability test, load test, performance test, compatibility test, stress testing, scalability testing.
• The black box testing techniques are mentioned below
  • Graph based testing - All objects(node) have the expected relationship to one another.
  • Equivalence partitioning – In this step, divides the input domain into classes of data
  • Boundary value analysis -Not completely clear, a more no. of errors will occur at the boundaries
  • Comparison testing – To minimize the redundant errors occurs in hardware or software
  • Orthogonal array test – which the input domain is relatively limited
• White Box testing: This test performed by the developers with detailed knowledge of codebase itself. The code internal structure is known to the tester. It is about to comparing how the system actually functioning against the how the systems should function. It is very comprehensive testing (more time & high cost). If this is done during development, the bugs can be avoided during deployment. (Preventive). As it is very technical test, there may be chance of missing simple bugs (low level). This tests sometimes referred as clear test, transparent testing.
• Testing process include the below steps:
  • Identify the which features to be tested
  • Plot all possible paths in the flowgraph
  • Identify all possible paths
  • Create test cases
  • Execute the test cases
  • Repeat the cycle as necessary.
• The White box testing techniques are mentioned below
  • Decision coverage – This is the most important once in white box, because it provides data on the true of false results of Boolean expressions in the source code.
  • Condition coverage – It covers the expressions
  • Multiple condition coverage – This verifies the different combinations of conditions and evaluate the decision that the code make for each combination
  • Finite state machine coverage
  • Control flow testing – Seeks to establish the execution of the program by using a simple control structure.
  • Statement coverage – The testers should cover as much as possible of the source code
  • Branch coverage – This part basically talks about how wide the coverage of particular elements of the cod
  • Path coverage – Assess paths within a software application
• Grey Box testing: It is a combination of black and white box testing, where the internal structure of the code is partially known. The focus more about the how the system is working and whether this meets the end-user requirement.
• The Grey box testing techniques are mentioned below
  • Matrix testing – States the port of the project
  • Regression testing – Re running of test cases, if any new changes made in code
  • Pattern testing – Re running of test cases, if any new changes made in code
  • Orthogonal array testing – subset of all possible combination
• Based on the above detailed understanding, if any DMAIC project is associated with digital/tech solution, then as a business excellence professional, we should focus on black box, grey box testing to ensure that the given solution is giving the correct output also fulfilling the project outcome. But we need to take help of company IT team to go through the white box testing. By Black box, grey box testing, we will ensure that the solution is delivering the value to the business. But the underlaying model design, language used in the code, structure of the code should be flexible enough for the upgradation of IT systems as well as aligned with the company IT guidelines & futuristic scenarios. To ensure that the solution is sustainable for long term, we should do the white box testing.
• For example, we are working a project which predicts the sheet break in the paper machine, which is resulting into low productivity & low OEE. After doing the historical analysis & found the root cause at multiple places, which is very difficult to monitor & control by the shop floor operators. Then the team come up with the tech solution, which identify the variation in the critical causes and suggest operator about the adjustment of the parameters. In this case, as BE professional, we will ensure that the tech solution is properly identifying the variation of critical causes and timely giving the alerts to operators (Black box). But this prediction tool is designed and developed by the third-party vendor. Hence it is very difficult to know, what language they write the code, what are logic they built inside the code. Then we should involve the IT/SF professional to understand the technical details (such as conditions, logics, version control, scalability, performance) which refers as white box. As BE professional, we can play role, while building the models and at the end we should also focus on the in what condition, the crucial KPIs are identifying (Grey Box).
• In Summary, if any DMAIC project associated with tech solution, BE professional will do the Black box, grey box testing to ensure the projects out come are matching with the business expectation. Also take help of IT/SF professional to carry out the white box testing, to ensure that the developed solution is sustainable, capable enough.

Differences between White Box / Black Box / Gray Box software testing:

White Box testing	Black Box testing	Gray Box testing
Testers have sufficient knowledge of algorithms, codes, and logic	Testers do not know the codes/logic and internal structures/algorithms. This is mostly business user acceptance testing where business units may not have any IT background and try to check all desired functionalities for an external user perspective	Testers have partial knowledge of codes/logic
Thorough testing can be possible due to the availability of  information	Limited testing can be possible and support from AI tools can help to do more effective testing	This is in between the white box and black box testing but more effective output is possible using AI-enabled tools
Detailed code-level testing by internal IT organization user	Functionality-based testing with no code knowledge – mostly for business users	The middle ground – combining partial code insights with functional testing

 

 

In DMAIC, the suitability of white box, Gray box, or black box testing depends on the specific phase and nature of the tech-enabled solution. Please find some examples

Real-life example:

Consider that you are renovating an old house to sell

DMAIC: is like renovating the house: First, you figure out what needs fixing (Define), measure how bad it is (Measure), diagnose the root problems (Analyze), make the improvements (Improve), and then maintain the new and improved house (Control)

Box Testing:
White Box: You're the handyman with the blueprints, poking into the walls and checking the wiring.
Gray Box: You're the inspector with partial blueprints, checking both the visible structure and peeking into the attic.
Black Box: You're the potential buyer, looking at the house from the outside, trying to see if the doorbell works and if the lights turn on without caring about what's behind the walls.

 Software Development Industry:

DMAIC Project Objective: Improve the reliability and performance of a financial software application.
 

Define: The problem is identified as frequent software crashes and slow performance during peak usage times.
Measure: Collect data on crash frequency, response times, and user complaints and perform MSA
Analyze: Examine the software development process, code quality, and user interaction patterns
Testing Approaches:

White Box Testing: Developers conduct thorough code reviews and unit testing. They look at the software’s internal structure to find inefficiencies, bugs, and areas for optimization.

Gray Box Testing: Quality assurance teams with some knowledge of the codebase perform integration tests and simulate real-world user scenarios. They check how different modules interact and ensure overall system stability.

Black Box Testing: End-users test the application’s functionality without knowing its internal workings. They focus on the user interface, performance, and overall user experience.
Improve:  Refactor code, optimize algorithms, and enhance the software architecture based on the testing insights.

Control: Implement continuous testing using white, gray, and black box methods to ensure sustained performance and reliability.

 

Healthcare Industry:

DMAIC Project Objective: Enhance the accuracy and efficiency of a medical diagnostic system in a hospital.

Define: The problem is identified as inconsistencies and errors in diagnostic results from a medical imaging system.

Measure: Gather data on diagnostic accuracy, error rates, and turnaround times and perform MSA

Analyze: Investigate the diagnostic process, from image capture to diagnosis.

Testing Approaches:

White Box Testing:  Medical technicians and IT professionals dissect the diagnostic system's software and hardware components, ensuring each part works correctly and efficiently. They analyze the algorithms used for image processing and diagnosis.

Gray Box Testing: Healthcare IT specialists with partial knowledge of the system evaluate the interaction between the software, hardware, and medical staff. They perform tests to ensure data integrity and accuracy of the diagnostic results.

Black Box Testing: Doctors and nurses use the diagnostic system without understanding its internal workings. They focus on the output, ensuring that the diagnostic reports are accurate and user-friendly.

Improve: Based on findings, upgrade the diagnostic software, train staff, and refine data processing protocols.

Control: Regularly test the system using all three approaches to ensure ongoing accuracy and efficiency.
 

Software testing is a process of evaluating and validating the software for identifying the errors/bugs/defects, conformance to the requirements and for enhancing the quality of the software. There are different types of testing approaches applied based on focus area required for validation by different types of testing - Manual, Automated, Functional and Non-functional. 

Different types of Testing approaches and applicability to DMAIC

• Black Box Testing: Black Box testing focuses on software’s external attributes and behavior without knowledge of its internal workings. Testers evaluate the application from a user’s perspective. This is a Low granularity approach. This is applicable/suitable for functional or business testing which is based on requirements and functional specifications, and this is less exhaustive than other approaches.
  • Steps of testing
    • Define test cases based on user stories or requirements.
    • Execute tests without knowledge of internal code.
    • Verify expected outcomes.                               
  • Example:
    • Validating checkout process by testing user flows and expected outcomes in an online website/application focused on clothing e-commerce.
    • For a mobile app, test login, navigation, and data retrieval
• White Box Testing: White Box testing examines the internal code, data structures, and logic flow. It’s also known as glass-box testing or structural testing. This is a high granularity approach. This is suitable for algorithm testing and provides better variety and depth in test cases due to knowledge of internals. 
  • Steps of testing
    • Review code and identify paths.
    • Create test cases targeting specific code segments.
    • Execute tests with knowledge of internals.                         
  • Example:
    • Validating analytical model/algorithm built to provide insights on prediction of stock movement in capital markets
    • Test a function that calculates shipping costs
• Gray Box Testing: Gray Box testing merges elements of both Black Box and White Box testing. It uses inputs and outputs for testing but considers information about the code. This is a medium granularity approach. This is suited well for web application testing and considers high level design and compatible conditions
  • Steps of testing
    • Understand high-level design.
    • Create test cases considering both inputs and code.
    • Execute tests with partial knowledge
  • Example:
    • Testing a web-based CRM system, considering both user interactions and database queries
    • Testing an API by sending valid and invalid requests

DMAIC (Define, Measure, Analyze, Improve, Control) is a structured Lean Six Sigma approach for process improvement. 

• Define: Identify project goals and customer requirements and also define the test objectives.
  • Example: To develop an application/website with an easy and effective UI for customers for seamless purchasing of apparel with secure payment platform integrated with robust testing.
• Measure: Gather data and determine current performance for baseline
  • Example: Identify the number defects relating to failed purchases  
• Analyze: Identify root causes of defects or inefficiencies by using Gray box or White box testing approaches
  • Example: Primary root causes are frequent website/application down and payment issues
• Improve: Develop and implement solutions post validation using relevant testing approaches
  • Example: Payment API integration testing and Mobile app development testing for more reachability
• Control: Establish controls to sustain improvements by monitoring ongoing performance
  • Example: Secured and high-performance website for avoiding frequent payment/website related crashes

In DMAIC projects, Black Box Testing is often suitable for defining requirements and measuring performance, while Gray Box Testing can help analyze and improve processes. White Box Testing may be less common in DMAIC projects but could be useful for specific scenarios, such as algorithm validation in software development. Remember that the choice depends on the specific context and goals of the project.

 

 

Very interesting answers specially with respect to usage of testing methods in DMAIC. Strongly recommend to review all answers.

 

The best answer has been provided by Nethaji. Well done.