QualiMaster - A configurable real-time date processing infrastructure mastering autonomous quality adaptation

abstract

The evolution of a Software Product Line (SPL) typically affects multiple kinds of artifacts. The intensity (frequency and amount) in which developers change variability information in them was unknown, until we introduced a fine-grained approach for the variability-centric extraction and analysis of changes to code, build, and variability model artifacts. Its application to the commits of the Linux kernel revealed that changes to variability information occur infrequently and only affect small parts of the analyzed artifacts. Further, we outlined how these results may improve certain analysis and verification tasks during SPL evolution. However, the sole analysis of a single SPL did not allow for generic conclusions.

In this paper, we extend our previous work to a comparative evolution analysis of four SPLs. We provide a detailed analysis of the individual intensities of variability changes by applying our updated approach to each SPL. A comparison of these results confirms our findings of infrequent and small changes to variability information in our previous study. However, differences in the details of these changes exist, which we cannot explain solely by the characteristics of the SPLs or their development processes. We  discuss their implications on supporting SPL evolution and on our previous optimization proposals.

abstract

Verification approaches for Software Product Lines (SPL) aim at detecting variability-related defects and inconsistencies. In general, these analyses take a significant amount of time to provide complete results for an entire, complex SPL. If the SPL evolves, these results potentially become invalid, which requires a time-consuming re-verification of the entire SPL for each increment.

However, in previous work we showed that variability-related changes occur rather infrequently and typically only affect small parts of a SPL. In this paper, we utilize this observation and present an incremental dead variable code analysis as an example for incremental SPL verification, which achieves significant performance improvements. It explicitly considers changes and partially updates its previous results by re-verifying changed artifacts only. We apply this approach to the Linux kernel demonstrating that our fastest incremental strategy takes only 3.20 seconds or less for most of the changes, while the non-incremental approach takes 1,020 seconds in median. We also discuss the impact of different variants of our strategy on the overall performance, providing insights into optimizations that are worthwhile.

abstract

Context: Software metrics, as one form of static analyses, is a commonly used approach in software engineering in order to understand the state of a software system, in particular to identify potential areas prone to defects. Family-based techniques extract variability information from code artifacts in Software Product Lines (SPLs) to perform static analysis for all available variants. Many different types of metrics with numerous variants have been defined in literature. When counting all metrics including such variants, easily thousands of metrics can be defined. Computing all of them for large product lines can be an extremely expensive process in terms of performance and resource consumption.

Objective: We address these performance and resource challenges while supporting customizable metric suites, which allow running both, single system and variability-aware code metrics.

Method: In this paper, we introduce a partial parsing approach used for the efficient measurement of more than 42,000 code metric variations. The approach covers variability information and restricts parsing to the relevant parts of the Abstract Syntax Tree (AST).

Conclusions: This partial parsing approach is designed to cover all relevant information to compute a broad variety of variability-aware code metrics on code artifacts containing annotation-based variability, e.g., realized with C-preprocessor statements. It allows for the flexible combination of single system and variability-aware metrics, which is not supported by existing tools. This is achieved by a novel representation of partially parsed product line code artifacts, which is tailored to the computation of the metrics. Our approach consumes considerably less resources, especially when computing many metric variants in parallel.

abstract

EASy-Producer is an open-source research toolset for engineering product lines, variability-rich software ecosystems, and dynamic software product lines. In this tutorial, we will introduce its (textual) variability modeling capabilities realized by the Integrated Variability Modeling Language (IVML) and its model-based development and implementation capabilities, which are realized by the Variability Instantiation Language (VIL) and the Variability Template Language (VTL).

abstract

This summary refers to the paper Metrics for analyzing variability and its implementation in software product lines: A systematic literature review. It was online first in 2018 and was finally published 2019 in the Information and Software Technology (IST) journal.

The use of metrics for assessing software products and their qualities is well established in traditional Software Engineering (SE). However, such traditional metrics are typically not applicable to Software Product Line (SPL) engineering as they do not address variability management, a key part of product line engineering. Over time, various specialized product line metrics for SPLs have been described in literature, but no systematic description of these metrics and their characteristics is currently available.

This paper presents a systematic literature review, where we identify metrics explicitly designed for variability models, code artifacts, and metrics taking both kinds of artifacts into account. This captures the core of variability management for product lines. We discovered 42 relevant papers reporting 147 metrics designed for SPLs.We provide a categorization of these metrics and discuss problematic issues regarding their definitions. We also systematically assess the evaluation status of the metrics showing a current lack of high-quality evaluation in the field. Researchers and practitioners can benefit from the published catalog of variability-aware metrics

abstract

We present preliminary insights into the perception of researchers and practitioners of the software product line (SPL) community on previous, current, and future research efforts. We were particularly interested in up-and-coming and outdated topics and whether the views of academics and industry researchers differ. Also, we compared the views of the community with the results of an earlier literature survey published at SPLC 2018. We conducted a questionnaire-based survey with attendees of SPLC 2018. We received 33 responses (about a third of the attendees) from both, very experienced attendees and younger researchers, and from academics as well as industry researchers. We report preliminary findings regarding popular and unpopular SPL topics, topics requiring further work, and industry versus academic researchers’ views. Differences between academic and industry researchers become visible only when analyzing comments on open questions. Most importantly, while topics popular among respondents are also popular in the literature, topics respondents think require further work have often already been well researched. We conclude that the SPL community needs to do a better job preserving and communicating existing knowledge and particularly also needs to widen its scope.

abstract

This summary refers to the paper Metrics for analyzing variability and its implementation in software product lines: A systematic literature review [EYS19]. The paper was online first in 2018 and was finally published 2019 in the Information and Software Technology (IST) journal.

abstract

This extended abstract summarizes the paper Identifying the Intensity of Variability Changes in Software Product Line Evolution [KGS18] published in the proceedings of the SPLC 2018 [BBB+18].

abstract

Variability-aware metrics are designed to measure qualitative aspects of software product lines. As we identified in a prior SLR [6], there exist already many metrics that address code or variability separately, while the combination of both has been less researched. MetricHaven fills this gap, as it extensively supports combining information from code files and variability models. Further,we also enable the combination of well established single system metrics with novel variability-aware metrics, going beyond existing variabilityaware metrics. Our tool supports most prominent single system and variability-aware code metrics. We provide configuration support for already implemented metrics, resulting in 23,342 metric variations. Further, we present an abstract syntax tree developed for MetricHaven, that allows the realization of additional code metrics.

abstract

During the three decades since the invention of the first variability modeling approach [27], there have been multiple attempts to introduce advanced variability modeling capabilities. More recently, there has been increased attention on textual variability modeling languages. In this paper,we summarize the main capabilities of state of the art textual variability modeling languages, based on [23], including some updates regarding more recent work.We also provide a short discussion of some additional concerns that are relevant when designing (textual) variability modeling languages. From this perspective, the paper also summarizes relevant contributions the authors made to the field in earlier research.

abstract

Variation is central to today’s software development. There are two fundamental dimensions to variation: Variation in time refers to the fact that software exists in numerous revisions that typically replace each other (i.e., a newer version supersedes an older one). Variation in space refers to differences among variants that are designed to coexist in parallel. There are numerous analyses to cope with variation in space (i.e., product-line analyses) and others that cope with variation in time (i.e., regression analyses). The goal of this work is to discuss to which extent product-line analyses can be applied to revisions and, conversely, where regression analyses can be applied to variants. In particular, we discuss requirements to existing analyses and variability representations that are required for those applications. In addition, we discuss which combinationsof product-line and regression analyses are feasible. The overall goal is to increase the efficiency of analyses by exploiting the inherent commonality between variants and revisions.

abstract

Experimentation with software prototypes plays a fundamental role in software engineering research. In contrast to many other scientific disciplines, however, explicit support for this key activity in software engineering is relatively small. While some approaches to improve this situation have been proposed by the software engineering community, experiments are still very difficult and sometimes impossible to replicate.In this paper, we propose the concept of an experimentation workbench as a means of explicit support for experimentation in software engineering research.

In particular, we discuss core requirements that an experimentation workbench should satisfy in order to qualify as such and to offer a real benefit for researchers. Beyond their core benefits for experimentation, we stipulate that experimentation workbenches will also have benefits in regard to reproducibility and repeatability of software engineering research. Further, we illustrate this concept with a scenario and a case study, and describe relevant challenges as well as our experience with experimentation workbenches.

abstract

Context: Software Product Line (SPL) development requires at least concepts for variability implementation and variability modeling for deriving products from a product line. These variability implementation concepts are not required for the development of single systems and, thus, are not considered in traditional software engineering. Metrics are well established in traditional software engineering, but existing metrics are typically not applicable to SPLs as they do not address variability management. Over time, various specialized product line metrics have been described in literature, but no systematic description of these metrics and their characteristics is currently available.

Objective: This paper describes and analyzes variability-aware metrics, designed for the needs of software product lines. More precisely we restrict the scope of our study explicitly to metrics designed for variability models, code artifacts, and metrics taking both kinds of artifacts into account. Further, we categorize the purpose for which these metrics were developed. We also  analyze to what extent these metrics were evaluated to provide a basis for researchers for selecting adequate metrics. Method: We conducted a systematic literature review to identify variability-aware implementation metrics. We discovered 42 relevant papers reporting metrics intended to measure aspects of variability models or code artifacts. Results: We identified 57 variability model metrics, 34 annotation-based code metrics, 46 code metrics specific to composition-based implementation techniques, and 10 metrics integrating information from variability model and code artifacts. For only 31 metrics, an evaluation was performed assessing their suitability to draw any qualitative conclusions.

Conclusions: We observed several problematic issues regarding the definition and the use of the metrics. Researchers and practitioners benefit from the catalog of variability-aware metrics, which is the first of its kind. Also, the research community benefits from the identified observations in order to avoid those problems when defining new metrics.

abstract
Systematic exploration of hypotheses is a major part of any empirical research. In software engineering, we often produce unique tools for experiments and evaluate them independently on different data sets. In this paper, we present KernelHaven as an experimentation workbench supporting a significant number of experiments in the domain of static product line analysis and verification. It addresses the need for extracting information from a variety of artifacts in this domain by means of an open plug-in infrastructure. Available plug-ins encapsulate existing tools, which can now be combined efficiently to yield new analyses. As an experimentation workbench, it provides configuration-based definitions of experiments, their documentation, and technical services, like parallelization and caching. Hence, researchers can abstract from technical details and focus on the algorithmic core of their research problem. KernelHaven supports different types of analyses, like correctness checks, metrics, etc., in its specific domain. The concepts presented in this paper can also be transferred to support researchers of other software engineering domains. The infrastructure is available under Apache 2.0: github.com/KernelHaven. The plug-ins are available under their individual licenses. Video: youtu.be/IbNc-H1NoZU

abstract
Development of Software Product Lines (SPLs) requires additional implementation concepts to manage variability and to facilitate the derivation of individual products based on a common platform. These variability implementation concepts are not required for the development of single systems and, thus, are not considered in traditional software engineering. Metrics are well established in traditional software engineering, but are typically not applicable to SPLs as they do not address variability management. Over time, a number of specialized product line metrics have been described in literature. However, no systematic description of the characteristics of these metrics is currently available. We conducted a systematic literature review to identify variability-aware implementation metrics, designed for the needs of SPLs. We list these metrics according to the measured artifact types: variability models, code artifacts, and combined metrics measuring both artifact types. Further, we analyze to what extent these metrics were evaluated as a basis for qualitative conclusions.

abstract

A number of SAT-based analysis concepts and tools for software product lines exist, that extract code dependencies in propositional logic from the source code assets of the product line. On these extracted conditions, SAT-solvers are used to reason about the variability. However, in practice, a lot of software product lines use integer-based variability. The variability variables hold integer values, and integer operators are used in the conditions. Most existing analysis tools can not handle this kind of variability; they expect pure Boolean conditions.

This paper introduces an approach to convert integer-based variability conditions to propositional logic. Running this approach as a preparation on an integer-based product line allows the existing SAT-based analyses to work without any modiications. The pure Boolean formulas, that our approach builds as a replacement for the integer-based conditions, are mostly equivalent to the original conditions with respect to satisfiability. Our approach was motivated by and implemented in the context of a real-world industrial case-study, where such a preparation was necessary to analyze the variability.

Our contribution is an approach to convert conditions, that use integer variables, into propositional formulas, to enable easy usage of SAT-solvers on the result. It works well on restricted variables (i.e. variables with a small range of allowed values); unrestricted integer variables are handled less exact, but still retain useful variability information.

abstract

KernelHaven is an open infrastructure for Software Product Line (SPL) analysis. It is intended both as a production-quality analysis tool set as well as a research support tool. Its design follows the principle of an experimentation workbench [13] to support researchers in systematically exploring research hypothesis. For flexibility and ease of experimentation KernelHaven components are plug-ins for extracting certain information from SPL artifacts and processing this information, e.g., to check the correctness and consistency of variability information or to apply metrics. A configuration-based setup along with automatic documentation functionality allows different experiments and supports their easy reproduction.

Here, we describe KernelHaven as a product line analysis research tool and highlight its basic approach as well as its fundamental capabilities. In particular, we describe available information extraction and processing plug-ins and how to combine them. On this basis, researchers and interested professional users can rapidly conduct a first set of experiments. Further, we describe the concepts for extending KernelHaven by new plug-ins, which reduces development effort when realizing new experiments.

abstract

EASy-Producer is an open-source research toolset for engineering product lines, variability-rich software ecosystems, and dynamic software product lines. In this tutorial, we will focus on its (textual) variability modeling capabilities as well as its configuration and validation functionality. Further, we will provide an outlook on how EASy-Producer can be applied to variability instantiation.

abstract

EASy-Producer is an open-source research toolset for engineering product lines, variability-rich software ecosystems, and dynamic software product lines. In this tutorial, we will focus on its model-based development and implementation capabilities, which are realized by the Variability Instantiation Language (VIL) and the Variability Template Language (VTL). Further, we will provide a basic introduction into the Integrated Variability Modeling Language (IVML) in order to use the provided information of IVML variability models and configurations during instantiation defined with VIL and as part of the templates created with VTL.

abstract

The evolution of a Software Product Line (SPL) typically affects a variety of artifact types. The intensity (the frequency and the amount) in which developers change variability information in these different types of artifacts is currently unknown. In this paper, we present a fine-grained approach for the variability-centric extraction and analysis of changes to code, build, and variability model artifacts introduced by commits. This approach complements existing work that is typically based on a feature-perspective and, thus, abstracts from this level of detail. Further, it provides a detailed understanding of the intensity of changes affecting variability information in these types of artifacts. We apply our approach to the Linux kernel revealing that changes to variability information occur infrequently and only affect small parts of the analyzed artifacts. Further, we outline how these results may improve certain analysis and verification tasks during SPL evolution.

abstract

Ideally, a variability model is a correct and complete representation of product line features and constraints among them. Together with a mapping between features and code, this ensures that only valid products can be configured and derived. However, in practice the modeled constraints might be neither complete nor correct, which causes problems in the configuration and product derivation phases. This paper presents an approach to reverse engineer variability constraints from the implementation, and thus improve the correctness and completeness of variability models.

We extended the concept of feature effect synthesis [18] to extract domain knowledge from code artifacts of the Bosch PS-EC product line. We present an application of the approach to a large-scale industrial product line and discuss its required modifications to obtain meaningful results in an industrial case.

abstract
Ideally the variability of a product line is represented completely and correctly by its variability model. However, in practice additional variability is often represented on the level of the build system or in the code. Such a situation may lead to inconsistencies, where the actually realized variability does not fully correspond to the one described by the variability model. In this paper we focus on configuration mismatches, i.e., cases where the effective variability differs from the variability as it is represented by the variability model. While previous research has already shown that these situations still exist even today in well-analyzed product lines like Linux, so far it was unclear under what circumstances such issues occur in reality. In particular, it is open what types of configuration mismatches occur and how severe they are. Here, our contribution is to close this gap by presenting a detailed manual analysis of 80 configuration mismatches in the Linux 4.4.1 kernel and assess their criticality. We identify various categories of configuration issues and show that about two-thirds of the configuration mismatches may actually lead to kernel misconfigurations.

abstract
Development of Software Product Lines (SPLs) requires additional implementation concepts to manage variability and to facilitate the derivation of individual products based on a common platform. These variability implementation concepts are not required for the development of single systems and, thus, are not considered in traditional software engineering. Metrics are well established in traditional software engineering, but are typically not applicable to SPLs as they do not address variability management. Over time, a number of specialized product line metrics have been described in literature. However, no systematic description of the characteristics of these metrics is currently available. We conducted a systematic literature review to identify variability-aware implementation metrics, designed for the needs of SPLs. We list these metrics according to the measured artifact types: variability models, code artifacts, and combined metrics measuring both artifact types. Further, we analyze to what extent these metrics were evaluated as a basis for qualitative conclusions.

abstract
Software Product Line (SPL) evolution affects a variety of artifact types, each containing their artifact-specific information as well as variability information. While the former information type defines the basic content of an artifact, like the general program definition in code artifacts, the latter supports the customization of these artifacts for different products of the SPL. Existing work that aims at characterizing the state and evolution of a product line identifies general SPL evolution scenarios or evaluates the (co-)evolution of variability information in different types of artifacts. However, these results are typically based on a feature perspective, which abstracts from the SPL evolution at large. We argue that an artifact-based analysis of product line evolution will complement existing work by analyzing the intensity of changes to different artifact and information types. In this report, we therefore present an approach for and the results of the extraction and analysis of changes introduced to artifact-specific and variability information in code, build and variability model artifacts. This approach has been developed for and applied to the Linux kernel. In order to broaden our analysis, we also apply the approach to the Coreboot firmware. The results reveal the intensity of changes over the evolution history of both SPLs. Further, we compare the intensity of changes across both SPLs with respect to the size of the changes as well as over time.

abstract
In contrast to traditional software systems, the evolution of a Software Product Line (SPL) affects not only artifacts like source code or requirements, but also variability information, which supports the customization of these artifacts across different products of the SPL. While some work exists that aims at characterizing the state and evolution of a product line from a feature perspective, this abstracts away the details of code evolution, hence, ignoring aspects like the difference in size of features. In this paper, we present an approach for the extraction and analysis of changes introduced to code, build and variability model artifacts. This approach has been developed for and applied to the Linux product line.