In the second year of the PIACERE project, the DevSecOps framework definition and its components implementation have reached a maturity level that consolidates the PIACERE architecture optimizing each component and their integration.

Fully adopting the DevSecOps methodology and CI/CD approach, modelling language and appropriate tools have been developed and continuously updated for each of the following phases Plan, Code, Build, Test, Release, Deploy, Operate and Monitor.

The CI/CD flow will trigger on Pull/Merge requests to ensure that the code-to-be-integrated passes the defined tests; its phases are represented in the diagram showed below:

Figure 1. CI/CD flow

 

GitLab’s CI/CD functionalities are used to deliver the integration and testing pipeline.

At the technological level, in order to enable the implementation of the DevSecOps and CI/CD model in the software lifecycle management, containerization and microservices strategies have been adopted for the deployment and configuration of PIACERE components. Portability, consistency, orchestration and automation are key benefits of these strategies, along with the ease and speed of provisioning resources in a cloud computing environment.

Going in depth into PIACERE framework logical view, it can be divided into two macro areas called Design and Runtime asshown in Figure 2. The Design involves the initial tasks to design the desired infrastructure using the PIACERE tool while the Runtime involves managing the running infrastructure that was previously designed at design phase.

Figure 2: PIACERE components

In version 2.0 of the framework the general architecture workflow has been improved and updated in the communication mechanisms introducing new interaction capabilities and integration between the two areas. In particular, some components of one area can be recalled in the other one: like e.g. the ICG component (Infrastructural Code Generator), which is mainly involved in the Design, can now also be invoked in the Runtime and similarly IOP (IaC Optimizer Platform), mainly involved in the Runtime, can now be invoked also in the Design. Other new functionalities among the components of the PIACERE framework have been implemented, like the IaC Scan Runner component that has been added to manage the security checks and the updated communication mechanisms between the SelfLearning, SelfHealing and others monitoring components.

The resulting updated workflows are shown below for Design and Runtime areas:

PIACERE Design

The design architecture area describes the components that perform the design and planning phases of the automation code, providing the user with the tools to design, plan and verify the trustworthiness of the IaC code created and packaged for the deployment phase.

The Design components and  their interactions are depicted in the picture below:

Figure 3. PIACERE Design Time workflow

In this phase the DOML (DevOps Modelling Language) and ICG (Infrastructural Code Generator) together with IDE (Integrated Development Environment) are the main components involved to model and generate the IaC (Infrastructure as Code) files to deploy the infrastructure plus software components and their configuration.

Through the IDE GUI, the users can define a model, using the DOML language, that at the end of the design phase will contain the abstract and the concrete specification of the desired Infrastructure.

The IDE component integrates also the VT (Verification Tool) and the ICG. Thanks to the VT, it is possible to validate the defined models and to obtain improvement suggestions.  The ICG tool, triggered by the IDE, will automatically generate the corresponding IaC files for a specific target environment (e.g. AWS, Azure or Openstack). The initial set of supported IaC languages includes Terraform and Ansible, but extensions to other IaC languages are possible.

Other verification tools are focused on static analysis and manage the security checks through the IaC Security Inspectors and Component Security Inspectors (grouped into IaC Scan Runner executor).

The next figure shows the usage sequence of the components involved in the design phase.

Figure 4 PIACERE Design Components’ sequence

The IOP (IaC Optimized Platform) component is the responsible for finding the best possible infrastructure given the input data received. Before generating the IaC, it is possible to invoke the IOP by specifying a set of criteria, to obtain the best solution for the concretization layer. Artificial intelligence algorithms are used to optimise deployment configuration of the IaC on the appropriate infrastructure elements that best fit the predefined constraints. IOP proposes the most optimised deployment configuration for the infrastructure code.

All the information produced at design time is stored in the PIACERE data repositories: DOML models and IaC code in the “DOML and IaC” repository, while the descriptions of the infrastructure elements together with their historical and statistical data are stored in the “Infrastructural Elements Catalogue” repository.

PIACERE Runtime

The PIACERE Runtime architectural domain describes the components necessary for automated deployment, for the dynamic environment created during the deployment phase itself and for the activation and deactivation activities of the infrastructure resources monitoring.

The components and the runtime sequence allowing a user to implement and manage the execution and the monitoring of the environment are listed below:

Figure 5. Runtime components’ sequence

To simplify the reading of the PIACERE Runtime phase, the related workflow has been separated in two diagrams according to different component roles like depicted in the figures 6 and 7.

Figure 6. PIACERE Runtime – Release, configure, check, and deploy workflow

Execution– The PRC (PIACERE Runtime Controller) is the main control component of the PIACERE Runtime. It is a workflow engine that controls the overall workflow within the PIACERE runtime area. The PRC informs the other components about the new deployments that need to be tracked and the deployments that no longer need to be tracked. Actions of the PRC are directed against a specified set of Resource Providers to create the execution environment for the application. Examples of resource providers include AWS, Azure and OpenStack.

In the Runtime workflow, PRC communicates with the IEM (IaC Executor Manager) to initiate a deployment; then IEM retrieves the IaC files related with the initiated deployment and hands over the request to the IaC executors tools (Terraform and Ansible), which will enact the deployment in the infrastructure RP (Resource Provider). Finally, the status of the deployment is updated in the persistence layer so it can be queried appropriately.

Test- The user interacts with the IDE interface to dynamically test the IaC using the Canary environment (Canary Sandbox Environment Provisioner and Canary Sandbox Environment Mocklord).

The second part of the Runtime workflow shows the monitoring activities (figure 7):

Figure 7: PIACERE Runtime – Monitor, SelfHealing, SelfLearning workflow

Monitoring, SelfLearning and SelfHealing- Deep neural networks (deep learning as a subset of the machine learning field) have been adopted in these components and applied to anomaly detection and optimization.

The IA (Infrastructure Advisor) is a group of components responsible for ensuring the optimal deployment of the application specified in the DOML (DevOps Modelling Language). The user interacts with the IA and vice versa via the IDE GUI. The components of IA are:

• IDE Plug-in/Dashboard
• MonitoringController
• PerformanceMonitoring, SecurityMonitoring
• PerformanceSelfLearning, SecuritySelfLearning
• SelfHealing
• IaC Optimizer Platform (IOP)

The MonitoringController interacts with other components in the PIACERE ecosystem with different requests like start and stop operations of the concrete deployments monitoring.

Currently two non-functional measures are supported, performance and security, for each of the following categories: the PerformanceMonitoring component focuses on collecting performance related metrics from infrastructure resources such as memory usage, disk usage, processes, CPU usage. PerformanceMonitoring also sends information about the monitored infrastructure resources to the Infrastructural Elements Catalogue.

The SecurityMonitoring component gathers data from the security monitoring agents and notifies the SelfHealingcomponent on the potential issues to be acted upon and gather data for the SecuritySelfLearning component for detecting anomalies regarding security events.

 

The SelfLearning component is responsible for checking that the various elements present on the platform are in good condition and do not show any degraded or anomalous behaviour. The PerformanceSelfLearning component focuses on incrementally online learning and predicting the performance of the elements to guarantee their constant high-level performance. To do that, the component receives monitoring data from the PerformanceMonitoring component. The PerformanceSelfLearning will use the stored data by the monitoring of performance to forecast events in the infrastructure resources.

The SecuritySelfLearning component makes use of state-of-the-art Natural Language Processing (NLP) to parse log streams and model their normal operating conditions. These models can then be used to detect deviations from the normal behaviour.

The SelfHealing component receives event or prediction notifications from monitoring components. Based on the typology of the notification the SelfHealing component identifies the mitigation strategy to be applied (redeploy, scale, quarantine) and proceeds with its execution by calling the PRC.

Collaboration with Gaia-X

It’s in progress the study to propose ICG as a Gaia-X deployment instructions generator, as PIACERE development tools could be proposed to Providers to simplify the production of Gaia-X deployment instructions[1].

Integration with Gaia-X would be possible and beneficial also for the Infrastructural Elements Catalogue component as Gaia-X language is used to describe resources[2].

 

 

[1] https://www.gxfs.eu/orchestration

[2] https://docs.gaia-x.eu/technical-committee/architecture-document/22.04/self-description