Deployment

This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report As per the reports of Global Market Insight, the automation testing market size surpassed $20 billion (USD) in 2022 and is projected to witness over 15% CAGR from 2023 to 2032. This can be attributed to the willingness of organizations to use sophisticated test automation techniques as part of the quality assurance operations (QAOps) process. By reducing the time required to automate functionalities, it accelerates the commercialization of software solutions. It also offers quick bug extermination and post-deployment debugging, and it helps the integrity of the software through early notifications of unforeseen changes. Figure 1: QAOps cycle What Is the Automated Testing Lifecycle? The automation testing lifecycle is a multi-stage process that covers the process of documentation, planning, strategy, and design. The cycle also involves development of the use cases using technology and deploying it to an isolated system that could run on specific events or based on a schedule. Phases of the Automated Testing Lifecycle There are six different phases of the automated testing lifecycle: Determining the scope of automation Architecting the approach for test automation (tools, libraries, delivery, version control, CI, other integrations) Setting the right test plan, test strategy, and test design Automation environment setup Test script development and execution Analysis and generation of test reports Figure 2: Automated testing lifecycle Architecture Architecture is an important part of the automation lifecycle that leads to defining the strategy required to start automation. In this phase of the lifecycle, the people involved need to have a clear understanding of the workflows, executions, and required integrations with the framework. Tools of the Trade In today’s automation trends, the new buzzword is "codeless automation," which helps accelerate test execution. There are a few open-source libraries as well, such as Playwright, which use codeless automation features like codegen. Developing a Framework When collaborating in a team, a structured design technique is required. This helps create better code quality and reusability. If the framework is intended to deliver the automation of a web application, then the team of automation testers need to follow a specific design pattern for writing the code. Execution of Tests in Docker One important factor in today’s software test automation is that the code needs to be run on Docker in isolation every time the test runs are executed. There are a couple of advantages to using Docker. It helps set up the entire testing environment from scratch by removing flaky situations. Running automation tests on containers can also eliminate any browser instances that might have been suspended because of test failures. Also, many CI tools support Docker through plugins, and thus running test builds by spinning a Docker instance each time can be easily done. Continuous Testing Through CI When it comes to testing in the QAOps process, CI plays an important role in the software release process. CI is a multi-stage process that runs hand in hand when a commit is being made to a version control system to better diagnose the quality and the stability of a software application ready for deployment. Thus, CI provides an important aspect in today’s era of software testing. It helps to recover integration bugs, detect them as early as possible, and keep track of the application's stability over a period of time. Setting up a CI process can be achieved through tools like Jenkins and CircleCI. Determining the Scope of Test Automation Defining the feasibility for automation is the first step of the automation lifecycle. This defines the scope and automates the required functionality. Test Case Management Test case management is a technique to prioritize or select the broader scenarios from a group of test cases for automation that could cover a feature/module or a service as a functionality. In order to ensure the top quality of products, it is important that complexity of test case management can scale to meet application complexity and the number of test cases. The Right Test Plan, Test Strategy, and Test Design Selecting a test automation framework is the first step in the test strategy phase of an automated testing lifecycle, and it depends on a thorough understanding of the product. In the test planning phase, the testing team decides the: Test procedure creation, standards, and guidelines Hardware Software and network to support a test environment Preliminary test schedule Test data requirements Defect tracking procedure and the associated tracking tool Automation Environment Setup The build script to set up the automation environment can be initiated using a GitHub webhook. The GitHub webhook can be used to trigger an event in the CI pipeline that would run the build scripts and the test execution script. The build script can be executed in the CI pipeline using Docker Compose and Docker scripts. docker-compose.yml: version: "3.3" services: test: build: ./ environment: slack_hook: ${slack_hook} s3_bucket: ${s3_bucket} aws_access_key_id: ${aws_access_key_id} aws_secret_access_key: ${aws_secret_access_key} aws_region: ${aws_region} command: ./execute.sh --regression Dockerfile FROM ubuntu:20.04 ENV DEBIAN_FRONTEND noninteractive # Install updates to base image RUN apt-get -y update && \ apt-get -y install --no-install-recommends tzdata && \ rm -rf /var/lib/apt/lists/* # Install required packages ENV TZ=Australia/Melbourne RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone RUN dpkg-reconfigure --frontend noninteractive tzdata RUN apt-get -y update && \ apt-get install -y --no-install-recommends software-properties-common \ apt-utils \ curl \ wget \ unzip \ libxss1 \ libappindicator1 \ libindicator7 \ libasound2 \ libgconf-2-4 \ libnspr4 \ libnss3 \ libpango1.0-0 \ fonts-liberation \ xdg-utils \ gpg-agent \ git && \ rm -rf /var/lib/apt/lists/* RUN add-apt-repository ppa:deadsnakes/ppa # Install chrome RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/ google-chrome.list' RUN apt-get -y update \ && apt-get install -y --no-install-recommends google-chrome-stable \ && rm -rf /var/lib/apt/lists/* # Install firefox RUN apt-get install -y --no-install-recommends firefox # Install python version 3.0+ RUN add-apt-repository universe RUN apt-get -y update && \ apt-get install -y --no-install-recommends python3.8 \ python3-pip && \ rm -rf /var/lib/apt/lists/* RUN mkdir app && mkdir drivers # Copy drivers directory and app module to the machine COPY app/requirements.txt /app/ # Upgrade pip and Install dependencies RUN pip3 install --upgrade pip \ -r /app/requirements.txt COPY app /app COPY drivers /drivers # Execute test ADD execute.sh . RUN chmod +x execute.sh ENTRYPOINT ["/bin/bash"] Seeding Test Data in the Database Seed data can be populated for a particular model or can be done using a migration script or a database dump. For example, Django has a single-line loader function that helps seeding data from a YML file. The script to seed the database can be written in a bash script and can be executed once every time a container is created. Take the following code blocks as examples. entrypoint.sh: #!/bin/bash set -e python manage.py loaddata maps/fixtures/country_data.yaml exec "$@" Dockerfile FROM python:3.7-slim RUN apt-get update && apt-get install RUN apt-get install -y libmariadb-dev-compat libmariadb-dev RUN apt-get update \ && apt-get install -y --no-install-recommends gcc \ && rm -rf /var/lib/apt/lists/* RUN python -m pip install --upgrade pip RUN mkdir -p /app/ WORKDIR /app/ COPY requirements.txt requirements.txt RUN python -m pip install -r requirements.txt COPY entrypoint.sh /app/ COPY . /app/ RUN chmod +x entrypoint.sh ENTRYPOINT ["/app/entrypoint.sh"] Setting up the Workflow Using Pipeline as Code Nowadays, it is easy to run builds and execute Docker from CI using Docker plugins. The best way to set up the workflow from CI is by using pipeline as code. A pipeline-as-code file specifies actions and stages for a CI pipeline to perform. Because every organization uses a version control system, changes in pipeline code can be tested in branches for the corresponding changes in the application to be deployed. The following code block is an example of pipeline as code. config.yml: steps: - label: ":docker: automation pipeline" env: VERSION: "$BUILD_ID" timeout_in_minutes: 60 plugins: - docker-compose#v3.7.0: run: test retry: automatic: - exit_status: "*" limit: 1 Checklist for Test Environment Setup Test data List of all the systems, modules, and applications to test Application under test access and valid credentials An isolated database server for the staging environment Tests across multiple browsers All documentation and guidelines required for setting up the environment and workflows Tool licenses, if required Automation framework implementation Development and Execution of Automated Tests To ensure test scripts run accordingly, the development of test scripts based on the test cases requires focusing on: Selection of the test cases Creating reusable functions Structured and easy scripts for increased code readability Peer reviews to check for code quality Use of reporting tools/libraries/dashboards Execution of Automated Tests in CI Figure 3 is a basic workflow that defines how a scalable automation process can work. In my experience, the very basic need to run a scalable automation script in the CI pipeline is met by using a trigger that would help set up the test dependencies within Docker and execute tests accordingly based on the need. Figure 3: Bird's eye view of automation process For example, a test pipeline may run a regression script, whereas another pipeline may run the API scripts. These cases can be handled from a single script that acts as the trigger to the test scripts. execute.sh: #!/bin/bash set -eu # Check if csv_reports, logs directory, html_reports, screenshots is present mkdir app/csv_reports app/logs mkdir app/html_reports/screenshots # Validate that if an argument is passed or not if [ $# -eq 0 ]; then echo "No option is passed as argument"; fi # Parse command line argument to run tests accordingly for i in "$@"; do case $i in --regression) pytest -p no:randomly app/test/ -m regression --browser firefox --headless true --html=app/html_reports/"$(date '+%F_%H:%M:%S')_regression".html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log break ;; --smoke) pytest app/test -m smoke break ;; --sanity) pytest app/test -m sanity --browser chrome --headless true --html=app/html_reports/ sanity_reports.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log break ;; --apitest) npm run apitest break ;; --debug) pytest app/test -m debug --browser chrome --headless true --html=app/html_reports/ report.html --log-file app/logs/"$(date '+%F_%H:%M:%S')".log break ;; *) echo "Option not available" ;; esac done test_exit_status=$? exit $test_exit_status Analysis of Test Reports By analyzing test reports, testing teams are able to determine whether additional testing is needed, if the scripts used can accurately identify errors, and how well the tested application(s) can withstand challenges. Reports can be represented either using static HTML or dynamic dashboard. Dashboards can help stakeholders in understanding trends in the test execution by comparing the current data with the past data of execution. For example, allure reporting creates a concise dashboard with the test outcomes and represents it using data collected from test execution. Conclusion Automated testing lifecycle is a curated process that helps testing applications meet specific goals within appropriate timelines. Furthermore, it is very important for the QAOps process to gel properly with the SDLC and rapid application development. When completed correctly, the six phases of the lifecycle will achieve better outcomes and delivery. Additional Reading: Cloud-Based Automated Testing Essentials Refcard by Justin Albano "Introduction to App Automation for Better Productivity and Scaling" by Soumyajit Basu This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report

IBM App Connect Enterprise (ACE) is a powerful and widely used integration tool. Developers create integration flows by defining an entry point that receives a message, then processing that message, and finishing by sending or placing the transformed message. Flows consist of a series of nodes and logical constructs. ACE is powerful and flexible — there are many nodes provided specifically to interact with the systems being integrated, however there are also nodes that can run a script or Java code. Because of this, ACE can do pretty much anything, and as such could be considered (although this is not its intended purpose) as an application runtime environment. An ACE flow is a deployable unit that is inherently stateless, although it can manage its own state. In a traditional server environment, many flows are deployed on an integration server and their execution can be managed and scaled using the workload management features. This makes ACE a natural fit for a Kubernetes environment. It is therefore tempting to consider an ACE flow as a microservice and deploy each flow to a cloud in that way. Whilst this is certainly possible, there are many considerations to ensure a successful deployment of ACE on Kubernetes. The Problem Deployment Overhead Deploying many microservices can result in increased resource overhead compared to traditional application server architectures. Each microservice requires its own operating system and instance of whatever runtime it is using. In ACE terms, this means creating many integration servers with one flow on each. However, an integration server itself requires resources. Consider an application with 100 flows deployed on a server with 4 CPUs and 8GB of RAM. Some of the flows are used heavily throughout the business day, but some are only called once a day for a batch job in the evening. During high usage the busy flows can consume most of the CPU. At peak times they will slow down a bit because they compete for CPU, but this is okay because the CPU can manage many threads. The flow instances handling the batch job will remain idle, consuming no CPU until they are needed when the busy flows are idle. In ACE 12.0.5 and above, the effective minimum CPU requirement is 0.1 CPU and 350MB of memory, so in an application with 100 flows that’s a minimum of 10 CPUs and 35GB of memory which is a significant increase in terms of both infrastructure and license cost. Not only that, but in this example every flow has the minimum allocated CPU at all times, which is nowhere near enough for the busy flows and far more than the once-per-day flows need. Logical vs. Operational Optimization It is possible to analyze the requirements of all your flows and group together flows based on release schedule, demand profile, common connections/assets and so on (see section Efficiency of Scaling Grouped Flows). These criteria would be purely operational, and this could be an entirely valid way of doing it. However, there would be no relationship between where a flow is deployed and its function. Pods and deployments would not have names that identified their functions. This may be possible in a highly automated DevOps setup, but this may not be ideal for most projects. On the other hand, deploying one flow per pod maps deployments perfectly to logical functions — your flow and its pod would have a name relating to its function. Generally modern apps rely on naming conventions and design related to logical function to reduce the need for documentation and make the project as easy and transparent as possible to work on. Software applications are created and maintained by humans after all, and humans need things to be clear and straightforward from a human perspective. But as we have seen this is not necessarily feasible in a large ACE project. The aim of this document is to demonstrate ways to link the logical and operational deployment patterns — in other words, to create operational optimization without losing logical design. ACE vs. Other Runtimes If microservices can be deployed in their own pods and scaled as required, what is the difference between ACE and other microservice runtimes such as Open Liberty or NodeJS? Start-up time and cost: Integrations deployed in BAR files need to be compiled before use. If uncompiled BAR files are deployed this adds a CPU spike on start-up. This can also be picked up by the horizontal pod autoscaler. License cost: Since ACE is proprietary software it adds license cost. Performance Planning Pods vs. Instances One of the key concepts of cloud-native microservice architecture is that multiple instances of a service can exist, and state should be stored outside the instance. This means that as many instances as are needed can be deployed. This can be determined at design time, and can be altered whilst running either as a manual operation or automatically according to load. In Kubernetes, a microservice is represented as a deployment, and this creates a given number of pods which are service instances. The number of pods can be scaled up or down. A Kubernetes cluster should have multiple worker nodes, to provide redundancy. Because it is a distributed system, there should also be an odd number of nodes, so that if one fails the other two still constitute a quorum. In practice, this means three worker nodes. This then dictates that deployments should have three pods — one on each node — to spread the load appropriately. It is possible to configure deployments to deploy more pods (via a horizontal pod autoscaler or HPA) to provide more processing power when demand is high. Pod Scaling vs. ACE Workload Management ACE includes native workload management features, one of which is the ability to create additional instances of a deployed message flow within the integration server in the pod as opposed to more pods. Additional instances correspond to additional processing threads. It is possible to start with zero additional instances (i.e., one instance) and set the integration server to create new instances on demand up to a maximum value. The additional instances will be destroyed when no longer needed, and their onward connections (e.g., to a DB) will be closed, however the RAM will not be released back to the OS. So whilst pod CPU usage will return to lower levels RAM will not. This may or may not be a problem depending on your constraints — RAM usage does not affect license cost, however the infrastructure still needs to be provisioned and paid for. Therefore, the additional instances property should be carefully considered in conjunction with Kubernetes pod scaling. Additional flow instances are much more resource-efficient than additional pods containing integration servers. The number of additional instances must be determined in conjunction with the number of CPUs allocated to the pod: If a flow is an asynchronous process, e.g., reading from a queue and writing to another, then it can be busy almost all the time assuming the processing done by the flow is CPU intensive. If that is the case then the total number of flow instances (bear in mind that zero additional instances means one total instance) should match the number of CPUs configured on the pod. However, if a flow is a synchronous process e.g., it calls a downstream HTTP service or a database, then the flow will be blocked waiting for that downstream service to complete, during which time the thread would be idle. If a flow spends 50% of its time idle waiting for a downstream response and 50% of its time processing the results, then two flow instances (i.e. one additional) can be configured in a pod that has one CPU. Of course, you could create two pods instead of two flow instances, but this is much less efficient because of the overhead of running second pod and an entire integration server inside it. For a synchronous flow, the following equation is a good starting point if you can measure the CPU utilization when running a single instance: If you are analyzing performance using timings printed out in debugging statements or the user trace features, or using the CP4I tracing tool, then you should know how long the total flow execution takes and how long is spent waiting for a downstream service call. If so, then the equation can be written as follows: If pods are configured like this, then the CPU of the pod should be well utilized, but not so much so that a single pod restarting or failing will cause the others to be overloaded. Of course, if a flow is configured to have two flow instances, then each new pod results in two more flow instances at a time. The benefit of using the HPA for scaling over is that when load decreases the additional pods will be destroyed, which will release resources. The disadvantage is that more resources are required for the extra pod overhead. In general, the workload management properties should be used for modest scaling, and the HPA used sparingly for highly elastic demand — for example, a bank that needs to deploy large numbers of pods at the end of each month for paydays, or for a retailer needing to handle a Black Friday sale. When using the HPA, it is important to realize that ACE integration servers can have high start-up CPU load if BAR files are not pre-compiled, or if flows are written with start-up tasks e.g., cache building, resource loading etc. The HPA may pick up this high CPU usage on start-up and start deploying more pods, which will then need to start up as well — and this could cause runaway resource consumption. To overcome this, configure a stabilization window on your HPA. Performance tuning is of course a complex subject, and a detailed description of ACE performance tuning is outside the scope of this document. Scheduled Scaling An alternative to using the HPA is to use a tool like KEDA to provide scaling based on events such as external messages, or according to a schedule. For example, an e-commerce organization could scale-up pods in advance of a product being released or a discount event. This has the benefit of being proactive rather than reactive, like the HPA. Grouping Flows Operationally With a typical application containing many hundreds of flows we know that we will need to group them together on the same integration servers and therefore pods. There are several operational and deployment/packaging criteria that can be used for this. Group integrations that have stable and similar requirements and performance. Group integrations with common technical dependencies. A popular use case is dependency on MQ. Group integrations where there are common cross dependencies, i.e., integrations completely dependent on the availability of other integration. Group integrations that have common integration patterns, e.g., sync vs. async. Synchronous calls between flows are much better done in the same integration server because they can use the callable flow pattern rather than HTTP interaction which would be much less efficient. Group integrations with common or similar scalability model is needed. Group integrations with common or similar resilience (high availability) model is needed. Group integrations with common or similar shared lifecycle. A common situation where this may occur is where integrations use shared data models, libraries, etc. Group integrations that require common Integration Server runtime components i.e., C++, JVM, NodeJS, etc. to leverage and optimize deployment using the new Dynamic Resource Loading feature. Logical Patterns for Grouping Flows Large enterprise applications depend on consistent design principles for manageability and quality. Choosing specific design patterns for implementation helps this greatly. Integration flows can be deployed anywhere and will still work, however this kind of deployment will make it hard to create logical groupings. Here we will look at patterns into which integration flows fit; if the physical deployments follow the logical architecture the flows will be easier to manage. Here are some examples of ideas that could help to group flows by business function and/or integration style, so that they will still fit into a microservice concept. Composite Service Consider a system where ACE is acting as a wrapper for downstream services. Each flow invokes a single operation on a downstream service. Related flows, for example, flows that invoke database stored procedures to create, update or delete customer records, can be grouped into a single Application and deployed on their own integration server. When considered this way, the deployed integration server becomes the customer management service. Workload management can be used statically (i.e., selecting "deploy additional instances at startup") to set the number of instances for each thread relative to the others, if some flows need to handle more traffic than others. Message Hub/Broker If your application has flows that place messages onto queues or Kafka topics, it might be worth considering these as related functions and grouping them all into a single Application deployed in a single pod, called a Message Hub or Broker. Business Function Grouping by business functional area into single services reduces scaling flexibility, which ultimately results in less efficiency — if one flow is overloaded and a new pod is scaled up, then new capacity for other flows may be created and not needed. But, if suitable CPU request and limits are configured, and workflow management, this may ultimately be more efficient in terms of absolute CPU use. Efficiency of Scaling Grouped Flows In an ideal microservice environment, each flow would be a microservice that could be scaled independently. However, due to the inherent overhead of integration server pods, this is not efficient as we have seen. If flows are grouped together though, then when scaling one flow you may end up with instances of other flows that you don’t need. This is also inefficient. But in some circumstances it may still be more efficient than multiple pods. Consider two flows, A and B each in their own pod. Each might initially be using only a negligible amount of CPU, but the minimum requirement for each pod is 0.1 so that is the CPU request. Replicated across three nodes that means 0.6 CPU request. Let’s assume that the flows are single threaded, so the limit for each pod should be 1 CPU. That means that the upper limit is 6 CPU. Now, if flow A becomes very busy that means a maximum of 3.3 CPU used by both flows — 3 from three instances of flow A and the original 0.3 from three instances of flow B. If it becomes even busier, you can scale deployment A up by one pod so there are four flow A pods, and this means the total CPU is now 4.3 CPU. If flows A and B are both deployed in the same pod, then when traffic is low the total request is only 0.3 CPU. When flow A becomes busy, this becomes 3 CPU. When it scales up to 4 pods, the total is 4 CPU. Flow B has been deployed twice but since it is not using significant CPU flow A still has the resources it needs. In this scenario, the extra overhead of deploying the two flows independently is greater than the cost of duplicating flow B unnecessarily. Node Level Licensing IBM licenses pods based on the specified CPU limits. However, if ACE pods are restricted to running on a single worker node IBM will not charge more than the CPUs available to that node. This means that pod affinity can be used to deploy ACE only on certain nodes. If no CPU limits are set, then any of the pods can use as much CPU resources as are available to the node, and the node’s operating system can schedule the threads as it sees fit in the same way that it does in a non-container environment. This can be a useful technique when doing a lift-and-shift migration from a non-container environment.

GitHub Actions has a large ecosystem of high-quality third-party actions, plus built-in support for executing build steps inside Docker containers. This means it's easy to run end-to-end tests as part of a workflow, often only requiring a single step to run testing tools with all the required dependencies. In this post, I show you how to run browser tests with Cypress and API tests with Postman as part of a GitHub Actions workflow. Getting Started GitHub Actions is a hosted service, so all you need to get started is a GitHub account. All other dependencies, like Software Development Kits (SDKs) or testing tools, are provided by the Docker images or GitHub Actions published by testing platforms. Running Browser Tests With Cypress Cypress is a browser automation tool that lets you interact with web pages in much the same way an end user would, for example by clicking on buttons and links, filling in forms, and scrolling the page. You can also verify the content of a page to ensure the correct results are displayed. The Cypress documentation provides an example first test which has been saved to the junit-cypress-test GitHub repo. The test is shown below: describe('My First Test', () => { it('Does not do much!', () => { expect(true).to.equal(true) }) }) This test is configured to generate a JUnit report file in the cypress.json file: { "reporter": "junit", "reporterOptions": { "mochaFile": "cypress/results/results.xml", "toConsole": true } } The workflow file below executes this test with the Cypress GitHub Action, saves the generated video file as an artifact, and processes the test results. You can find an example of this workflow in the junit-cypress-test repository: name: Cypress on: push: workflow_dispatch: jobs: build: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v1 - name: Cypress run uses: cypress-io/github-action@v2 - name: Save video uses: actions/upload-artifact@v2 with: name: sample_spec.js.mp4 path: cypress/videos/sample_spec.js.mp4 - name: Report uses: dorny/test-reporter@v1 if: always() with: name: Cypress Tests path: cypress/results/results.xml reporter: java-junit fail-on-error: true The official Cypress GitHub action is called to execute tests with the default options: - name: Cypress run uses: cypress-io/github-action@v2 Cypress generates a video file capturing the browser as the tests are run. You save the video file as an artifact to be downloaded and viewed after the workflow completes: - name: Save video uses: actions/upload-artifact@v2 with: name: sample_spec.js.mp4 path: cypress/videos/sample_spec.js.mp4 The test results are processed by the dorny/test-reporter action. Note that test reporter has the ability to process Mocha JSON files, and Cypress uses Mocha for reporting, so an arguably more idiomatic solution would be to have Cypress generate Mocha JSON reports. Unfortunately, there is a bug in Cypress that prevents the JSON reporter from saving results as a file. Generating JUnit report files is a useful workaround until this issue is resolved: - name: Report uses: dorny/test-reporter@v1 if: always() with: name: Cypress Tests path: cypress/results/results.xml reporter: java-junit fail-on-error: true Here are the results of the test: The video file artifact is listed in the Summary page: Not all testing platforms provide a GitHub action, in which case you can execute steps against a standard Docker image. This is demonstrated in the next section. Running API Tests With Newman Unlike Cypress, Postman does not provide an official GitHub action. However, you can use the postman/newman Docker image directly inside a workflow. You can find an example of the workflow in the junit-newman-test repository: name: Cypress on: push: workflow_dispatch: jobs: build: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v1 - name: Run Newman uses: docker://postman/newman:latest with: args: run GitHubTree.json --reporters cli,junit --reporter-junit-export results.xml - name: Report uses: dorny/test-reporter@v1 if: always() with: name: Cypress Tests path: results.xml reporter: java-junit fail-on-error: true The uses property for a step can either be the name of a published action, or can reference a Docker image directly. In this example, you run the postman/newman docker image, with the with.args parameter defining the command-line arguments: - name: Run Newman uses: docker://postman/newman:latest with: args: run GitHubTree.json --reporters cli,junit --reporter-junit-export results.xml The resulting JUnit report file is then processed by the dorny/test-reporter action: - name: Report uses: dorny/test-reporter@v1 if: always() with: name: Cypress Tests path: results.xml reporter: java-junit fail-on-error: true Here are the results of the test: Behind the scenes, GitHub Actions executes the supplied Docker image with a number of standard environment variables relating to the workflow and with volume mounts that allow the Docker container to persist changes (like the report file) on the main file system. The following is an example of the command to execute a step in a Docker image: /usr/bin/docker run --name postmannewmanlatest_fefcec --label f88420 --workdir /github/workspace --rm -e INPUT_ARGS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/junit-newman-test/junit-newman-test":"/github/workspace" postman/newman:latest run GitHubTree.json --reporters cli,junit --reporter-junit-export results.xml This is a complex command, but there are a few arguments that we're interested in. The -e arguments define environment variables for the container. You can see that dozens of workflow environment variables are exposed. The --workdir /github/workspace argument overrides the working directory of the Docker container, while the -v "/home/runner/work/junit-newman-test/junit-newman-test":"/github/workspace" argument mounts the workflow workspace to the /github/workspace directory inside the container. This has the effect of mounting the working directory inside the Docker container, which exposes the checked-out files, and allows any newly created files to persist once the container is shutdown: Because every major testing tool provides a supported Docker image, the process you used to run Newman can be used to run most other testing platforms. Conclusion GitHub Actions has enjoyed widespread adoption among developers, and many platforms provide supported actions for use in workflows. For those cases where there is no suitable action available, GitHub Actions provides an easy way to execute a standard Docker image as part of a workflow. In this post, you learned how to run the Cypress action to execute browser-based tests and how to run the Newman Docker image to execute API tests. Happy deployments!

In a Cloud Pak for Integration (CP4I) environment, the IBM App Connect operator does a good job monitoring the Integration Server pods and will restart any pods where the Integration Server fails. Either in the rare event that it has crashed or because it is no longer responsive to liveness checks. It is worth noting that by far the most common reason for an Integration Server pod to fail in one of these ways is because it has insufficient resources to process its workload however there are some cases where the user may wish to initiate a restart in response to some other external criteria. In this example, we will consider the case where user supplied Java code in a Java Compute Node has caused an OutOfMemory error. In this instance, the error would not ordinarily be fatal to the Integration Server; however, a user may wish to restart the Integration Server on receiving an error of this type. The same approach can be generalized to any error condition which can be detected in a file on the IS container internal file system. We also assume that the target environment is a Cloud Pak for Integration instance; however, the same approach can be applied to any IBM App Connect Enterprise Operator installation including on plain Kubernetes. Indeed, some of the elements we discuss in this article can be further generalized to traditional on-premise environments. Overall Approach The approach we will take here has the following steps: A background script will monitor a file or set of files looking for specific error text. When the text is matched, the script will kill the parent process of the IntegrationServer. The OpenShift system will notice that the pod has failed and start a new instance of this pod. The Monitor Script Let's take a look at the script we will be using: echo "Starting monitor script" nohup tail -n0 -F $1/log/*events* | awk '/OutOfMemoryError/ { system("echo \"Detected Error, restarting pod\"") system("kill 1") }' 2>$1/log/monitor.out 1> $1/log/monitor.err & echo "Script started" This script starts off by writing a message that it is about to start. This is useful as it allows us to also confirm in the IntegrationServer stdout that the monitoring script is actually installed. The next line initiates a background process. The nohup command along with the "&" at the end of this line of the script means that the next set of commands will be run as a background process and will not terminate when the parent shell terminates. This is important, as we need to return control to the Integration Server after the script is launched. It should also be noted that it is necessary to redirect the stdout and stderr of the background process to ensure that the script as a whole continues executing and returns control to the Integration Server. Without the redirection when the script completes, it attempts to close the stdout/stderr stream of the child process, but this can't be done because the command we are running (tail) will never actually complete. Now let's consider what we are actually doing inside the background process. Specifically, this inner portion of the 2nd line: tail -n0 -F $1/log/*events* | awk '/OutOfMemoryError/ { system("echo \"Detected Error, restarting pod\"") system("kill 1") }' In a CP4I environment, a copy of all BIP messages is written to the "Event Log" which is located in /home/aceuser/ace-server/log. These files have names of the form integration_server.test-oom.events.txt.X where X is either a number or empty. Each time the IS restarts it will "roll over" onto a new log file and similarly if one of these log files reaches its size limit a new one will be created. Once the maximum number of log files is reached the IS will start writing to the first log file. Since this means that the log files may not all exist at startup and over time, they may be rolled over we need to use the -F flag for tail in order to ensure that if the filedescriptor changes as a result of the log file rolling over or a new log file being created tail will still pickup changes. The other thing to note about the tail command is that it is set to -n0 so that it does not print any context on initial execution and will instead simply output all new lines added to the monitored file after the initial execution of the tail command. This is important because if the monitored files are either in the work dir (like the log files) or on a persistent volume then they may survive a pod restart. So, in order to prevent the same log lines being re-processed when a pod restarts we need to make sure only new lines are output by tail. The next part of the command is a simple awk script. This script contains a regex that matches the error text we want to use as a trigger condition. In this case, we want to capture any message that contains the string "OutOfMemoryError". If the line output by tail matches, then the awk script will write a message to the stdout of the background process (which will be redirected to the /home/aceuser/ace-server/log/monitor.out file) and then issues a kill command against pid 1. In a CP4I environment, pid 1 will always be the runaceserver process and killing this process will cause the OpenShift platform to recognize that the pod has failed and restart the pod. Deploying the Monitoring Script So now we have our monitoring script, we need to actually configure the Integration Server to execute it during IS startup. To do this, we can deploy it to the CP4I environment using a "generic" configuration. To do this, we first need to place the script in a zip file and then obtain the contents of the zip file as a base64 encoded string. For example: davicrig:run$ zip monitor.zip monitor.sh adding: monitor.sh (deflated 30%) davicrig:run$ base64 -w 0 monitor.zip UEsDBBQAAAAIAHli81aGdruxqwAAAPUAAAAKABwAbW9uaXRvci5zaFVUCQAD1sa3ZNbGt2R1eAsAAQToAwAABOgDAABdj70Og kAQhHueYnIx/hAV8QGo1M5Q2NoQWPUC3JK9RULUd9fgT2E1yWRmv52mKwLKL4xR/FZz0EzUujNqdlZZ4HOxjZrA8aVtoJmtsHA rLHavTlTxOQrpSk59iDuyrsQkSltNT3uqWfqtCEuEG3zvleqpGSBHsyGlXKkAhsQcEPJfcsPF0ZjZr1PaqkL8Mh4TYJ18sJ //ltwq4gR/Lolg/J00LMBwnwoTPAFQSwECHgMUAAAACAB5YvNWhna7sasAAAD1AAAACgAYAAAAAAABAAAA /4EAAAAAbW9uaXRvci5zaFVUBQAD1sa3ZHV4CwABBOgDAAAE6AMAAFBLBQYAAAAAAQABAFAAAADvAAAAAAA= The long base64 encoded string can be copied to the clipboard. Once the contents of the zip file has been copied, we need to create a new configuration from the IBM App Connect operator using the following settings: When this configuration is made available to the Integration Server, it will unzip the provided zip file into the /home/aceuser/generic directory: /home/aceuser/generic sh-4.4$ ls monitor.sh Customizing an Integration Server to Run the Monitoring Script So the next step is to actually instruct the Integration Server to run the script during startup. To do this, we can use the server.conf.yaml "StartupScripts" stanza. StartupScripts: FirstScript: command: /home/aceuser/generic/monitor.sh ${WORK_DIR} directErrorToCommandOutput: false includeCommandOutputInLogs: true stopServerOnError: true There are a few important things to note about the settings here. In the "command" property we list the fully qualified script name, but we also pass in the ${WORK_DIR} token. This is a special token that is dynamically replaced with the Integration Server's work directory at runtime. For a CP4I environment we could have used a fully qualified path however in traditional on premise deployments we need to use this token in preference to the MQSI_WORKPATH environment variable to cope with cases where a stand-alone Integration Server has its work directory located in a different location to MQSI_WORKPATH. It is also important to note that we must have directErrorToCommandOutput set to false. Setting this value to true prevents the script from exiting properly and causes the startup script to hang which means that the Integration Server is never passed back control and never properly starts. So, once we have our configured server.conf.yaml snippet we need to create a configuration to make this available to the IntegrationServer on the CP4I environment. We can do this by creating a Configuration of the "serverconf" type: Here the "Data" field in the form should contain the server.conf.yaml snippet we are changing encoded into base64. Creating a Test Flow We are almost ready to configure the Integration Server itself, but first of all, we need to create an application which will actually simulate our error condition so that we can demonstrate the monitor script in action. To do this, simply create the following flow in the App Connect Enterprise Toolkit: Here we have 2 HTTP Input nodes, one which will simply immediately issue a successful reply and a second which will execute a Java Compute Node. The Java Compute Node will deliberately throw an error in order to simulate a real failure: Java // ---------------------------------------------------------- // Add user code below if(true) { throw new OutOfMemoryError("oh no! out of memory"); } // End of user code // ---------------------------------------------------------- In order to make this available to the Integration Server, we must deploy this to a dashboard server and obtain the barfileURI as shown below: Making the Deployed Configurations Available to an Integration Server Now we are ready to configure an Integration Server to use the newly uploaded BAR file and the 2 custom configurations that we created earlier. This example assumes that we will be modifying an existing Integration Server however it is also possible to create a new Integration Server by following the same process. We make the deployed configurations available to the Integration Server by updating the "spec" section of the Integration Server yaml as shown below: When you hit the save button, the Operator will reconcile the changes and create a new pod with the Configurations deployed. If we examine the logs for the Integration Server, we can verify that the startup script has executed: 2023-07-19 13:48:39.418412: BIP1990I: Integration server 'test-oom' starting initialization; version '12.0.7.0' (64-bit) 2023-07-19 13:48:39.418960: BIP9560I: Script 'FirstScript' is about to run using command '/home/aceuser/generic/monitor.sh /home/aceuser/ace-server'. Starting monitor script Script started 2023-07-19 13:48:39.424804: BIP9565I: Script 'FirstScript' has run successfully. 2023-07-19 13:48:39.518768: BIP9905I: Initializing resource managers. Testing the Solution We can test our script by issuing a curl command against the test flow that we just deployed: sh-4.4$ curl -v http://localhost:7800/throwOOM * Trying ::1... * TCP_NODELAY set * connect to ::1 port 7800 failed: Connection refused * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 7800 (#0) > GET /throwOOM HTTP/1.1 > Host: localhost:7800 > User-Agent: curl/7.61.1 > Accept: */* > command terminated with non-zero exit code: exit status 137The terminal connection has closed. Here the terminal closes because as soon as the runaceserver process is killed OpenShift will terminate and restart the pod. The pod logs for the pod will show the following: 2023-07-19T14:25:00.122Z Signal received: terminated 2023-07-19T14:25:00.124Z Stopping metrics gathering 2023-07-19T14:25:00.125Z Stopping Integration Server 2023-07-19 14:25:00.127057: BIP1989I: Integration server is terminating due to a shutdown event. If we log back in to the terminal once the pod is restarted we can examine the previous entries in the events file and confirm that a message matching our error filter was received: sh-4.4$ cat integration_server.test-oom.events.txt.1 | grep Out 2023-07-19 14:25:00.114267Z: [Thread 202] (Msg 3/4) BIP4367E: The method 'evaluate' in Java node 'Java Compute' has thrown the following exception: java.lang.OutOfMemoryError: oh no! out of memory. 2023-07-19 14:25:00.114306Z: [Thread 202] (Msg 4/4) BIP4395E: Java exception: 'java.lang.OutOfMemoryError'; thrown from class name: 'ThrowOOM_JavaCompute', method name: 'evaluate', file: 'ThrowOOM_JavaCompute.java', line: '24' Extending to Other Actions Restarting the pod is not the only action we can take based on a trigger from a monitoring script. We can also execute arbitrary App Connect Enterprise Administration REST API commands. The following script used in place of the original example for instance will enable service trace when the trigger condition is met: echo "Starting monitor script" nohup tail -n0 -F $1/log/*events* | awk '/OutOfMemoryError/ { system("echo \"Detected Error, enabling trace\"") system("curl --unix-socket /home/aceuser/ace-server/config/IntegrationServer.uds -X POST http://localhost:7600/apiv2/start-service-trace") }' 2>$1/log/monitor.out 1> $1/log/monitor.err & echo "Script started" Note that here we are using the unix domain socket interface to the web admin API rather than the tcp interface. This is the same interface used by mqsi* commands and relies on operating system user security. It is possible to use the normal admin port running on port 7600 to make a standard REST request, however it is not possible to extract the basic auth credentials from the script itself since the script runs as aceuser and the credentials are owned by the root user (for security reasons). Therefore instead of passing the basic auth credential into the script where they would be available in plaintext to aceuser I have opted instead to use the unix domains socket. If instead of the original monitor.sh we deploy this new update copy, we can see that when the test flow is executed the pod logs will show that trace is enabled: 2023-07-19 21:08:33.037476: BIP2297I: Integration Server service trace has been enabled due to user-initiated action. Similarly, from within the pod we can confirm that trace files have been created: sh-4.4$ find ace-server/ -name *trace* ace-server/config/common/log/integration_server.test-oom.trace.0.txt sh-4.4$ ls -la ace-server/config/common/log/integration_server.test-oom.trace.0.txt -rw-r-----. 1 1000660000 1000660000 21852199 Jul 19 21:13 ace-server/config/common/log/integration_server.test-oom.trace.0.txt Conclusion In this article, we have seen how to deploy a startup script to an Integration Server running in a CP4I environment and use this to monitor the error logs in order to take corrective action to restart the pods or to run arbitrary admin commands using the REST interface. Using a monitor script provides a great deal of flexibility for reacting to specific conditions for both recovery and troubleshooting purposes.

In this article, we’ll explain how to use Ansible to build and deploy a Quarkus application. Quarkus is an exciting, lightweight Java development framework designed for cloud and Kubernetes deployments, and Red Hat Ansible Automation Platform is one of the most popular automation tools and a star product from Red Hat. Set Up Your Ansible Environment Before discussing how to automate a Quarkus application deployment using Ansible, we need to ensure the prerequisites are in place. First, you have to install Ansible on your development environment. On a Fedora or a Red Hat Enterprise Linux machine, this is achieved easily by utilizing the dnf package manager: Shell $ dnf install ansible-core The only other requirement is to install the Ansible collection dedicated to Quarkus: Shell $ ansible-galaxy collection install middleware_automation.quarkus This is all you need to prepare the Ansible control machine (the name given to the machine executing Ansible). Generally, the control node is used to set up other systems that are designated under the name targets. For the purpose of this tutorial, and for simplicity's sake, we are going to utilize the same system for both the control node and our (only) target. This will make it easier to reproduce the content of this article on a single development machine. Note that you don’t need to set up any kind of Java development environment, because the Ansible collection will take care of that. The Ansible collection dedicated to Quarkus is a community project, and it’s not supported by Red Hat. However, both Quarkus and Ansible are Red Hat products and thus fully supported. The Quarkus collection might be supported at some point in the future but is not at the time of the writing of this article. Inventory File Before we can execute Ansible, we need to provide the tool with an inventory of the targets. There are many ways to achieve that, but the simplest solution for a tutorial such as this one is to write up an inventory file of our own. As mentioned above, we are going to use the same host for both the controller and the target, so the inventory file has only one host. Here again, for simplicity's sake, this machine is going to be the localhost: Shell $ cat inventory [all] localhost ansible_connection=local Refer to the Ansible documentation for more information on Ansible inventory. Build and Deploy the App With Ansible For this demonstration, we are going to utilize one of the sample applications provided as part of the Quarkus quick starts project. We will use Ansible to build and deploy the Getting Started application. All we need to provide to Ansible is the application name, repository URL, and the destination folder, where to deploy the application on the target. Because of the directory structure of the Quarkus quick start, containing several projects, we'll also need to specify the directory containing the source code: Shell $ ansible-playbook -i inventory middleware_automation.quarkus.playbook \ -e app_name='optaplanner-quickstart' \ -e quarkus_app_source_folder='optaplanner-quickstart' \ -e quarkus_path_to_folder_to_deploy=/opt/optplanner \ -e quarkus_app_repo_url='https://github.com/quarkusio/quarkus-quickstarts.git' Below is the output of this command: PLAY [Build and deploy a Quarkus app using Ansible] **************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Build the Quarkus from https://github.com/quarkusio/quarkus-quickstarts.git.] *** TASK [middleware_automation.quarkus.quarkus : Ensure required parameters are provided.] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Define path to mvnw script.] ***** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure that builder host localhost has appropriate JDK installed: java-17-openjdk] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Delete previous workdir (if requested).] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure app workdir exists: /tmp/workdir] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Checkout the application source code.] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Build the App using Maven] ******* ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Display build application log] *** skipping: [localhost] TASK [Deploy Quarkus app on target.] ******************************************* TASK [middleware_automation.quarkus.quarkus : Ensure required parameters are provided.] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure requirements on target system are fullfilled.] *** included: /root/.ansible/collections/ansible_collections/middleware_automation/quarkus/roles/quarkus/tasks/deploy/prereqs.yml for localhost TASK [middleware_automation.quarkus.quarkus : Ensure required OpenJDK is installed on target.] *** skipping: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure Quarkus system group exists on target system] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure Quarkus user exists on target system.] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure deployement directory exits: /opt/optplanner.] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Set Quarkus app source dir (if not defined).] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Deploy application as a systemd service on target system.] *** included: /root/.ansible/collections/ansible_collections/middleware_automation/quarkus/roles/quarkus/tasks/deploy/service.yml for localhost TASK [middleware_automation.quarkus.quarkus : Deploy application from to target system] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Deploy Systemd configuration for Quarkus app] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Perform daemon-reload to ensure the changes are picked up] *** ok: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure Quarkus app service is running.] *** changed: [localhost] TASK [middleware_automation.quarkus.quarkus : Ensure firewalld configuration is appropriate (if requested).] *** skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=19 changed=8 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 As you can see, the Ansible collection for Quarkus does all the heavy lifting for us: its content takes care of checking out the source code from GitHub and builds the application. It also ensures the system used for this step has the required OpenJDK installed on the target machine. Once the application is successfully built, the collection takes care of the deployment. Here again, it checks that the appropriate OpenJDK is available on the target system. Then, it verifies that the required user and group exist on the target and if not, creates them. This is recommended mostly to be able to run the Quarkus application with a regular user, rather than with the root account. With those requirements in place, the jars produced during the build phase are copied over to the target, along with the required configuration for the application integration into systemd as a service. Any change to the systemd configuration requires reloading its daemon, which the collection ensures will happen whenever it is needed. With all of that in place, the collection starts the service itself. Validate the Execution Results Let’s take a minute to verify that all went well and that the service is indeed running: Shell # systemctl status optaplanner-quickstart.service ● optaplanner-quickstart.service - A Quarkus service named optaplanner-quickstart Loaded: loaded (/usr/lib/systemd/system/optaplanner-quickstart.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2023-04-26 09:40:13 UTC; 3h 19min ago Main PID: 934 (java) CGroup: /system.slice/optaplanner-quickstart.service └─934 /usr/bin/java -jar /opt/optplanner/quarkus-run.jar Apr 26 09:40:13 be44b3acb1f3 systemd[1]: Started A Quarkus service named optaplanner-quickstart. Apr 26 09:40:14 be44b3acb1f3 java[934]: __ ____ __ _____ ___ __ ____ ______ Apr 26 09:40:14 be44b3acb1f3 java[934]: --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ Apr 26 09:40:14 be44b3acb1f3 java[934]: -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \ Apr 26 09:40:14 be44b3acb1f3 java[934]: --\___\_\____/_/ |_/_/|_/_/|_|\____/___/ Apr 26 09:40:14 be44b3acb1f3 java[934]: 2023-04-26 09:40:14,843 INFO [io.quarkus] (main) optaplanner-quickstart 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.16.6.Final) started in 1.468s. Listening on: http://0.0.0.0:8080 Apr 26 09:40:14 be44b3acb1f3 java[934]: 2023-04-26 09:40:14,848 INFO [io.quarkus] (main) Profile prod activated. Apr 26 09:40:14 be44b3acb1f3 java[934]: 2023-04-26 09:40:14,848 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, hibernate-orm-panache, hibernate-orm-rest-data-panache, jdbc-h2, narayana-jta, optaplanner, optaplanner-jackson, resteasy-reactive, resteasy-reactive-jackson, resteasy-reactive-links, smallrye-context-propagation, vertx, webjars-locator] Having the service running is certainly good, but it does not guarantee by itself that the application is available. To double-check, we can simply confirm the accessibility of the application by connecting to it: PowerShell # curl -I http://localhost:8080/ HTTP/1.1 200 OK accept-ranges: bytes content-length: 8533 cache-control: public, immutable, max-age=86400 last-modified: Wed, 26 Apr 2023 10:00:18 GMT date: Wed, 26 Apr 2023 13:00:19 GMT Writing up a Playbook The default playbook provided with the Ansible collection for Quarkus is quite handy and allows you to bootstrap your automation with a single command. However, most likely, you’ll need to write your own playbook so you can add the automation required around the deployment of your Quarkus app. Here is the content of the playbook provided with the collection that you can simply use as a base for your own: YAML --- - name: "Build and deploy a Quarkus app using Ansible" hosts: all gather_facts: false vars: quarkus_app_repo_url: 'https://github.com/quarkusio/quarkus-quickstarts.git' app_name: optaplanner-quickstart' quarkus_app_source_folder: 'optaplanner-quickstart' quarkus_path_to_folder_to_deploy: '/opt/optaplanner' pre_tasks: - name: "Build the Quarkus from {{ quarkus_app_repo_url }." ansible.builtin.include_role: name: quarkus tasks_from: build.yml tasks: - name: "Deploy Quarkus app on target." ansible.builtin.include_role: name: quarkus tasks_from: deploy.yml To run this playbook, you again use the ansible-playbook command, but providing the path to the playbook: Shell $ ansible-playbook -i inventory playbook.yml Conclusion Thanks to the Ansible collection for Quarkus, the work needed to automate the deployment of a Quarkus application is minimal. The collection takes care of most of the heavy lifting and allows its user to focus on the automation needs specific to their application and business needs.

Kubernetes, the de-facto standard for container orchestration, supports two deployment options: imperative and declarative. Because they are more conducive to automation, declarative deployments are typically considered better than imperative. A declarative paradigm involves: Writing YAML manifest files that describe the desired state of your Kubernetes cluster Applying the manifest files Letting the Kubernetes controllers work out their magic The issue with the declarative approach is that YAML manifest files are static. What if you want to deploy the same app in two different environments (for example, “staging” and “production”) with some slight changes (for instance, allocating resources to production)? Having two YAML files is inefficient. A single parameterized YAML file is ideal for this scenario. Helm is a package manager for Kubernetes that solves this problem. It supports manifest templating and enables parameterization of otherwise static YAML configurations. A set of templated manifest files. A Helm chart is a package consisting of templated manifest files and metadata that can be deployed into a Kubernetes cluster. A Helm chart takes input variables and uses them to process templated YAML files to produce a manifest that can be sent to the Kubernetes API. Before deploying a Helm chart into your Kubernetes cluster, it’s wise to understand how it will behave. Helm dry run — specifically the helm install --dry-run command — addresses this use case and enables a preview of what a Helm chart will do without deploying resources on a cluster. Helm dry run also streamlines troubleshooting and testing Helm charts. This article will take a closer look at Helm dry run concepts, including related Helm commands and how to use Helm dry run to troubleshoot templates. Summary of Key Helm Dry Run Concepts The table below summarizes the Helm dry run concepts we will explore in this article. Concept Description Helm lint Performs some static analysis to check a Helm chart for potential bugs, suspicious constructs, and deviations from best practices. Helm template Generates the output manifest and prints it out. Only checks for YAML syntax and not whether the generated YAML is a valid Kubernetes manifest. Helm install --dry-run Generates the output manifest and sends it to the Kubernetes API for verification. Helm Dry Run and Related Helm Commands The three Helm commands we will explore in this article are: helm template helm lint helm install --dry-run The helm template command renders the template but does not check the validity of the generated YAML files beyond a simple YAML syntax check. It will stop generating the output when it encounters invalid YAML. Use the --debug flag to force the helm template command to display full output, including invalid YAML. The helm template command has the advantage of not needing a running cluster. Use cases for helm template include: To test the output of a Helm chart you are developing To see how certain values will change the output manifest file The helm lint command runs a basic static analysis that checks a Helm chart for potential bugs, suspicious constructs, and best practices deviations. This command is only useful when writing a Helm chart. The helm install --dry-run command requires a running Kubernetes cluster and will test the manifest against that specific cluster. This is useful because a Helm chart may be compatible with one cluster but not another. Potential differences across clusters include: Kubernetes version Custom resources Presence of required node types How to Run “helm template” Now let’s look at how to run the helm template command. First, let’s create a simple Helm chart: $ helm create mychart This will create a simple chart deploying NGINX. Now let’s see what the helm template command shows: $ helm template mychart mychart --- # Source: mychart/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: mychart labels: helm.sh/chart: mychart-0.1.0 app.kubernetes.io/name: mychart app.kubernetes.io/instance: mychart app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm <--- snip ---> Next, let’s introduce an error in one of the YAML manifest files. Edit the “mychart/templates/serviceaccount.yaml” file and add some invalid YAML like this: {{- if .Values.serviceAccount.create -} apiVersion: v1 kind: ServiceAccount invalid: yaml metadata: name: {{ include "mychart.serviceAccountName" . } labels: <--- snip ---> Let’s see what the helm template command does now: $ helm template mychart mychart Error: YAML parse error on mychart/templates/serviceaccount.yaml: error converting YAML to JSON: yaml: line 3: mapping values are not allowed in this context Use --debug flag to render out invalid YAML As we can see, it failed because the YAML is invalid. We can view the invalid output with the --debug flag: $ helm template mychart mychart --debug install.go:178: [debug] Original chart version: "" install.go:195: [debug] CHART PATH: /home/muaddib/Work/Square/tmp/mychart <--- snip ---> --- # Source: mychart/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount invalid: yaml metadata: name: mychart labels: helm.sh/chart: mychart-0.1.0 app.kubernetes.io/name: mychart app.kubernetes.io/instance: mychart app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm <--- snip ---> Error: YAML parse error on mychart/templates/serviceaccount.yaml: error converting YAML to JSON: yaml: line 3: mapping values are not allowed in this context helm.go:84: [debug] error converting YAML to JSON: yaml: line 3: mapping values are not allowed in this context YAML parse error on mychart/templates/serviceaccount.yaml helm.sh/helm/v3/pkg/releaseutil.(*manifestFile).sort helm.sh/helm/v3/pkg/releaseutil/manifest_sorter.go:146 <--- snip ---> As you can see, the errors eventually cause Helm to crash. But at least you should have gotten enough output to troubleshoot your problem. Using helm template --debug is mostly useful when writing a Helm chart and you need to understand whether your Helm chart is doing what you want. Lastly, let’s explore the helm template command’s main limitation: it generates output even if the result is not a valid Kubernetes manifest. To demonstrate, edit the previous file, undo the error we introduced, and modify it like this: {{- if .Values.serviceAccount.create -} apiVersion: v1 kind: ServiceAccountInvalid metadata: name: {{ include "mychart.serviceAccountName" . } labels: There is no kind “ServiceAccountInvalid” in Kubernetes, so let’s see what the helm template command will do: $ helm template mychart mychart <--- snip ---> --- # Source: mychart/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccountInvalid metadata: name: mychart labels: helm.sh/chart: mychart-0.1.0 app.kubernetes.io/name: mychart app.kubernetes.io/instance: mychart app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm <--- snip ---> As you can see, helm template happily generates the output, even though it is not a valid Kubernetes manifest file. How to Run “helm lint” The helm lint command tests whether a generated manifest can be deployed on a specific Kubernetes cluster. This helps address issues such as helm template generating invalid manifest files and provides static analysis during chart creation. The helm lint command is run like this: $ helm lint mychart ==> Linting mychart [INFO] Chart.yaml: icon is recommended 1 chart(s) linted, 0 chart(s) failed Helm will flag potential issues and make some recommendations related to best practices. How to Use Helm Dry Run to Validate Helm Charts In this tutorial, we’ll use the helm install --dry-run command to validate a Helm chart without actually deploying it on a cluster. To keep things simple for this tutorial, we’ll run a local minikube cluster, but you can use any compatible cluster deployment to follow along. $ minikube start minikube v1.25.2 on Ubuntu 22.04 Using the virtualbox driver based on user configuration Starting control plane node minikube in cluster minikube Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... ▪ kubelet.housekeeping-interval=5m ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 Enabled addons: default-storageclass, storage-provisioner Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default If we run the helm install --dry-run command with the flawed Helm chart we used in the previous section, here is what happens: $ helm install mychart mychart --dry-run Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceAccountInvalid" in version "v1" As we can see, the helm install --dry-run command connects to the Kubernetes API and sends the resulting manifest file for verification, which fails as expected. Now let’s edit the “mychart/templates/serviceaccount.yaml” again and fix the error we introduced. After the edits, the command will succeed: $ helm install mychart mychart --dry-run NAME: mychart LAST DEPLOYED: Sat Jul 22 09:57:03 2023 NAMESPACE: default STATUS: pending-install REVISION: 1 HOOKS: --- # Source: mychart/templates/tests/test-connection.yaml apiVersion: v1 kind: Pod metadata: <--- snip ---> Finally, to demonstrate that the command connects to the Kubernetes API, let’s delete the minikube cluster and rerun the command: $ minikube delete Deleting "minikube" in virtualbox ... Removed all traces of the "minikube" cluster. $ helm install mychart mychart --dry-run Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused Conclusion The helm template command generates a given Helm chart's output manifest and simulates output for input variables. It checks the YAML syntax of the generated output but does not check whether the output is a valid Kubernetes manifest. It is useful both when writing a Helm chart and before deploying a Helm chart into an existing cluster.

To maintain the rapid pace of modern software development, efficiency and productivity are paramount. Build automation plays a crucial role in streamlining the software development lifecycle by automating repetitive tasks and ensuring consistent and reliable builds. With the help of dedicated build automation software tools, development teams can enhance collaboration, reduce errors, and accelerate the delivery of high-quality software. This article explores some popular software tools used for build automation, their key features, and how they contribute to optimizing the development process. Jenkins Jenkins is an open-source, Java-based automation server that provides a flexible and extensible platform for building, testing, and deploying software. With its vast plugin ecosystem, Jenkins supports a wide range of programming languages, build systems, and version control systems. Its key features include continuous integration, continuous delivery, and distributed build capabilities. Jenkins allows developers to define and automate build pipelines, schedule builds, run tests, and generate reports. It also integrates with popular development tools and provides robust security and access control mechanisms. Jenkins' extensive community support and active development make it a go-to choice for many development teams seeking a reliable and customizable build automation solution. Gradle Gradle is a powerful build automation tool that combines the flexibility of Apache Ant with the dependency management of Apache Maven. It uses Groovy or Kotlin as a scripting language and offers a declarative build configuration. Gradle supports incremental builds, parallel execution, and dependency resolution, making it efficient for large-scale projects. It seamlessly integrates with various IDEs, build systems, and version control systems. Gradle's build scripts are highly expressive, allowing developers to define complex build logic and manage dependencies with ease. With its plugin system, Gradle can be extended to handle specific build requirements. Its performance and versatility make it an attractive choice for projects ranging from small applications to enterprise-level software systems. Apache Maven Apache Maven is a widely adopted build automation tool known for its dependency management capabilities. Maven uses XML-based project configuration files to define builds, manage dependencies, and automate various project tasks. It follows a convention-over-configuration approach, reducing the need for manual configuration. Maven supports a standardized project structure and provides a rich set of plugins for building, testing, and packaging software. It integrates seamlessly with popular IDEs and version control systems. Maven's extensive repository of dependencies and its ability to resolve transitive dependencies make it an ideal choice for projects with complex dependency requirements. With its focus on project lifecycle management and dependency-driven builds, Maven simplifies the build process and helps maintain consistency across projects. Microsoft MSBuild MSBuild is a build platform developed by Microsoft and primarily used for building .NET applications. It is an XML-based build system that provides a flexible and extensible framework for defining build processes. MSBuild supports parallel builds, incremental builds, and project file transformations. It integrates with Microsoft Visual Studio and other development tools, enabling a seamless development experience. MSBuild's integration with the .NET ecosystem makes it well-suited for building .NET applications, libraries, and solutions. Its extensive logging capabilities and support for custom tasks and targets allow developers to tailor the build process to their specific requirements. Apache Ant Apache Ant is a popular Java-based build automation tool that uses XML-based configuration files. It provides a platform-independent way to automate build processes, making it suitable for Java projects. Ant's strength lies in its simplicity and flexibility. It offers a rich set of predefined tasks for compiling, testing, packaging, and deploying software. Ant can also execute custom scripts and tasks, allowing developers to incorporate specific build logic. While Ant lacks some advanced features found in other build automation tools, its simplicity and ease of use make it a popular choice for small to medium-sized projects. Make Make is a classic build automation tool that has been around for decades. It uses a simple syntax to define build rules and dependencies, making it suitable for small-scale projects. Make is primarily used in Unix-like environments and supports parallel builds, incremental builds, and dependency tracking. Its build scripts are written in makefile format, which can be easily customized and extended. Make can be integrated with various compilers, linkers, and other development tools, enabling a streamlined build process. While Make is not as feature-rich as some of the other build automation tools, it remains a reliable and efficient choice for many developers. Bamboo Bamboo, developed by Atlassian, is a commercial build automation and continuous integration server. It offers a comprehensive set of features for building, testing, and deploying software. Bamboo supports parallel and distributed builds, allowing teams to scale their build processes efficiently. It integrates with popular version control systems and provides real-time feedback on build status and test results. Bamboo's user-friendly interface and intuitive configuration make it a suitable choice for both small and large development teams. Additionally, Bamboo offers seamless integration with other Atlassian products, such as Jira and Bitbucket, creating a unified and streamlined development environment. CircleCI The build automation and continuous integration platform CircleCI is hosted in the cloud. It gives programmers the ability to scale-up and effectively automate the build, test, and deployment processes. The fact that CircleCI supports a variety of programming languages, build systems, and cloud platforms enables teams to use their preferred technologies. Developers can define build pipelines with ease using its user-friendly configuration, guaranteeing quick feedback and quick iteration cycles. With the highly adaptable environment that CircleCI offers, teams can customize their build procedures to meet particular needs. Because of its cloud-based infrastructure, managing the infrastructure is easier and there is less administrative work involved in maintaining dedicated build servers. Conclusion Modern software development methodologies require an effective build automation system. The tools covered in this article, such as Jenkins, Gradle, Apache Maven, and Microsoft MSBuild, provide reliable options for streamlining collaboration, automating the build process, and managing dependencies. Despite the fact that the approaches and target domains of these tools vary, they all help to shorten the development lifecycle, lower errors, and increase productivity. Project requirements, language preferences, and integration are some of the variables that affect which build automation tool is selected. The optimization of the software development process and timely delivery of high-quality software depend on effective build automation. Developers can concentrate on more valuable tasks like coding and testing by using build automation tools to automate repetitive tasks. For build automation, some well-known software tools include Jenkins, Gradle, Apache Maven, MSBuild, Apache Ant, and Make. Each tool has distinctive advantages and disadvantages, and the selection of a tool is based on the particular requirements of the project. With their advanced features, extensive plugin ecosystems, and robust community support, these tools have revolutionized software development, allowing teams to collaborate more effectively and deliver high-quality software more efficiently.

Kubernetes enables orchestration for deploying, scaling, and managing containerized applications. However, it can be challenging to manage Kubernetes clusters effectively without proper deployment practices. Kubernetes operates on a distributed architecture involving multiple interconnected components such as the control plane, worker nodes, networking, storage, and more. Configuring and managing this infrastructure can be complex, especially for organizations with limited experience managing large-scale systems. Similarly, there are many other challenges to Kubernetes deployment, which you can solve by using best practices like horizontal pod autoscaler (HPA), implementing security policies, and more. In this blog post, we will explore proven Kubernetes deployment practices that will help you optimize the performance and security of your Kubernetes deployments. But first, let’s understand the underlying architecture for Kubernetes deployment. Understanding the Kubernetes Architecture The Kubernetes architecture involves several key components. The control plane includes the API server, scheduler, and controller-manager, which handle cluster management. Nodes host pods and are managed by Kubelet, while etcd is the distributed key-value store for cluster data. The API server acts as the single source of truth for cluster state and management. Lastly, kube-proxy enables communication between pods and services, ensuring seamless connectivity within the Kubernetes environment. Advantages of Using Kubernetes for Deployment Kubernetes allows for efficient resource utilization through intelligent scheduling and horizontal scaling, providing scalability and automatic workload distribution. It also simplifies the management of complex microservices architectures by supporting self-healing capabilities that monitor and replace unhealthy pods. Organizations can use Kubernetes' support for rolling updates and version control, making application deployment seamless. By using Kubernetes for deployment, organizations can streamline their application management process. Key Factors To Consider Before Deploying Kubernetes When deploying Kubernetes, it's essential to consider several key factors. You need to evaluate the resource requirements of your workloads to determine the appropriate cluster size. This will ensure that you have enough resources to handle the workload efficiently. It is also crucial to define resource limits and requests to ensure fair allocation of resources among different workloads. Consider network connectivity and firewall requirements for inter-pod communication. Planning for storage requirements and exploring the different storage options that Kubernetes supports is also essential. Understanding the impact of the Kubernetes deployment on existing infrastructure and processes is essential for a smooth transition. Now that we have discussed the benefits of effective Kubernetes deployment and critical factors to consider, let’s discuss some of the best practices. 1. Best Practices for Using Kubernetes Namespaces When using Kubernetes namespaces, it is important to separate different environments or teams within a cluster logically. By doing so, you can effectively manage resource consumption by defining resource quotas for each namespace. Implement role-based access control (RBAC) to enforce access permissions within namespaces. Additionally, apply network policies to restrict communication between pods in different namespaces. Regularly reviewing and cleaning up unused namespaces optimizes cluster resources. By following these best practices, you can ensure efficient and secure namespace management in Kubernetes deployments. 2. Kubernetes Deployment Security Practices To ensure the security of your Kubernetes deployment, there are several best practices you should follow. Enable Role-Based Access Control (RBAC) to control access and permissions within the cluster. Implement network policies to restrict communication and enforce security boundaries. Regularly scan for vulnerabilities and apply patches to Kubernetes components. Enable audit logging to track and monitor cluster activity. Follow security best practices for container images and only use trusted sources. By implementing these practices, you can enhance the security of your Kubernetes deployment. Setting up Role-Based Access Control (RBAC) In Kubernetes To ensure fine-grained access permissions in Kubernetes, it is crucial to create custom roles and role bindings. You can effectively manage access for applications running within pods by utilizing service accounts. Implementing role inheritance simplifies RBAC management across multiple namespaces. Regularly reviewing and updating RBAC policies is essential to align with evolving security requirements. Following RBAC's best practices, such as the least privilege principle, minimizes security risks. Emphasizing these practices enables secure configuration and automation of Kubernetes deployments. 3. Best Practices for Securing Kubernetes API Server To ensure the security of your Kubernetes API server, it is essential to implement several best practices. Enable access control using RBAC to ensure only authorized users can access the API server. Implement network policies to restrict access to the server, preventing unauthorized access. Regularly update and patch the API server to avoid any vulnerabilities. Enable audit logs to monitor and track activity on the API server. Lastly, use role-based access control (RBAC) to manage user permissions effectively. Implementing Kubernetes Network Policies for Security To enhance security in Kubernetes deployments, implementing network policies is crucial. These policies allow you to control inbound and outbound traffic between pods, ensuring only authorized communication. Network segmentation with different namespaces adds an extra layer of security by isolating resources. Applying firewall rules further restricts communication, preventing unauthorized access. You can utilize network plugins like Calico or Cilium to manage network policies effectively, which provide advanced policy management capabilities. Monitoring network traffic and implementing IP whitelisting/blacklisting adds additional security against potential threats. 4. Scaling Kubernetes Deployments Implementing automatic scaling is a proven Kubernetes deployment best practice. You can optimize resource utilization using the Kubernetes horizontal pod autoscaler (HPA). It allows you to scale up or down based on CPU metrics, ensuring efficient allocation of resources. Another helpful tool is kube-state-metrics, which helps monitor the status of your Kubernetes deployments. Additionally, implementing cluster autoscaler enables automatically adjusting the number of nodes in your Kubernetes cluster. Continuously monitoring resource consumption and adjusting resource requests and limits is essential for smooth scaling. Automatic Scaling With Kubernetes Horizontal Pod Autoscaler (HPA) Configure the Kubernetes horizontal pod autoscaler (HPA) to automatically scale the number of pods based on CPU or custom metrics. Set the target CPU utilization for HPA to trigger scaling and enable the metrics server to monitor CPU utilization accurately. HPA can also be used with custom metrics to scale based on application-specific requirements. It's essential to monitor HPA events and adjust the HPA settings to ensure optimal performance and resource utilization. 5. Optimizing Resource Utilization With Kubernetes Resource Requests and Limits To optimize resource utilization in Kubernetes, it's essential to set resource requests specifying the minimum CPU and memory requirements for a pod. Additionally, resource limits should be used to prevent pods from exceeding allocated resources. Monitoring resource utilization through metrics like CPU and memory allows for adjustments to resource requests and limits based on observed consumption. Furthermore, optimizing container images helps reduce resource usage and improves overall performance. Implementing these practices can effectively optimize resource utilization in your Kubernetes deployments. 6. Monitoring and Logging Kubernetes Deployments Monitoring and logging Kubernetes deployments is vital for smooth operation. Prometheus and Grafana provide real-time metrics and alerts for critical events. The ELK stack centralizes logging, making troubleshooting and identifying bottlenecks easier. Custom metrics exporters monitor application-specific metrics. Optimize performance and troubleshoot effectively with monitoring and logging. Monitoring Kubernetes Clusters With Prometheus and Grafana Configure Prometheus to collect metrics from the Kubernetes API server and Kubelet. Utilize Grafana dashboards to visualize Prometheus metrics for comprehensive monitoring. Establish alerting rules in Prometheus to receive notifications for critical events. Monitor cluster components such as etcd, kube-proxy, and kube-dns using Prometheus exporters. Customize Grafana dashboards to track specific resource and application metrics. Conclusion To achieve successful Kubernetes deployments, follow best practices. Understanding architecture and benefits, considering resource requirements and security, setting up clusters and configuring networking, Implementing role-based access control and network policies, scaling efficiently, monitoring and logging for health, and troubleshooting common issues are some of the best practices to follow. However, which one to use depends on specific project requirements.

Git is a distributed revision control system. We learned in Understanding Git - DZone that Git stores different objects - commits, blobs, trees, and tags, in its repository, i.e., inside the .git folder. The repository is just one of the four areas that Git uses to store objects. In this article, we'll explain the four areas in Git, we'll delve deeper into each of these areas, uncovering their significance in facilitating tracking changes made to files, maintaining a history of revisions, and collaboration among developers. Understanding these areas empowers you to harness Git's capabilities to the fullest. The Four Areas Git stores objects in four areas illustrated below. These four areas represent the flow of changes in a typical Git workflow. Working Area The working area is where you edit, create, delete, and modify files as you work on your project. It represents the current state of the project and contains all the files and directories that make up your codebase. It is important to remember that the modifications made in the working area are temporary unless changes are committed. Repository The repository area is where all the project's history, metadata, and versioned files are stored. In Git, the repository is represented by the .git folder, which is typically located at the root of the project directory. The Git objects, which are immutable, are stored in the objects subdirectory. .git/objects blob tree commit tag In order to gain a deeper understanding of Git, we should be able to answer the following questions when issuing a Git command: How does this command move information across the four areas? How does this command change the Repository area? We'll explain these by taking illustrative Git commands. Let's review how Git maintains a project's history. Project History The Git objects linked together represent a project's history. Each commit is a snapshot of the working area at a certain point in time. Branches are entry points to history. A branch refers to a commit, HEAD points to the current branch. Pictorially, this is represented as shown below. This brings us to the third area in Git - the Index. Index The Index, also known as the Staging Area, is an intermediate step between the Working Area and the Repository. It helps in preparing and organizing changes before they are committed to the repository. Let's see a basic Git workflow that touches the three areas that we've encountered so far - the working area, the index, and the repository. Basic Workflow Files in the working directory can be in different states, including untracked, modified, or staged for commit. We use the git add command to move changes from the working directory to the staging area. The git commit command saves the changes in the Git repository by creating Git objects, e.g., commit. This is illustrated below. To see the differences in code made between these areas, we can use git diff the command as shown below. Shell $ echo "DS 8000" >> storage_insights.txt $ git diff diff --git a/storage_insights.txt b/storage_insights.txt index 972d3ae..210df65 100644 --- a/storage_insights.txt +++ b/storage_insights.txt @@ -2,3 +2,4 @@ Flash 9000 Storwize XIV SVC +DS 8000 $ git add storage_insights.txt $ git diff --cached diff --git a/storage_insights.txt b/storage_insights.txt index 972d3ae..210df65 100644 --- a/storage_insights.txt +++ b/storage_insights.txt @@ -2,3 +2,4 @@ Flash 9000 Storwize XIV SVC +DS 8000 The git checkout command that is used to move to a specific branch, copies changes from the repository area to both the working area and the index. Remove File The git rm command is used to remove file(s) from both the working directory and the index to ensure that Git is aware of their removal, so the changes can be committed. The --cached option removes the file(s) from the index but leaves them in the working directory, effectively untracking them without deleting them locally. Rename File To rename a file we can use the git mv command. This moves the file from both the working area and the index. It does not touch the repository. The git reset Command The git reset command in Git is used to move the HEAD pointer and branch references to a specific commit, effectively rewinding or resetting the state of the repository to a previous point in its history. The options are: Soft Reset (--soft): A soft reset moves the HEAD and branch reference to a different commit while keeping the changes in the staging area. It does not move any data between areas. Mixed Reset (Default Behavior, --mixed): A mixed reset moves the HEAD and branch reference while also unstaging the changes. The changes remain in your working directory. Hard Reset (--hard): A hard reset moves the HEAD and branch reference and discards all changes in both the staging area and the working directory. It effectively removes commits and changes. Thus, a reset moves the current branch, and optionally copies data from the repository area to the other areas as illustrated above. Now that we've covered the three areas and illustrated by way of examples how Git moves data between these areas, it is time to introduce the fourth area - the Stash. Stash (Clipboard) git stash is a handy Git command that allows you to temporarily save and stash changes in your working directory without committing them. This is useful when you need to switch to a different branch, work on something else, or pull changes from a remote repository while preserving your current changes. The stashed changes can later be reapplied or discarded as needed. Shell git stash --include-untracked This command moves all data from the working area and index to the stash and checks out the current commit. Working With Paths A commit typically includes changes from multiple files. We've worked with commits so far in our journey. It is possible to operate at a more granular level than a commit, e.g., a file. The following examples illustrate how to work with the individual file rather than commits. To restore a file from the repository to the index, we use git reset command at the file level. Shell $ git reset HEAD storage_insights.txt Unstaged changes after reset: M storage_insights.txt This command moves the file storage_insights.txt from the repository to the index. At the file level, the option --hard is not supported. To restore a file from the repository to both the working area and the index, we use git checkout command at the file level. Shell $ git checkout HEAD storage_insights.txt Updated 1 path from 43134cc Parts of a File Git can operate on things that are smaller than a file. The --patch option in Git refers to an interactive mode that allows you to selectively stage changes within individual files or even specific lines of code, giving you fine-grained control over what gets committed. It's commonly used with commands like git add git reset git checkout git stash We'll illustrate how to use this option with git add a command. We've two local changes in the file, we want to stage only one of the changes, called hunk, and not stage the other hunk. Shell $ git add --patch storage_insights.txt diff --git a/storage_insights.txt b/storage_insights.txt index 972d3ae..16a4557 100644 --- a/storage_insights.txt +++ b/storage_insights.txt @@ -1,4 +1,6 @@ Flash 9000 +DS 8000 Storwize XIV SVC +Spectrum Scale (1/1) Stage this hunk [y,n,q,a,d,s,e,?]? s Split into 2 hunks. @@ -1,4 +1,5 @@ Flash 9000 +DS 8000 Storwize XIV SVC (1/2) Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? y @@ -2,3 +3,4 @@ Storwize XIV SVC +Spectrum Scale (2/2) Stage this hunk [y,n,q,a,d,K,g,/,e,?]? n $ git diff --cached diff --git a/storage_insights.txt b/storage_insights.txt index 972d3ae..30280ea 100644 --- a/storage_insights.txt +++ b/storage_insights.txt @@ -1,4 +1,5 @@ Flash 9000 +DS 8000 Storwize XIV SVC The option --patch allows us to process changes, not on a file-by-file basis, but on a hunk-by-hunk basis. Switch and Restore The switch and restore commands are used to perform specific operations related to branches and file management. They help to switch between branches and restore files to specific states. The following command will move to a different branch. Shell git switch <branch-name> git restore will allow us to restore files in the working directory to a specified state. It's used to undo changes, either by discarding modifications made to files or by reverting them to a previous commit. This will replace the modified file with the committed version. Shell git restore --source=HEAD storage_insights.txt This command moves the changes from the staging area back to the working directory, effectively uncommitting them. Shell git restore --staged storage_insights.txt Summary It is crucial to understand the four areas that Git uses to move data around in order to implement a revision control system. In this article, we reviewed the four areas and illustrated data movements by taking examples of different Git commands. For every Git command that we use, if we can explain the data movements through these areas, it will help us with a deeper understanding of Git workflows.

Although relatively new to the world of continuous integration (CI), GitHub’s adding of Actions has seen its strong community build useful tasks that plug right into your repository. Actions let you run non-standard tasks to help you test, build, and push your work to your deployment tools. In no particular order, here are 10 of our favorites, plus how to install them. 1. Test Reporter Showing all your test results in GitHub, the test reporter action helps keep the important parts of your code and testing processes in one place. Providing the results in XML or JSON formats as part of a ‘check run’, this action tells you where your code failed with useful stats. Test reporter already supports most of the popular testing tools for the likes of .NET, JavaScript, and more. Plus, you can add more by raising an issue or contributing yourself. Supported frameworks: .NET: xUnit, NUnit, and MSTest Dart: test Flutter: test Java: JUnit JavaScript: JEST and Mocha 2. Build and Push Docker Images Doing what the title says, the build and push Docker images action lets you build and push Docker images. Using Buildx and Moby BuildKit features, you can create multi-platform builds, test images, customize inputs and outputs, and more. Check out the action’s page for the full list of features, including advanced use and how to customize it. 3. Setup PHP The Setup PHP action allows you to set up PHP extensions and .ini files for application testing on all major operating systems. It’s also compatible with tools like GitHub’s composer, PHP-config, Symfony, and more. See the marketplace page for the full list of compatible tools. 4. GitTools Actions The GitTools Action allows you to use both GitVersion and GitReleaseManager in your pipeline. GitVersion helps solve common versioning problems with semantic versioning (also known as ‘Semver’), for consistency across your projects. GitVersion helps avoid duplication, saves rebuilding time, and much more. Benefiting CI, it creates version numbers that labels builds and makes variables available to the rest of your pipeline. Meanwhile, GitReleaseManager automatically creates, attaches, and publishes exportable release notes. If you only need the versioning of GitVersion, there is an alternative action with the same name later in this list. 5. Action Automatic Releases Once set to react to the GitHub events of your choosing (such as commits to your main branch), the action automatic releases workflow can: Auto-upload assets Create changelogs Create a new release Set the project to ‘pre-release’ 6. Repository Dispatch The repository dispatch action makes it easier to trigger actions from a 'repository dispatch' event. Plus, it lets you trigger and chain the actions from one or more repositories. You need to create a personal access token for this action to work as GitHub won’t support it by default. 7. PullPreview The PullPreview action allows you to preview pull requests by spinning up live environments for code reviews. When making a pull with the "pullpreview" label, this action checks out your code and deploys to an AWS Lightsail instance with Docker and Docker Compose. This allows you to play with your new features as your customers would, or to show off working models of your ideas. It also promises compatibility with your existing tools and full integration with GitHub. The only thing you should be aware of is that you need to buy a license if using it with commercial products. 8. ReportGenerator The ReportGenerator action can extract the most useful parts of coverage reports into easier-to-read formats. It allows you to read your data in the likes of HTML, and XML, plus various text summaries and language-specific formats. 9. Git Version While a little like the GitVersion tool enabled by the GitTools action, this Git version is an action itself. Like the external tool, however, it offers simple Semver versioning to help track your different releases. This is useful if you only want help versioning and don’t need GitReleaseManager. 10. GitHub Action Tester (github-action-tester) The github-action-tester is an action that lets you kick off shell scripts for testing. After it's installed, just add your scripts to your repository and kick them off with the events you need. How to Install Actions Installing actions in GitHub is simple: Find the action you want on the GitHub Marketplace. Read the Marketplace page to check for prerequisites. Click Use latest version in the top right (or select an older version if you need). Copy the code from the pop-up, paste it into your repository’s .yml file, and save. Make sure you read the action’s documentation to check for any extra setup and how to use the action. What Next? If the actions we chose don’t suit your project or you need something outside the scope of CI, there's plenty more to choose from. Search through the GitHub marketplace for more. Happy deployments!