Common Database Documentation

This page contains documentation common to all SCALE database stacks (MongoDB, PostgreSQL, MariaDB, Redis). Individual database pages link here for shared procedures and configuration details.

Getting Started

Refer to the following link to learn about getting started in the Azure DevOps environment: Getting Started in Azure DevOps Environment

Repository Structure

The structure of the repository for a database stack will contain a conf directory with few sub directory structures for custom config files, and one or two directory trees with details for deployment, as follows:

• conf - this directory will contain directory stucture to place custom configuration files.

• appCode-serviceName - (named as the AppCode followed by the application or service name) contains the Helm chart deployment materials if Helm v2 charts are used (this is the default)

• manifests - contains Kustomize manifests if Kustomize overlay files are used as an alternative to Helm charts

Also at the top of the repository is a file called azure-pipelines.yml. This file contains reference to the appropriate version of the CI/CD pipeline logic, some variables unique to the application (e.g. container version) as well as YAML data structures providing key information about the environments into which to deploy the application and the sequence of events to complete the deployment (e.g. dependencies, additional steps to retrieve secrets to be passed to the deployed container, etc).

Additional items in this repository will generally not be modified and should not be changed to avoid risk of breaking the pipeline workflows.

CI/CD Pipeline Overview

Once navigated to the appropriate repository, it can be cloned to a local development environment (i.e. local workstation) for the actual software development and configuration work for the database to be deployed. Configuration of the pipeline would include setting the appropriate container version (reflected in the appVersion variable under the parameters: section) and defining the target workspace and dataspace through template: under extends:. Once it is ready to be deployed into the SCALE workspace and the new application version has been set as well, create a Git pull request for the updates to the code. The creation of this pull request will trigger the Continuous Integration pipeline to start.

Please note that the document "SCALE Concepts and Procedures" contains more details about the specific flow of the application through the CI/CD pipelines

Once a pull request has been created against the Git repository, it will trigger the start of the CI/CD pipeline in order to build, scan and prepare the application to deploy into the Workspace. Although the CI/CD pipeline flow is automatically triggered, there are times when it needs to be manually run (or re-run) and the pipeline should be examined for the results of its preparation of the application and deployments. Details for examining the CI/CD pipeline and analyzing its results can be found here: Continuous Integration and Continuous Delivery Pipelines

Pipeline Definition

The logic for the database CI/CD pipeline is a common and shared code base for all applications; however the configuration of the pipeline that applies the common logic to the specific application is defined in the top level directory of the source code repository for the application in a file named azure-pipelines.yml.

The structure of the azure-pipelines.yml file is a YAML file using standard YAML syntax. For general information about YAML structures, there are many available resources, including the tutorial at this link: YAML Tutorial.

There are certain required data elements that must be defined within the azure-pipelines.yml file as a prerequisite to the CI/CD pipeline running while other elements are optional and used to modify the standard behavior or the CI/CD pipeline.

More details about the azure-pipelines.yml file can be found here: General Structure of the azure-pipelines.yml File

Global Variables

The variables YAML object defines simple variables accessible across the pipeline logic. For database pipelines, there is one required variable:

extends:
  parameters:
    appVersion: <version>

which specifies the version of the container to be pulled from Artifactory. The name of the container will match the name of the application defining the pipeline.

Pipeline Modes

Along with deployments, the pipeline enables you to run backups, list available backups, and restore a backup.

parameters:
  - {
      name: mode,
      displayName: "Select action to perform",
      type: "string",
      values: ["refresh", "backup", "backup-list", "restore"],
      default: "refresh",
    }
  - { name: restoreFrom, displayName: "Provide source to restore from", type: string, default: "<dataspace repo name>" }
  - { name: backupId, displayName: "Provide backup id to restore from", type: string, default: "latest" }

Default mode to run the pipeline is refresh and can be changed as per the requirments. Values for restorefrom and backupId are required only during restore operations.

Backup Schedule

The pipeline enables you to run scheduled backups within azure-pipelines.yaml. (Optional) schedules:

- cron: "05 04 * * *"
  displayName: "Daily at 04:05 AM"
  branches:
    include:
    - <branch-name>
  always: true

Please note that schedule must be defined in the master branch. You can refer this site Crontab to build your cron schedule.

CPU Request Tuning

By default database pods will start with default CPU size mentioned in the spaces file. To override this value to save the CPU usage, use the following section in the azure-pipelines.yaml

extends:
  parameters:
    appVersion: <version>
    <database-type>:
      podSpec:
        cpuRequests: 200m	

Replace <database-type> with the appropriate value for your database (e.g. mongodb, postgresql, mariadb, redis).

For the existing dataspace, review the splunk report and set the value 2 or 3 times of max usage. For example, if the max CPU usage for the database pod is "50m" then set the "cpuRequests" value as "150m".

You need to review the splunk reports periodically and tune the "cpuRequests" value.

Database Configuration and Deployment

Database will be automatically configured with random password during provision. To modify the details of database passwords, you need to update <repo-name>.props variable group in pipeline library.

Refer to your specific database documentation page for screenshots and database-specific props configuration.

Backups using DBaaS self-service option is discontinued for version 4.3, please refer to backup and restore document.

Helm Override Files

To override values for dataspace, you can update the following section in azure-pipelines.yaml

workspace:
  helm:
    overrideFiles: |
      dbaas-<dbtype>/values.workspace.yaml
dataspace:
  helm:
    overrideValues: |

Replace <dbtype> with the appropriate value for your database (e.g. mongodb1, postgres1, mariadb1, redis1).

Changing Database User Password

1. Update the password in props variable group and send it to ops team.
2. Ops team need to update the password value in props variable group located in pipeline library.
3. Update the release pipeline and modify the download secret file task and select the uploaded secret file from dropdown list.
4. Create the release to update the rancher secrets.
5. Now the DB pod got recreated with new password. Login to database using new password and check.

Refer to your specific database documentation page for the password change video walkthrough.

Exposing Databases Outside the Cluster

See our documention on SCALE Stunnel Client

Sysdig Monitoring

Sysdig is a powerful monitoring and troubleshooting tool that provides deep visibility into system-level metrics and events. It allows you to monitor and analyze the performance of your systems and databases, by capturing and analyzing system-level metrics such as CPU usage, memory consumption, disk I/O, and network traffic. With Sysdig, you can identify database-related processes, monitor resource utilization, analyze system events, and create custom metrics and alerts. Overall, Sysdig is a versatile tool that can help you gain a better understanding of the performance and health of your databases and the underlying systems they rely on.

Sysdig exporter can be enabled to facilitate monitoring and data collection using Sysdig. By updating the values.dataspace.yaml file as following.

## Flag to enable sysdig exporter
exporter:
  enabled: true

DR Changes

With the introduction of pipeline v4.4+, we have implemented DR capabilities that require changes in the Helm chart. Please refer to the document below for the necessary modifications and kindly update your Helm chart accordingly.

Getting Started with DR

Detailed Pipeline Configuration

The remainder of the configuration work in the azure-pipelines.yml file focuses primarily on defining the target workspace and dataspace for the database and providing details for these spaces, any additional tasks needed to prepare the deployments, and dependencies that will dictate the sequence of deployments in these spaces beyond the established SCALE environment deployment approval processes. Details for configuring these elements of the pipeline can be found here: Pipeline Configuration Details

Kubernetes Deployment Objects

In order to deploy an application a number of Kubernetes objects must be defined and deployed. The definition of these objects is controlled through a set of files in one of two forms: Either as Helm charts (the default method) or as Kustomize files (a newer form of deployment description which has limited support in V4). Information about the contents and customization of these Kubernetes deployment objects can be found here: Kubernetes Deployment Objects

Troubleshooting

If something fails to deploy, the information about why the deployment failed (or was not even initiated) will be found in the logs of the CI/CD pipeline and can be tracked down using the methods described earlier in the "Continuous Integration and Continuous Delivery Pipelines" section.

However, additional information may be required either to better troubleshoot a failed deployment or to investigate the runtime behavior of an application that has been successfully deployed. In those cases, much of the information can be found in the rancher web console. Information about navigating and analyzing information from the rancher web console can be found here: Navigating the rancher Web Console