Skip to main content

Installation artifacts lifecycle

Delivering installation artifacts​

  1. DGCLI downloads the installation artifacts from Public Update Servers.

  2. DGCLI places the fetched datasets into the installation artifacts storage. (The Docker images can be placed into the Docker Registry.)

    See more in the DGCLI utility description.

    Monitor the storage status

    Installation artifacts storage requires regular maintenance to clear out the outdated installation artifacts. This helps to prevent overflow of the storage space.

    DGCLI does not track and does not manage free space in installation artifacts storage or Docker Registry. It is recommended to set up monitoring for these parts of infrastructure and perform regular maintenance.

  3. All artifacts then migrate from public to private network, so that they become available to Helm and On-Premise services.

    The migration process can be implemented in different ways depending on the specifics of the project. For example, you can install Docker Registry and an S3 compatible storage in the private network and configure synchronization between them and the corresponding entities in the public network.

Importing data during installation or updating of a service​

Installation of a service includes copying required datasets from the installation artifacts storage (see the previous section) into one or multiple storages that the service will use, e.g., into a PostgreSQL database. Oftentimes, a special Kubernetes Importer job exists for this purpose, providing the following lifecycle for a dataset:

  1. The job reads a manifest file from the installation artifacts storage. This file contains a list of objects stored in the installation artifacts storage and their latest versions.

  2. The job uses the manifest to determine if there is a new piece of data for the service. If there is no new data, the job stops.

  3. The job spawns some workers. Each worker fetches the necessary installation artifacts and imports the new data to the service's data storage as a separate copy. Data import time depends on the size of the territory included in the key and can take up to several hours.

  4. After the workers complete the data import, the job performs a series of health checks to ensure the integrity of the new data.

    If all checks are completed successfully, the job removes the original data, replacing them with the new data.

    If one or more checks fails, the job stops the updating process and requires actions from the system administrator. The original data is left intact.

Common scenarios for updating services and datasets​

The first step of updating any service is fetching installation artifacts using the DGCLI pull mode. In this mode, a manifest file is created, which contains information about the service including its version. Every time the DGCLI pull mode is run, a new manifest file is created, the older files are not changed.

Next, different scenarios for updating services and datasets are possible:

Updating a service without updating its data​

Helm updates the service a similar way to how the Kubernetes job updates the data (see the previous section): new instances of the services will be deployed in addition to the current ones, and if health checks are completed successfully, traffic is redirected to the new instances. Otherwise, the process stops, requiring actions from the system administrator.

To update a service, specify the required version in the --version flag when running the helm upgrade command, for example:

helm upgrade --version=VERSION --atomic --values ./values-search.yaml search-api 2gis-on-premise/search-api
warning

Versions of services within a component must be the same and match the component version.

Updating a service and its data with Helm​

note

This scenario is not supported by certain services.

The update process includes two stages:

  1. Helm launches the service's Kubernetes Importer job to update the data. To make the job recognize that new data is available, specify a new manifest in the dgctlStorage.manifest parameter of the values.yaml service configuration file.
  2. Helm updates the service to the version specified in the --version flag (see the previous scenario).

Updating a service data only​

Differences in the update process

Some services may not support updating datasets, or the updating process may differ from the described one.

For a specific service's updating process description, see its documentation in the Update section.

The corresponding Kubernetes Importer job is scheduled to run, for example, on an everyday basis.

To avoid updating a service version, specify its current version in the --version flag when running the helm upgrade command. To make the job recognize that new data is available, specify a new manifest in the dgctlStorage.manifest parameter of the values.yaml service configuration file.

Cleaning old data​

To free up space in the storage, regularly clean irrelevant data. You also need to keep manifests for the current and some previous versions of data so that you can revert to it in case of problems.

If you use DGCLI utility version 3 or later, you can delete all manifests that were created before the data was split into components.

Do the cleanup in the following stages:

  1. For the first cleanup:

    1. Get a list of all manifests in the storage:

      dgctl manifest list

    2. Remove the manifests, based on which your environment is deployed, from the list.

    3. Delete other manifests:

      dgctl manifest delete --manifest-name <manifest-name>.json

  2. Create two files with lists of manifests for daily and monthly data fetch.

  3. For the next cleanups:

    1. After each run of the dgctl pull command, write the name of the received manifest to the end of the corresponding file.

    2. Decide how many manifests you want to keep to enable revering to the previous version.

    3. Delete all other manifests:

      dgctl manifest delete --manifest-name <manifest-name>.json