2GIS CLI (dgCtl) | On-Premise | Urbi Documentation

2GIS CLI (dgCtl)

The 2GIS CLI utility (also known as dgCtl) is used for downloading and updating the artifacts required for deploying and updating the On-Premise services.

The updating and data management workflow is described in Overview.

The utility allows to retrieve artifacts from various sources. Depending on the presence of an internet connection, the available data sources and operation modes will vary.

The retrieved artifacts are kept in the selected type of storage and (if necessary) in the Docker registry. Two types of storage are supported:

  • an S3-compatible storage;
  • a filesystem.
dgctl apps to registry argument behavior

2GIS CLI allows to control how the artifacts are saved. Either of the following options is allowed:

  1. Keep the services data and other auxiliary artifacts in the selected type of storage while keeping the Docker images of the services in the Docker registry.

    To do so, run 2GIS CLI with the --apps-to-registry argument.

    Note:

    This option is recommended, because it gives you more control over storing the artifacts and helps to avoid mixing them.

  2. Keep all the artifacts in the selected type of storage (without using the Docker registry).

dgctl pull mode

If internet is available, 2GIS CLI can work in the pull mode: the utility will download the latest deployment artifacts from the 2GIS update servers to the selected storage and (if necessary) to the Docker registry.

When using a filesystem as a storage (the type: fs setting), the 2GIS CLI utility also creates a manifest file called manifests/latest.json containing the information about all the On-Premise services. The manifest file can be used for subsequent runs of 2GIS CLI in the restore mode with the --from-dir or --by-manifest arguments.

If the --service argument is passed in the pull mode, the utility will download the deployment artifacts for the specified service only. Note that even with this argument, the information in the manifest file will still be updated for all the On-Premise services.

dgctl restore mode

Without internet available, 2GIS CLI can work in the restore mode: the utility will load the deployment artifacts that are already stored in the source filesystem to the selected storage and (if necessary) to the Docker registry.

Important note:

The artifacts must be downloaded to the source filesystem in advance, using the pull mode.

  1. An access key for 2GIS CLI that allows to retrieve the deployment artifacts related to the purchased 2GIS products.

    To get this key, fill out the form at dev.2gis.com/order.

  2. Shared infrastructure:

    • Docker registry for storing Docker images.
    • Depending on the selected storage type:
      • S3-compatible deployment artifact storage.
      • Free disk space in the filesystem.
    • For the pull operation mode: internet access to the 2GIS update servers.

Detailed requirements for each service are listed in the overview section. Additional information can be found in Deployment considerations.

The configuration file has the following structure:

key: <2GIS CLI access key>
log-format: <text | json>

storage:
    type: <s3 | fs>

    # S3 storage settings
    host: <Deployment Artifacts Storage hostname and port>
    bucket: <Deployment Artifacts Storage bucket name>
    access-key: <Deployment Artifacts Storage access key>
    secret-key: <Deployment Artifacts Storage secret key>

    # FS (filesystem) storage settings
    directory: <target directory on a filesystem>

docker:
    registry:
        username: <Docker Registry username>
        password: <Docker Registry password>
        server-address: <Docker Registry URL>
        image-prefix: <Additional prefix (optional)>

Where:

  • key: access key for 2GIS CLI.

  • log-format: log format - text or json. All logs are printed into the standard output stream (stdout).

  • storage: settings for the selected storage type.

    • For an S3-compatible storage:

      • type: the storage type. Must be s3.
      • host: FQDN of the S3-compatible storage endpoint.
      • bucket: the bucket name for storing the deployment artifacts.
      • access-key: S3 access key.
      • secret-key: S3 secret key.
    • For a file system storage:

      • type: the storage type. Must be fs.
      • directory: the target directory in the filesystem to which the artifacts will be downloaded. The directory must exist and be mounted to the Docker container (see Running the utility).

    Important note:

    Only a single storage type's settings can be present in a single configuration file. If you need to run 2GIS CLI for different types of storage, create separate configuration files for them.

  • docker.registry: settings for accessing the Docker registry.

    • username: username.

    • password: password.

    • server-address: the registry URL.

    • image-prefix: additional prefix under which the On-Premise images will be stored.

      All On-Premise images downloaded using 2GIS CLI have a preset prefix 2gis-on-premise. You do not need to specify it here.

    Note:

    If the selected usage scenario of 2GIS CLI does not involve using the registry, this configuration section may be omitted.

To run 2GIS CLI, run the following command:

docker run --rm -v <path to YAML config file>:/config.yaml -v /var/run/docker.sock:/var/run/docker.sock -v /mnt/source-dgctl:/source-dgctl 2gis/dgctl:latest <command to execute> --config=/config.yaml <additional arguments>

Where:

  • <path to YAML config file> - path to YAML config file, e.g. ./config.yaml.

  • -v /var/run/docker.sock:/var/run/docker.sock - passing the Docker socket into the Docker container.

    This argument is used together with the --apps-to-registry argument. If you are not going to use the Docker registry, the arguments for connecting to the Docker socket can be omitted.

  • -v /mnt/source-dgctl:/source-dgctl - mounting the directory for storing the deployment artifacts.

    For the pull mode, if you are using the fs storage type, use the directory specified in the storage.directory setting of the YAML configuration file.

    For the restore mode, use the directory specified in the --from-dir argument.

    In other cases, this argument can be omitted.

  • <command to execute> - one of the commands defining the operation mode:

  • <additional arguments> - additional run arguments:

    • --apps-to-registry - if this argument is passed, the Docker images will be placed into the Docker registry and not into the storage.

      It is recommended to always use this argument, otherwise the Docker images will be placed into the storage with other artifacts. See more in the Architecture section.

      This argument is used together with -v /var/run/docker.sock:/var/run/docker.sock.

    • --workers - number of parallel worker threads which will download the data (defaults to 3, maximum is 8).

    • --attempt - number of attempts to restore the artifacts in case of write errors. Defaults to 3.

    • --overwrite - if this argument is passed, the existing files in the S3 storage and in the Docker registry will be overwritten.

    • --only-apps — download/restore the service images only, not the data.

    • --only-data — download/restore the service data only, not the images.

    • --service=<service> — download/restore artifacts for the specified service only.

      This argument can be used together with --only-apps or --only-data to download only images or only data for the specified service.

    • --version=<version> (pull mode only) - the On-Premise services version to download (e.g., --version=1.0.0). See the Releases section for the list of available versions.

    • --by-manifest=<path> (pull mode only) - the path to the manifest file that should be used for downloading the deployment artifacts.

      The manifest file and the directory containing the artifacts referenced in it must be available from within the Docker container. To do this, you need to mount the directory using the -v argument as explained above.

      The simplest way to create the manifest and the correct structure is to run 2GIS CLI in the pull mode with the filesystem as the storage.

    • --from-dir=<path> (restore mode only) - the filesystem directory from which the deployment artifacts should be restored.

      The directory must be available from within the Docker container. To do this, you need to mount it using the -v argument as explained above.

      The directory must contain a manifest file (manifests/latest.json), which describes the services' directories structure. The simplest way to create the manifest and the correct structure is to run 2GIS CLI in the pull mode with the filesystem as the storage.

    • --dry-run (restore mode only) - if the argument is passed, the uploading step will be skipped. 2GIS CLI will only output a list of files to upload to the standard output.

      It is highly recommended to run 2GIS CLI with this argument each time before restoring deployment artifacts, so that you can keep track of the planned changes.

  1. Create the config.yaml file for the required storage type:

    For downloading artifacts to an S3 storage:

    # Access key for 2GIS CLI.
    key: DEMO-KEY-DGCTL-AAAAAA-BBBBBB
    # Log format.
    log-format: json
    storage:
        type: s3
        # FQDN of the S3-compatible storage endpoint.
        host: artifacts.storage.local
        # The bucket name for storing the deployment artifacts.
        bucket: dgctl-store
        # S3 access key.
        access-key: AKIAIOSFODNN7EXAMPLE
        # S3 secret key.
        secret-key: wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY
    docker:
        registry:
            # Username for the Docker registry.
            username: registry-user
            # Password for the Docker registry.
            password: DOCKERregistryP@ssW0rd
            # The Docker registry URL.
            server-address: http://docker.registry.local:5000
            # Additional prefix under which the On-Premise images will be stored.
            image-prefix: /
    

    For downloading artifacts into the filesystem:

    # Access key for 2GIS CLI.
    key: DEMO-KEY-DGCTL-AAAAAA-BBBBBB
    # Log format.
    log-format: json
    storage:
        type: fs
        # The target directory in the filesystem to which the artifacts will be downloaded. The directory must exist.
        directory: /mnt/source-dgctl
    docker:
        registry:
            # Username for the Docker registry.
            username: registry-user
            # Password for the Docker registry.
            password: DOCKERregistryP@ssW0rd
            # The Docker registry URL.
            server-address: http://docker.registry.local:5000
            # Additional prefix under which the On-Premise images will be stored.
            image-prefix: /
    
  2. Run one of the following commands:

    To download the Docker images into the registry, and all other artifacts into the storage:

    docker run --rm -v $(pwd)/config.yaml:/config.yaml -v /mnt/source-dgctl:/source-dgctl -v /var/run/docker.sock:/var/run/docker.sock 2gis/dgctl:latest pull --config=/config.yaml --version=1.0.3 --apps-to-registry
    

    To download all the artifacts including the Docker images into the storage:

    docker run --rm -v $(pwd)/config.yaml:/config.yaml -v /mnt/source-dgctl:/source-dgctl 2gis/dgctl:latest pull --config=/config.yaml --version=1.0.3
    

If artifacts were downloaded to the filesystem earlier, they can be copied to another storage and (if necessary) to the Docker registry.

  1. Create the config.yaml file for the required storage type (see step 1 in the previous subsection). The access key for 2GIS CLI can be omitted in this case.

  2. Run one of the following commands, specifying the directory containing the downloaded artifacts (e.g., /mnt/source-dgctl):

    To download the Docker images into the registry, and all other artifacts into the storage:

    docker run --rm -v $(pwd)/config.yaml:/config.yaml -v /mnt/source-dgctl:/source-dgctl -v /var/run/docker.sock:/var/run/docker.sock 2gis/dgctl:latest restore --config=/config.yaml --from-dir=/source-dgctl --apps-to-registry
    

    To download all the artifacts including the Docker images into the storage:

    docker run --rm -v $(pwd)/config.yaml:/config.yaml -v /mnt/source-dgctl:/source-dgctl 2gis/dgctl:latest restore --config=/config.yaml --from-dir=/source-dgctl