Installing the Data Agent - Kubernetes

Overview

The Validatar Data Agent can be launched within a Kubernetes cluster by following these steps:

1. In Validatar, navigate to the Settings > Configuration > Data Agents page, then click the New Data Agent button

2. Enter a name for the data agent. The data agent name should identify the environment where the data agent is installed (i.e. the cluster name).

3. Choose an existing group for the data agent or create a new group if needed. The group name should identify the environment where the data agent is installed (i.e. the cluster name). Multiple data agents can be assigned to the same group.

4. Click the Save button. It will take a few seconds for Validatar to generate the message queue for the data agent group.

5. Once the data agent is saved, click the Download YAML button to generate a YAML template that contains a deployment and a service for the data agent. Open the downloaded YAML file and adjust as needed for your environment.

6. Import the YAML into your Kubernetes cluster.

Data Agent Image

The data agent image published to Validatar's public repository contains the following libraries:

• Python 3.11 and pip, along with the following Python libraries:

  • pandas

  • numpy

  • pyarrow

  • fastavro

  • openpyxl

  • requests

  • urllib3

• PostgreSQL ODBC driver

• SQL Server ODBC driver (msodbcsql18)

• Snowflake ODBC driver (version 3.4.0)

• Tesseract OCR library

Image Customization

The generated YAML template references the latest data agent image located in Validatar's public repository. If you'd like to customize the image to add additional Python libraries or other ODBC drivers, create a separate Dockerfile using Validatar's public image as a starting point and apply additional layers as needed. Then build your Dockerfile, publish your customized image, and adjust the YAML file to point to your customized image before importing into your Kubernetes cluster.

The sample Dockerfile below can be used as a starting point to customize the base data agent image.

FROM public.ecr.aws/validatar/data-agent:2024.3.0.5396
WORKDIR /app
EXPOSE 3282

RUN \
  # TODO: Get updates and install additional packages
  apt-get update -y -q \
  && apt-get install -y -q --no-install-recommends curl \
  # Cleanup temporary files
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get remove -y curl
  
ENTRYPOINT ["./validatar-data-agent-service"]

Once you’ve customized the Dockerfile, you can build it using the following command run from the folder that contains your Dockerfile:

docker build -f ./Dockerfile -t custom-data-agent:2024.3.0.5396 .