Collecting logs
Below are recommendations for collecting and sending logs from services within an On-Premise installation.
Log format
On-Premise services in Kubernetes write logs to stdout in JSON or Plaintext format. In some cases, a combination of both formats is used, allowing flexible log output depending on system needs.
Tools for collecting logs
To collect and send logs, installing the following agents is recommended:
-
Fluent Bit - a lightweight tool with minimal resource consumption. Supports log filtering and routing.
-
Filebeat - optimal for integration with Elasticsearch (e.g., in ELK stacks).
Storing and analyzing logs
For centralized log collection, storage, and analysis, using the following tools is recommended:
-
Elasticsearch + Kibana (ELK stack). Suitable for complex queries and long-term log storage and provides the following features:
- full-text search and log analytics
- flexible dashboards and visualizations in Kibana
How it works: agents (Fluent Bit, Filebeat) collect logs from nodes and send them to Elasticsearch. Then data is indexed and becomes available in Kibana via index patterns.
-
Grafana Loki. Optimized for working with Kubernetes logs and provides the following features:
- cost-effective log storage (uses object storage, e.g., MinIO)
- integration with Grafana for combined log and metric analysis.
How it works: agents (Fluent Bit) send logs to Loki with labels for quick search. Then logs become available in Grafana via LogQL queries.