How to Install Curator For Cleansing Log in Kibana on RHEL 7

Hello everyone, welcome to our very proud kitsake blog. We hope you are all doing well. On this occasion, I will write about How to Install Curator with Yum For Cleansing Log in Kibana on RHEL 7.

Introduction

Curator provides a way to automate the process of deleting or managing older indices, which can help in optimizing disk space usage and improving the overall performance of the Elasticsearch cluster. 

It allows users to define rules for index management, such as retaining data for a certain period and then automatically deleting or archiving it.

Preparation

Download and install the public signing key:

# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Make Repository

# vi /etc/yum.repos.d/curator.repo

[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Installing

This will install the necessary files into /opt/elasticsearch-curator and make a symlink at /usr/bin/curator that points to the curator binary in the aforementioned directory.

# yum install elasticsearch-curator

Check Index Log

# curator_cli show_indices --verbose

How to Install Curator with Yum For Cleansing Log in Kibana on RHEL 7

Configure

In this configuration, you need to replace the Elasticsearch server's IP, and after making the changes, save and exit the configuration file.

# mkdir -p /etc/curator
# vi /etc/curator/config.yml

client:
 hosts: 192.xxx.xx.xxx
 port: 9200
 url_prefix:
 use_ssl: False
 certificate:
 client_cert:
 client_key:
 ssl_no_validate: False
 http_auth:
 timeout: 30
 master_only: False

logging:
 loglevel: INFO
 logfile: /var/log/curator.log
 logformat: default
 blacklist: ['elasticsearch', 'urllib3']

Then, we need to configure the YAML file that will serve as the logic for the Curator. Here, I set the call to action to 'Delete indices older than 90 days' based on the index name.

# vi /etc/curator/action.yml

actions:
 1:
  action: delete_indices
  description: Delete indices older than 90 days based on index name
  options:
    ignore_empty_list: True
    timeout_override:
    continue_if_exception: False
    disable_action: False

  filters:
  - filtertype: pattern
    kind: prefix
    value: kitsake-*
  - filtertype: age                   # Filter old age index
    source: name
    direction: older
    timestring: '%Y.%m.%d'
    unit: days
    unit_count: 90

Trying

Manual hit and keep 90 file / day.

To ensure the configuration made earlier, I will perform a manual test first before applying it to the scheduler. This way, once it's set up in the scheduler, it won't fail. 

# /usr/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml

Automatic hit and keep 90 file / day with cron.d.

Once you are confident that the configuration is correct, the next step is to create a scheduler to run automatically, ensuring that the disk space on the server remains preserved and doesn't bloat. This way, it will be automatically cleared based on the predefined configuration, saving time for us as system engineers. 

# vi /etc/cron.d/curator

00 00 * * * root /usr/bin/curator --config /etc/curator/config.yml /etc/curator/action.yml

And that's how I manage servers, especially ensuring the disk on the server remains preserved. That concludes the explanation of How to Install Curator with Yum for Cleansing Log in Kibana on RHEL 7.

Maybe that's all I can share with you guys, hopefully this article will be useful.

Thank You.