Skip to main content

ELK SETUP

This doc cover's the complete setup of elk and kibana.

Elk and Kibana Installation Insight

Note: If need to install any specific version then change the version accordingly in all the above commands

Setup

  • Create a server along with new security group. Before creating a server first get all the details from team/lead such as:

    • Subnet
    • AMI (Current version 8.7.0 in AMI)
    • Vpc
    • Security Group
    • Iam Role

    Note: Also, you can take a refrence from exsisting elk server as well.

  • Single node architecture:

    • Elk config changes in elasticsearch.yml file inside /etc/elasticsearh/ directory.

      • For config related changes you need to take a reference from a file that is stored in s3 Location. Changes that you need to done are given below .
        • Add the cluster-name setting and enter a name for your cluster:
          • cluster.name: my-cluster
        • Add the node.name setting and enter a name for the node. The node name defaults to the hostname of the machine when Elasticsearch starts.
          • node.name: node-1
        • Add your server private ip in discovery.seed_hosts: ["private_ip"] attribute.
        • Creating the certificates for internode communication by folowing commands.
        • Command's for creating certificate for internode communication:
          • Change the user to root using sudo su command .
          • Go inside /usr/share/elasticsearch/ and check if elastic-stack-ca.p12 and elastic-certificates.p12 file exist then first need to delete these files using rm -rf elastic-stack-ca.p12 elastic-certificates.p12 command.
          • ./bin/elasticsearch-certutil ca this command is used to sign your certificates.The output file named elastic-stack-ca.p12 contains the public certificate for your CA and the private key used to sign certificates for each node.
          • ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 this command generate a certificate and private key for the nodes in your cluster.The output file is a keystore named elastic-certificates.p12. This file contains a node certificate, node key, and CA certificate.
          • Please verify if elastic-certificates.p12 file exist in /etc/elasticsearch/certs and elastic-stack-ca.p12 file in /etc/elasticsearch/ca then delete both these files.Also if any other files exist then please delete those files too.
          • After creating elastic-certificates.p12 file make sure to move this certificate inside /etc/elasticsearch/certs aslo move elastic-stack-ca.p12 inside /etc/elasticsearch/ca.And make sure the permisions for both files are 755 .
          • After creating a certificate make sure the following lines are present in config file.
            • xpack.security.enabled: true
            • xpack.security.transport.ssl.enabled: true
            • xpack.security.transport.ssl.verification_mode: none
            • xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
            • xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

      After complete all steps to set up basic security for the elastic stack.Encrypt HTTP client communications for elasticsearch. Steps are given below .

      • From the directory where you installed Elasticsearch, run the Elasticsearch HTTP certificate tool to generate a Certificate Signing Request (CSR).Command is given below.
        • First delete elasticsearch-ssl-http.zip file from /usr/share/elasticsearch/ directory if exist.
        • ./bin/elasticsearch-certutil http this command will ask for inputs through prompt, please refer to the section "Encrypt HTTP client communications for Elasticsearch" in the following link.Few prompt avoid by giving following inputs.
          • Skip while password prompt appears.
          • Skip When asked if you want to generate one certificate per node.
          • While ask for hotsname just provide *.oriserve.com
          • While ask for Ip address just skip this one.
        • After doing all the above steps it will create compressed file contains two folders one for Elasticsearch and another one for Kibana. Unzip the generated elasticsearch-ssl-http.zip file and make sure both elasticsearch and kibana folder created.
        • Copy /usr/share/elasticsearch/elasticsearch/http.12 to /etc/elasticsearch/ca/ this location. Also copy another file /usr/share/elasticsearch/kibana/elasticsearch-ca.pem to /etc/elasticsearch/kibana/ca/ .
        • After doing all the above steps make sure the following lines are in the file.
          • xpack.security.http.ssl.enabled: true
          • xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/ca/http.p12"
          • xpack.security.authc.api_key.enabled: true

    • Kibana config changes in kibana.yml file inside /etc/kibana/ directory.

      • For config related changes you need to take a reference from a file that is stored in s3 Location. Changes that you need to done are given below .
      • Replace elasticsearch.hosts: ["https://"private_ip":9200"] private ip with your server private ip.
      • Make sure below line are present in kibana.yml.
        • xpack.encryptedSavedObjects.encryptionKey: "123456781234567812345678123456789"
        • xpack.security.session.idleTimeout: "1h"
        • xpack.security.session.lifespan: "30d"
        • elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca/elasticsearch-ca.pem" ]
        • elasticsearch.ssl.verificationMode: "certificate"

  • Multiple node architecture:

  • After setting up the server, modify elk nginx conf according to your url and remove existing certificates from the conf. Path for the elk nginx conf /etc/nginx/sites-available/elk.

  • Then check the syntax using sudo nginx -t.

  • If syntax is okay then restart the nginx service using below command. sudo systemctl restart nginx

  • Then check the status as well.

  • Also map your domain with server ip in Route53 and generate the ssl certificate using certbot using below command.

    • sudo certbot -d domain_name --nginx

  • After doing all the above steps, check that all services are running fine or not with the help below commands:

    • sudo systemctl status nodeexporter
    • sudo systemctl restart elasticsearch.service
    • sudo systemctl restart kibana.service
    • sudo systemctl restart nginx
    • check status as well for all above commands.

  • If any of the above commands are not running fine.

    • Check the service logs and try to find out the issue.

  • If all the above commands are running fine.

    • Check the url on google , if it shows the elk page then you are good to go.

Data and logs cleanup of elasticsearch

  • Steps to cleanup are given below.
    • First go inside data directory at this location /var/lib/elasticsearch/ and delete all items from it using below command.
      • rm -rf *
    • Now delete all the data from logs directory which is present at /var/log/elasticsearch/ .
    • After cleaning up all the data and logs, we need to set the password for all built in users.
    • Before setting up the password for all users we need to change the value from true to false disable xpack.security.http.ssl.enabled: false in /etc/elasticsearch/elasticsearch.yml file.
    • Now generate the password for all users using below command .
      • /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
      • Prompt will appear to enter the password for all users, make sure the password should be difficult. Note: Password for users should be same, if you enter different password for all users then please make the list of all users password and share it with lead.
    • After generating the password make sure you have changed the kibana_system password in /etc/kibana/kibana.yml file.
    • After doing all the above steps make sure you have change the value from false to true disable xpack.security.http.ssl.enabled: true in /etc/elasticsearch/elasticsearch.yml file
    • Restart elasticsearch and kibana service.

Files share over S3

  • Steps to create S3 structure locally are given below:
    • Create a parent folder named as elkProduct.
    • Go to this S3 directory link and create all sub folders and files locally inside the parent folder.
    • After doing the above steps connect with team/lead to confirm the parent folder name.Also share the zip file of s3 structure to team/lead and told him to upload to given s3 directory link.
    • Once structure uploaded on s3 then from the server upload all certificates and ca files in respective folders.

Upgradation of elk and kibana

  • Updating elk and kibana version from 7.10.0 to 8.7.0.

    • Upgradation steps for elk:

    • Upgradation steps for Kibana:

      • Follow the same steps except last setp as Upgradation steps for elk section but the commands are changed for download and installating the kibana.
      • For reference commands are given below .
      • Update node.option file in /etc/kibana/node.option by taking the reference from s3 link.