Elasticsearch Tutorial for Beginners: Elasticsearch Basics

ElasticSearch (ES) is a noSQL JSON (not only SQL JavaScript Object Notation) database. Its primary application is to store logs from applications, network devices, operating systems, etc. But it is suitable for the storage of any kind of JSON document. So, you could use it instead of, for example, MongoDB. Yet, MongoDB has native support for JavaScript, as we explained here, which you will find useful. ES does has not such a REPL (read-eval-print loop) command line interface, except for Curator, which can be used for admin functions.

(This article is part of our ElasticSearch Guide. Use the right-hand menu to navigate.)

What’s included in this Guide

In this guide we will cover the most important ElasticSearch topics.

This includes:

• How to set up an ElasticSearch Cluster, giving both versions 6.x and 7.x instructions, since version 7 is substantially different.
• How to use Kibana (i.e., the ES dashboard)
• The Lucent Query Language
• How JSON databases differ from traditional SQL databases, like Oracle
• Using Apache PIG with ES
• Using Apache Spark and Apache Spark Machine Learning with ES
• ElasticSearch Indexes
• Using Filebeat and Logstash to parse web server, router, and custom and off-the-shelf application logs.
• Nesting documents within documents
• System administration
• Adding clusters and re-indexing documents

JSON documents and noSQL

First it is necessary to understand what a JSON database is and does. Traditionally, databases store records in tables in columns. That is called a relational database (RDBMS). The most widely-known example of this is Oracle, invented in the 1970s from a paper written by IBM, a product that has made Larry Ellison quite a rich man and a product that gave him a monopoly for many years. (There are open source alternatives, like MySQL. But what do monopolies do? They buy up the competition. Oracle has bought MySQL.)

Data stored in an RDBMS is put together in a join operation and the data is normalized in most cases. As an example of this, consider an inventory system. You have products and then a table of inventory movements (sale, stock, loss, etc.). So, you would probably store these in two separate tables, like this:

The common elements upon which you join the two are the UPC/EAN numbers (Universal Product Code and and European Article Number.) Those are the barcodes scanned at the grocery store. The product keeps the inventory count. The inventory movement table keeps the sale, receipt, loss, and other transactions on hand.

JSON databases do not support, or rather encourage, join operations, although that operation can be forced. Why? Because joining two tables is an expensive operation, because it must scan all the records of one table to find match records in another. Of course, that operation can be sped up using indexes.

Instead of doing joins, the JSON database embeds documents inside each other, like shown below, where the transactions are inside the product document.

{
"product": "led lightbulbs 10w 6 pack",
"EAN": 978020137962,
"transactions": [{
"trans": "sale",
"quantity": -100
},
{
"trans": "receive",
"quantity": 6100
}
],
"stock": 8000
}