Elastic - ecosystem
More than a search engine

Elastic does not come with a bare search engine only. As we could learn in the previous article, we do have a Lucene-based search engine here that exposes REST service. As you probably understand, there is no point in selling a product without the proper user interface tools. For example, having a database is the core of an app, but some CRUD interface has to be present. 

This article is a part of the Text Search series by Łukasz Kuczyński

1. You are not the only one searching 
2. Elastic – its engine
3. Elastic – ecosystem (you are here)

Even the tech-savvy database admin is pleased to work with administrative tools (PgAdmin for Postgres, but there are better options). By the way: We recently had a brief meeting at Exlabs when the guys were presenting their favorite tools, PgAdmin was one of the choices from a set of tools like DBeaver, QueryPie, Adminer, etc. 

This article is intended to help you understand what is inside the Elastic ecosystem and what tools we have there.

Low-level access

Elasticsearch exposes REST API which can be consumed or used in a variety of solutions. To be precise, we are not forced to use anything else but this API if we want to use the full 

range of features.

As time goes by, there are many clients built on top of Elasticsearch. One of the most developed is Python which is a great combination of two things: it is an official, up-to-date project, and it is written in the most popular scripting language in the programming world. With such a tool, all Elasticsearch-related operations are exposed as API calls, f.e. to index some documents it’s enough to follow these simple commands:

```py
from datetime import datetime
from elasticsearch import Elasticsearch
es = Elasticsearch()
doc = {
'author': 'Exlabs',
'text': 'Elasticsearch: cool. bonsai cool.',
'timestamp': datetime.now(),
}
res = es.index(index="test-index", id=1, body=doc)
print(res['result'])
res = es.get(index="test-index", id=1)
print(res['_source'])
es.indices.refresh(index="test-index")
res = es.search(index="test-index", body={"query": {"match_all": {}}})
print("Got %d Hits:" % res['hits']['total']['value'])
for hit in res['hits']['hits']:
print("%(timestamp)s %(author)s: %(text)s" % hit["_source"])
```

If you need to use different languages or embed Elasticsearch calls into your application written in a language different than Python, you can choose from a set of languages such as Java, Javascript, Ruby, Go to name a few (complete list here).

With such access, we must be aware that it is for development purposes. In real-world scenarios, there are already designed tools that facilitate easy browsing and searching (Kibana) and simplified ingestion of data (Beats, Logstash). We also have Enterprise Search tools that you can use as out-of-the-box apps for workplace scenarios and search clients for web applications.

ELK —> Elastic

Monitoring is a very important part of taking care of your business. It’s better to know what is happening around you than to be caught by surprise by the effects of malfunctioning. Logs are created by most of the system. If no cloud solution works for you, it’s good to think about centralised log management.

As far as Elastic company is concerned, the remedy for that problem is Elastic stack. For many years it was ELK that aggregated information access for Elasticsearch. ELK is the acronym for Elasticsearch + Logstash + Kibana, the stack of tools for log management. For years ELK has served as the only proper remedy for the log management domain. E stands for Elasticsearch being the heart or brain of the whole solution, serving as a search engine for all queries and indexing data coming from files. L stands for Logstash, a Ruby tool that serves as a simple ETL for Elasticsearch filling it with data (and to run it you had to know grok well :)). Finally, K stands for Kibana, which is a dashboarding tool created and tailored to the needs of the ELK stack. We can also use these tools nowadays, however, Elastic has changed the naming conventions from ELK to Elastic stack, which is the preferred option to run Logs management here.

After some years of community evaluation, Logstash was judged as a heavy tool. Usually, there is no need to run it with the whole harness of transformations. Users needed a simple, small app to “tail a file”, run and just forward a text to Elasticsearch with minimal to none transformations. And that was the day when Beats were born. They are now obvious “go-to” options when creating a log analysis stack with Elasticsearch. These simple and lightweight connectors serve as a solution to get real data onboard. Some of the transformation power has been moved to Elasticsearch when ingest nodes were introduced.

The aforementioned building blocks are a good way to create custom solutions – then you can build any custom solution based on Elasticsearch. 

On some occasions, though, you would like to have some off-the-shelf products. Something that you can just deploy and connect to an Elasticsearch instance. This is where Enterprise Search shines. Let us discuss some details here.

Enterprise Search

Enterprise Search is the Elastic company’s answer to the needs of search in business domains. Before Enterprise search, other solutions had a significant advantage over Elastic stack because they served specific needs without a need to write boilerplate code. Let us have a simple search component being an essential part of websites selling some products

Many websites require simple search functionality, a textbox in HTML terms that will initiate search and return results after someone hits Enter or clicks the magnifying glass symbol.

To make it happen, before Enterprise was released, you had to search for some custom components and connect them with Elasticsearch using REST calls.

Currently, one needs to employ and customize AppSearch, an off-the-shelf solution, to provide a search experience for a website’s customers. Let us discuss the value proposition of Enterprise Search components. 

AppSearch

Whenever one delivers a search solution based on Elasticsearch, he should think of some custom client app, which will connect to Elasticsearch API, reuse some client library or call REST APIs directly and provide results to GUI which will show it to users. AppSearch provides an additional layer of functionality which is great for creating a fast search experience that can be customized, tailored to the needs of search apps. Examples can be

• searching for restaurants in the area
• products discovery on your site

With AppSearch there are not only GUI components that can be used separately or employed in your website, but there is also a customization process that works well for administrative purposes. For example, with an easy set of instructions, the administrator can modify search settings that rule how items are searched and presented. This is about:

• controlling relevance
• working with synonyms
• curating results

All of that is employed into one abstract object called an engine.

It is hard to learn something new just in theoretical terms so AppSearch is delivered to you with a sample engine you can try out – a collection of 59 National Parks in the United States.

When you open the homepage of a search engine some stats help to dig into how it is used and how much data it contains. There are various typical search-connected tasks available in AppSearch, so you can test your queries and fine-tune them. There is also a cool way to export the simple search boilerplate in React, so it can be used as a basis to start a new website for search experience. There is no need to search for some custom 3rd party libraries if we have some examples from Elastic guys. There is a way to decide for yourself what fields of a collection will be used to filter or sort items. Below you will find how this configuration steps and how the sample result looks.

Elastic Workplace Search

Wouldn’t it be great to have the one and only place for all your documents, the focal point of tribal knowledge? This is what is valuable, especially when information is decentralized. Perhaps a company is just about to take steps toward digital transformation and data can be found virtually anywhere. 

Imagine, for instance, some information is in JIRA (yes, it is about task-management, but come on, some documentation is naturally created here), you use GSuite components for file sharing and mails, there is a ticketing system like ServiceNow, and you use Slack for internal communication. If you are searching for some particular information regarding a client or something else, you can either:

• ask yourself “where” and use search in the original search tool
• reuse centralized search experience of Workplace Search that was previously fed with this data

This is exactly why some companies decide to go for Workplace Search.

Workplace Search comes with a variety of connectors. With any of them, one can simply plug and play, configure sources, and leave the rest in the hands of Workplace Search. At the time of writing this article, we have almost 10 different connectors as shown in the image below. 

For instance, to create a Google Drive connector, it is enough to create an OAuth app in the Google Developer Console and use your credentials to configure the connector. The rest you can explore in the official docs. 

We are not limited to using proprietary data sources. Developers can leverage Custom API Sources. Thus, there are no limits to extend Workplace functionality.

Run and deploy

Every application or system has to live somewhere to work. For development and dry run we recommend 14 days of Free Trial on Elastic Cloud. This is the curated environment with always the latest version of the stack. You worry about nothing, just select the deployment setup and cloud provider you would like to choose, and all the plumbing is done behind the scenes. The only thing is to remember that we should decide if we go for it or cancel our subscription after the evaluation period.

When you are satisfied with the solution, this approach makes it easy for you to stick to the solution for both development and production purposes. PAAS solution can be sufficient for both you and your clients because it enables you to easily extend your infrastructure. Another reason is that you can perform rolling upgrades of the stack and you don’t need to worry about taking care of the hardware that runs it.

For developers who just want to use some basic tools (f.e. the former ELK stack) and connect it directly with their applications, Elasticsearch comes with Docker images that are easy to run and connect to.

We can also have Elastic stack deployed on virtual machines in IAAS scenarios. In this case, however, we should be aware of the time needed in advance to configure routing, security extensions, and care about data duplication, which is the default value in the Elasticsearch engine (so-called replication factor).

Conclusion

Text search is not a trivial task. There are many nice tools designed for this problem, so always be ready to learn and expand your horizons. Some may stay with their classic relational database solution, and it works great to some extent. If you need to employ a simple search to your website or work with e-commerce solutions, Algolia will be a great choice. If you want to use something curated and used for years, you need to check Elastic stack. With that, you can satisfy the needs of both Workplace search and creating a custom UI for e-commerce domains.

One tool does not fit it all, so keep on searching!