Elasticsearch and OpenSearch Used From Go Services

When users ask for a search box that finds things by meaning rather than exact match, with typo tolerance, ranking, highlighting and filters, a relational database starts to strain. Elasticsearch, and its fork OpenSearch, are built for it: JSON documents indexed into an inverted index and queried with a rich query language, spread across shards for scale. They are not a database you can trust with the only copy of your data. They are a derived index that you fill from your system of record and rebuild when needed. This guide covers using them from Go, and the design points that decide whether the integration works.

It was not run. No Elasticsearch or OpenSearch node was available in the lab behind this series, and a node needs a gigabyte of memory and a few minutes to start. The Go program below was compiled and passed go vet against go-elasticsearch v9.5.2, so the API calls exist and have the shapes shown, but no output is quoted and nothing was executed against a cluster. Statements about behaviour come from the Elastic documentation. Check them against the version you run.

It belongs to the database cornerstone. The in-database alternative for modest needs is Postgres full text search, and the way to feed an index reliably is the outbox pattern.

Elasticsearch or OpenSearch

OpenSearch is a fork of Elasticsearch 7.10 created by AWS after Elastic changed its licence in 2021. The two have diverged since. They share the core idea and much of the query language, and they differ in newer features, in the security and licensing story, and in Go clients: Elastic publishes github.com/elastic/go-elasticsearch, and OpenSearch publishes github.com/opensearch-project/opensearch-go. The client's major version tracks the server's: use go-elasticsearch/v9 against a version 9 server. If you are on a managed service, use the client its vendor supports. The concepts below apply to both, and the code uses the Elastic client.

The client

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/elastic/go-elasticsearch/v9"
	"github.com/elastic/go-elasticsearch/v9/typedapi/core/search"
	"github.com/elastic/go-elasticsearch/v9/typedapi/indices/create"
	"github.com/elastic/go-elasticsearch/v9/typedapi/types"
	"github.com/elastic/go-elasticsearch/v9/typedapi/types/enums/refresh"
)

type Post struct {
	Title string   `json:"title"`
	Body  string   `json:"body"`
	Tags  []string `json:"tags"`
	Views int      `json:"views"`
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	es, err := elasticsearch.NewTypedClient(elasticsearch.Config{Addresses: []string{"http://127.0.0.1:9200"}})
	if err != nil {
		log.Fatal(err)
	}

	// An explicit mapping: text for search, keyword for exact filters.
	_, err = es.Indices.Create("posts").Request(&create.Request{
		Mappings: &types.TypeMapping{Properties: map[string]types.Property{
			"title": types.NewTextProperty(),
			"body":  types.NewTextProperty(),
			"tags":  types.NewKeywordProperty(),
			"views": types.NewIntegerNumberProperty(),
		}},
	}).Do(ctx)
	if err != nil {
		log.Println("create index:", err)
	}

	// Index one document and make it searchable at once (for tests; production relies on the refresh interval).
	_, err = es.Index("posts").Id("1").Request(Post{Title: "Finding a goroutine leak", Body: "Use a pprof goroutine profile.", Tags: []string{"concurrency"}, Views: 10}).Refresh(refresh.True).Do(ctx)
	if err != nil {
		log.Fatal(err)
	}

	// A bool query: full text in must, exact filters in filter.
	res, err := es.Search().Index("posts").Request(&search.Request{
		Query: &types.Query{
			Bool: &types.BoolQuery{
				Must:   []types.Query{{Match: map[string]types.MatchQuery{"title": {Query: "goroutine leak"}}}},
				Filter: []types.Query{{Term: map[string]types.TermQuery{"tags": {Value: "concurrency"}}}},
			},
		},
		Size: someInt(10),
	}).Do(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("hits:", res.Hits.Total.Value)
	for _, h := range res.Hits.Hits {
		fmt.Println(*h.Id_, h.Score_, string(h.Source_))
	}

	// Bulk indexing: one request, many documents, and a result for each of them.
	bulk := es.Bulk().Index("posts")
	for i, p := range []Post{{Title: "Context timeouts"}, {Title: "Worker pools"}} {
		id := fmt.Sprint(100 + i)
		if err := bulk.IndexOp(types.IndexOperation{Id_: &id}, p); err != nil {
			log.Fatal(err)
		}
	}
	out, err := bulk.Do(ctx)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("bulk errors:", out.Errors)
}

func someInt(n int) *int { return &n }

elasticsearch.NewTypedClient returns a client whose methods build requests from Go structs and return typed responses, which is easier to get right than raw JSON. Alongside it the untyped NewClient takes raw bodies and is still useful for APIs the typed one lacks. Configure addresses, authentication (an API key or a username and password), TLS and timeouts through elasticsearch.Config. The client is safe for concurrent use and keeps connections alive, so create one per process.

Every request ends in .Do(ctx), which takes a context, so deadlines and cancellation work as usual. Errors come back as a *types.ElasticsearchError for a server rejection, which carries the status and the reason, and as a normal error for a network failure. Check the status: a missing index is a 404 and a version conflict is a 409, and they need different handling.

Mappings: decide before you index

A mapping tells Elasticsearch how to treat each field. The program creates the posts index with an explicit mapping: title and body as text, tags as keyword and views as an integer. The distinction between the first two is the most important idea in the product.

• A text field is run through an analyser: it is split into tokens, lower cased, and (for the standard analyser) split on punctuation, and each token goes into the inverted index. A match query on a text field analyses the query the same way, so "Goroutine Leaks" finds a title containing "goroutine leak". Text fields support relevance scoring and highlighting, and they cannot be used for sorting or exact aggregations.
• A keyword field is stored as one unanalysed term. It matches only exact values, and it is what you use for filters (status = "paid"), sorts and aggregations (counts per tag).

Elasticsearch can guess mappings from the first document it sees (dynamic mapping), and a guess is wrong often enough to hurt: a field that looks like a number in the first document and is a word in the second is rejected, and a string field becomes both text and a keyword sub field by default, doubling storage. Define the mapping when you create the index, and set dynamic to strict once the shape is settled, so that a new field is an error and not a silent addition. A mapping cannot be changed in place for existing fields: changing a field's type means creating a new index and reindexing into it, so treat the index name as versioned (posts-v3) and point an alias (posts) at the live one, so you can swap it without changing code.

Indexing and the refresh delay

Index("posts").Id("1").Request(doc) stores a document, and the program passes Refresh(refresh.True) to make it visible. By default an indexed document is not searchable instantly. Elasticsearch is near real time: new documents go into a segment that becomes visible to search at the next refresh, once a second by default. A search issued right after an index call can miss the document. For tests, request a refresh as the program does, or wait for it. In production, do not refresh on every write, since it is expensive and defeats batching; accept the one second delay and design the user interface around it, for instance by showing the new item from your database immediately and letting it appear in search a moment later.

Querying: bool, must and filter

The query in the program is a bool query with two parts, and understanding the split is worth the time:

• must clauses contribute to the relevance score. The match query on the title with goroutine leak scores documents by how well they match, using BM25.
• filter clauses only include or exclude documents, with no score. The term on tags for concurrency is a filter. Filters are cached and cheaper than scoring clauses, so put everything that is a yes or no condition (status, tenant, date range, tags) in filter, and only the free text in must.

Others you will use: should for optional boosts, must_not for exclusions, multi_match to search several fields with weights (title^3), match_phrase for exact phrases, range for numbers and dates, and fuzziness on match for typo tolerance. Responses carry Hits.Total, each hit's Score_ and Source_ as raw JSON, which you unmarshal into your struct. Ask for only the fields you need with a _source filter, and set Size.

Keep tenant or permission filters in every query by construction, in one function that builds the base query, as with multi tenant tables: a search index has no row level security unless you configure document level security in your edition.

Relevance, analysers and why results surprise people

Search quality is the reason to use the engine, and it needs more attention than the Go code. Relevance is computed by BM25, which scores a document higher when the query terms are frequent in it, rare across the index and found in a shorter field. That produces results that are correct by the formula and wrong to a user: a long article that mentions a term ten times can lose to a short one that mentions it twice. Tools to steer it: field boosts in multi_match (title above body), a function_score that adds popularity or recency, and the explain API, which shows how a score was computed for a given document.

Analysers decide what counts as a match. The standard analyser lowercases and splits on word boundaries, so http.NewRequest becomes three tokens and a search for NewRequest alone matches it. For code and identifiers, define a custom analyser, with a pattern tokenizer or a word delimiter filter, and test it with the _analyze API before indexing a million documents with it. For language specific search, the language analysers add stemming (leaking and leaks match leak) and stop words. Synonyms, edge n-grams for as you type search and a phonetic filter are all analyser configuration, defined in the index settings, and changing them means a reindex.

Test search quality with a fixed set of queries and the results you expect, kept in the repository, and run it against a small real index in CI. A change to a mapping or an analyser that improves one query often breaks another, and the only way to know is a regression set. Without one, relevance tuning is guessing, and every change feels like an improvement to the person who made it.

Aggregations

The other half of the product is aggregations: counting and summarising documents in the same request as a search. A terms aggregation on a keyword field returns the top tags with their counts, which is what a facet sidebar needs ("concurrency (42), testing (31)"). A date_histogram buckets by day, a range aggregation buckets prices, and metric aggregations compute averages and percentiles. They run on keyword, numeric and date fields, which is why those fields need doc_values (the default) and why a text field cannot be aggregated. In the typed client, aggregations are set on the search.Request next to Query, and read from res.Aggregations. Counts in terms aggregations are approximate when the data is spread across many shards, with an error bound that the response reports, so do not use them as accounting figures.

Bulk indexing

Indexing one document per request is slow. The bulk API sends many operations in one request, and the program's last block builds one with es.Bulk().Index("posts"), adding index operations with IndexOp and calling Do. The response has an overall Errors flag and a result for each item: a bulk request returns 200 even when some items failed, so you must inspect the items, retry the ones that failed with a 429 (the cluster is busy), and log the others. Sizes of a few megabytes and a few thousand documents per request are a sensible start, tuned by measurement. The library also provides a BulkIndexer helper in esutil that batches, runs workers and reports failures per item.

Paging deeply

from and size work for the first few pages, and Elasticsearch refuses to go past from + size = 10,000 by default (index.max_result_window), because every shard has to build and sort that many hits. For deeper paging, such as an export, use search_after: sort by a stable key, include a tiebreaker such as _shard_doc with a point in time (PIT), and pass the last hit's sort values to the next request. A PIT keeps a consistent view of the index while you page. Open it, page with search_after, close it. The idea is the cursor pagination of keyset pagination applied to a search index.

Keeping it in step with the database

The index is a copy, so the hard part is keeping it current. Three approaches, in the order I would try them:

1. An outbox and a worker. Writes to your database also write a change event in the same transaction; a worker reads the events and indexes or deletes the document, retrying until it works, as in the outbox pattern. It tolerates search being down and gives at least once delivery, which indexing handles well because indexing the same document twice with the same id is harmless.
2. Change data capture from the database log into the index, if you already run that infrastructure.
3. Dual writes from the application, which fail halfway and drift. Avoid them.

Whatever you choose, build a reindex job from the start: iterate the source table in key order, bulk index into a new versioned index, then flip the alias. You will need it for every mapping change and every bug that corrupts the index. Use the document's database id as the Elasticsearch _id, so replays overwrite and do not duplicate, and carry a version (an updated_at or a counter) with external versioning so that an old event cannot overwrite a newer document.

Operating it from the application side

• Never make search a hard dependency of the main request. If the cluster is down, show a degraded result or an error for search only. Put a short timeout on every call.
• Handle 429 and 503 with backoff; the client retries some failures by default and you should configure the maximum.
• Size shards deliberately. Many tiny shards waste memory, and one giant shard cannot be spread. The number of primary shards is fixed at index creation; tens of gigabytes per shard is the usual guidance in the documentation.
• Do not store the only copy there. A cluster can lose data to a bad deployment, a mapping mistake or a missed snapshot, and your recovery is a rebuild from the database.
• Watch the cluster health: status colour, unassigned shards, heap usage and rejected requests in the thread pools.
• Version the client with the server, and test upgrades, since query behaviour changes between majors.

When not to use it

Do not use a search cluster to answer queries a database answers well: lookups by id, exact filters over small tables and reports over transactional data. For a few hundred thousand documents and simple relevance, PostgreSQL's built in search is cheaper to run and has one fewer system to keep consistent. Add Elasticsearch when you need relevance tuning, fuzzy matching, facets over large data, or search load that should not touch your primary database. The Go client is documented in the Elastic Go client reference and on pkg.go.dev, and the query language in the Query DSL guide.