Month: January 2017

Object/System Design and Algorithms

  1. Design a meeting scheduler –
  2. How to create a min heap from an array –
  3. Inserting into a Heap –

4. Jump Search –

5.  Bubble Sort –

6.  Selection Sort –

7.  Merge Sort –

8.  Insertion Sort –

9.  Quick Sort –

10.  Counting Sort –







System Design Interview Questions Links


Elevator Design :

Log Management with ELK


ElasticSearch general considerations:

  1. _source field : its good to store it; would help when we need to re-index

2. Dynamic mappings support: think twice before giving support for querying on new dynamic fields.

3. Should you care about field data?

Elastic Search Mappings

Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. For instance, use mappings to define:

  • which string fields should be treated as full text fields.
  • which fields contain numbers, dates, or geolocations.
  • whether the values of all fields in the document should be indexed into the catch-all _allfield.
  • the format of date values.
  • custom rules to control the mapping for dynamically added fields.

Scalability considerations

3. On the face of it, though a search query is parallelized across many shards, it involves a lot of overhead in terms of a co-ordinating node talking to all the other nodes and multiple network hops which might impact performance.

4. Once the number of shards for an index is chosen, they cannot be changed. So, if the index cannot be fit into the designated number of shards, the whole index would have to be re-indexed

In some systems like log management (using logstash), one way of avoiding 3 and 4 is by having index per day. In these cases, when we need to search logs for the last x days, we can only restrict the query to those nodes containing the shards of those indexes.

In some systems where we need to do user specific search, we can have an index per user. However, if the number of users is more, even this becomes a bottleneck. So, we can have a shared index but use the routing feature to route user specific data to specific shards. Also, when the index outgrows the shard, only that shard that holds the index data for the specific user can be re-indexed

  1. Can documents/access be partitioned in a natural way
  2. Need to find documents by id (update/delete/get etc)
  3. know the relevant features (Routing, aliases, multi-index search)
  4. Measure the impact of distributed search
  5. Indices do not come for free
  6. Care about field data?

Donts for elastic search

  1. Dont create more shards than you need. More shards enable larger indices and can scale out operations on individual documents. If unsure, overallocate by a little not by a factor or 3 or 4. Evaluate the number of shards required
  2. Dont treat all nodes as equal (master, data, client/aggregator nodes etc). Have dedicated master nodes. Data nodes are used for querying and where the actual search happens (loading index data into memory etc). Also try to distinguish between data nodes and client nodes
  3. Dont run wasteful queries – Avoid deep pagination queries, Use scan + scroll instead of sorting. Only query indices/shards that may contain hits

ElasticSearch re-indexing

Re-indexing would be required if

1. Mapping changes

2. Index/shard reaches its capacity

3. Reduce/Increase number of shards

Factors to consider for re-indexing

  1. Where is the data that needs to be re-indexed? Is it in an external data source or the _source field in elastic search?
  2. Is it ok to have down time?
  3. Update API usage
  4. Deletes?
  5. Disable refresh and decrease number of replicas

ELK Architecture



Logstash (DataFlow Engine) Key features

  1. Open-source central processing engine for data logistics
  2. Construct dataflow pipelines to transform events and route streams
  3. Data source agnostic
  4. Horizontally scalable with native buffering out of the box
  5. Robust plugin ecosystem for integrations and processing

More articles on elastic stack


Elastic Search


  1. Logstash pipeline configuration (is there support for DAG pipelines?)