Big Data: What I am thankful this Thanksgiving!

MULTIVAC Platform at ISCPIF – This list represents what I am thankful to this Thanksgiving

Elasticsearch You know for search!

This is the heart and the soul of our discovery and curiosity. The first step of every data analytics is finding the right data. Behind every great data discovery there must be a great search engine 🙂 And this is where Elasticsearch comes to the rescue with all its advanced real-time data analytics and powerful full-text search abilities built over Apache Lucene.

MULTIVAC Platform uses Elasticsearch and its built-in distributed and high availability feature with over 900 million documents indexed in order to facilitate data discoveries and explorations for our researchers and scientific partners.

So I am thankful to Elasticsearch for the past 2 years, which made my life much easier and made me a better engineer.

Say hello to iscpif Elasticsearch 2.1 cluster:

{
  "name" : "iscpif-es-4",
  "cluster_name" : "iscpif-es",
  "version" : {
    "number" : "2.1.0",    
    "build_timestamp" : "2015-11-18T22:40:03Z",
    "lucene_version" : "5.3.1"
  },
  "tagline" : "You Know, for Search"
}

Kibana Explore & Visualize Your Data!

Man I wish I could say data means something without visualization, but I’m afraid it just doesn’t! Not much to be honest. Everyone needs an interactive dashboard of endless visualizations, charts, maps, etc. that just works right of the box. Kibana obviously plugs perfectly and easily into Elasticsearch and it just takes your data to a whole new level.

Kibana at ISCPIF

I am so happy that I upgraded the entire cluster into version 2.1 so I could upgrade the Kibana to 4.3 as well 🙂

MongoDB Launch your GIANT idea!

This has been always the backbone of every one of my architectures that I’ve ever designed and implemented. No matter what happens to the rest, I always know this preserves all my incoming data at any cost. Sure some ups and downs once in a while along the road (some my own faults, some maybe bugs), but which technology doesn’t have its own stuff anyway.

At ISCPIF, we have over 360K documents per hour and over 8M documents every day. Also some of our scientific computations generate over millions of records and we are thankful for new MongoDB 130K inserts/s and hundred thousands of queries and aggregations per second. Without a doubt I have built such great tools in the verity of topics since 2012 by the help of MongoDB. Whether it’s a Web-based application or a mobile application I know MongoDB just plays nicely with the rest of my architecture and provides everything I need from a database.

In a way MongoDB was my first NoSQL experience. Like they say, you won’t forget your first but it made me forget all about the relational databases with all its charms and easy going nature 🙂 I am thankful for that one for sure!

I am also thankful for its new pluggable storage feature (WiredTiger). The compression makes the life whole a lot easier. I almost save 50% of my storage just by using WiredTiger default compression. We are all proud for hitting our first 3 billion documents by the way.