Big Data: What I am thankful this Thanksgiving!

Big Data platform at ISCPIF: This list represents what I am thankful to this Thanksgiving

• Elasticsearch You know for search!

This is the heart and the soul of our discovery and curiosity. The first step of every data analytics is finding the right data. Behind every great data discovery there must be a great search engine 🙂 And this is where Elasticsearch comes to the rescue with all its advanced real-time data analytics and powerful full-text search abilities built over Apache Lucene.
ISCPIF Big Data platform uses Elasticsearch and its built-in distributed and high availability feature with over 900 million documents indexed in order to facilitate data discoveries and explorations for our researchers and scientific partners.

So I am thankful to Elasticsearch for the past 2 years, which made my life much easier and made me a better engineer.

Say hello to iscpif Elasticsearch 2.1 cluster:

{
  "name" : "iscpif-es-4",
  "cluster_name" : "iscpif-es",
  "version" : {
    "number" : "2.1.0",    
    "build_timestamp" : "2015-11-18T22:40:03Z",
    "lucene_version" : "5.3.1"
  },
  "tagline" : "You Know, for Search"
}

• Kibana Explore & Visualize Your Data!

Man I wish I could say data means something without visualization, but I’m afraid it just doesn’t! Not much to be honest. Everyone needs an interactive dashboard of endless visualizations, charts, maps, etc. that just works right of the box. Kibana obviously plugs perfectly and easily into Elasticsearch and it just takes your data to a whole new level.

I am so happy that I upgraded the entire cluster into version 2.1 so I could upgrade the Kibana to 4.3 as well 🙂

• MongoDB Launch your GIANT idea!

This has been always the backbone of every one of my architectures that I’ve ever designed and implemented. No matter what happens to the rest, I always know this preserves all my incoming data at any cost. Sure some ups and downs once in a while along the road  (some my own faults, some maybe bugs), but which technology doesn’t have its own stuff anyway.

At ISCPIF, we have over 360K documents per hour and 8M every day. Also some of our scientific computations generate over millions of records and we are thankful for new MongoDB 130K inserts/s and thousands of queries and aggregations per second. Without a doubt I have built such great tools in the verity of topics over the past 3 years by the help of MongoDB. Whether it’s a Web-based application or a mobile application I know MongoDB just plays nicely with the rest of my architecture pieces and provides everything I need from a database.

In a way MongoDB was my first NoSQL experience. Like they say, you won’t forget your first but it made me forget all about the relational databases with all its charm and easy going nature 🙂 I am thankful for that for sure!

Also thankful for its pluggable storage and WiredTiger. The compression makes the life whole a lot easier. I almost save 50% of my storage just by using WiredTiger default compression. We are all proud for hitting our first 3 billion documents by the way.

 

 

 

At the end this Thanksgiving, we are thankful to the entire Open Source community and all its contributors who work hard days and nights to build such great tools.

Open Source is not just free, it’s just amazing!

This list will be updated soon with MongoDB, NodeJS, Redis, RabbitMQ, etc. 😉

https://www.elastic.co/products/elasticsearch

https://www.elastic.co/products/kibana

https://www.mongodb.org/

http://redis.io/

https://www.rabbitmq.com/

https://nodejs.org/en/