{"id":4582,"date":"2014-02-17T11:53:36","date_gmt":"2014-02-17T03:53:36","guid":{"rendered":"http:\/\/maziyar.me\/?p=4582"},"modified":"2015-11-16T11:20:10","modified_gmt":"2015-11-16T09:20:10","slug":"building-search-engine-by-elasticsearch","status":"publish","type":"post","link":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/","title":{"rendered":"Building Search Engine by ElasticSearch"},"content":{"rendered":"

I’ve started to work with ElasticSearch for a while now. I gotta say it’s a powerful open source for building distributed real-time search engines and analytics engines.<\/p>\n

It also uses Shards and Replica in distributed machines to make your architecture reliable and scalable. It’s not just a simple full-text search engine even though it does that perfectly.<\/p>\n

What I do with ElasticSearch is that it’s connected to my MongoDB replica set and it indexes tweets as they are streamed by Twitter API. I track tweets for few projects (health, news, UN GlobalPulse, etc.) and try to index some of my projects in ES on the fly.<\/p>\n

<\/p>\n

I have indexed over 14.86 million documents in ElasticSearch and it has worked really smooth so far. It’s not that much yet but the fact that I have my primary database (MongoDB) and a fully-function search engine (ElasticSearch) and they both are working together and compatible is just fantastic. I have 20 to 30 updates per second and so many deletes (TTL documents in MongoDB) as well. Even those operations have worked without any problem for the past two weeks.<\/p>\n

Here is my architecture: (it’s just for development and experiment)<\/p>\n

Large EC2 instance (m1.Large 7.5G Memory) for ElasticSearch. The CPU usage is around 15%-25% and this instance has also few Node.js apps up and running 24\/7 as well.<\/p>\n

Two EC2 instances (m2.2xLarge 34GB Memory). This is connected to ElasticSearch by a plugin named elasticsearch-river-mongodb<\/a>. My ES version is 0.90.11 on Ubuntu 12.04 and MongoDB version is 2.4.9.<\/p>\n

\u00a0Btw, Marvel<\/a> is great dashboard for monitoring your ElasticSearch cluster: (the cpu usage is high because I just connected one of my MongoDB collections with 16 million documents and as it can be seen in the figure, it is indexing 300 to 400 documents per second)<\/p>\n

\"\"<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve started to work with ElasticSearch for a while now. I gotta say it’s a powerful open source for building distributed real-time search engines and analytics engines. It also uses Shards and Replica in distributed machines to make your architecture reliable and scalable. It’s not just a simple full-text search engine even though it does … <\/p>\n

Continue reading “Building Search Engine by ElasticSearch”<\/span><\/a><\/p>\n","protected":false},"author":65,"featured_media":5085,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[7],"tags":[12,18,19,26,36,38,43,46],"class_list":["post-4582","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-geeky","tag-aws","tag-ec2","tag-elasticsearch","tag-information-retrieving","tag-mongodb","tag-node-js","tag-search-engine","tag-twitter"],"yoast_head":"\nBuilding Search Engine by ElasticSearch<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building Search Engine by ElasticSearch\" \/>\n<meta property=\"og:description\" content=\"I’ve started to work with ElasticSearch for a while now. I gotta say it’s a powerful open source for building distributed real-time search engines and analytics engines. It also uses Shards and Replica in distributed machines to make your architecture reliable and scalable. It’s not just a simple full-text search engine even though it does … Continue reading "Building Search Engine by ElasticSearch"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/\" \/>\n<meta property=\"og:site_name\" content=\"Maziyar PANAHI\" \/>\n<meta property=\"article:published_time\" content=\"2014-02-17T03:53:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-11-16T09:20:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2840\" \/>\n\t<meta property=\"og:image:height\" content=\"1218\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Maziyar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Maziyar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/\"},\"author\":{\"name\":\"Maziyar\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\"},\"headline\":\"Building Search Engine by ElasticSearch\",\"datePublished\":\"2014-02-17T03:53:36+00:00\",\"dateModified\":\"2015-11-16T09:20:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/\"},\"wordCount\":297,\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2015\\\/06\\\/Marvel.png\",\"keywords\":[\"AWS\",\"EC2\",\"ElasticSearch\",\"Information Retrieving\",\"MongoDB\",\"Node.js\",\"Search Engine\",\"twitter\"],\"articleSection\":[\"Geeky\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/\",\"name\":\"Building Search Engine by ElasticSearch\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2015\\\/06\\\/Marvel.png\",\"datePublished\":\"2014-02-17T03:53:36+00:00\",\"dateModified\":\"2015-11-16T09:20:10+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#primaryimage\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2015\\\/06\\\/Marvel.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2015\\\/06\\\/Marvel.png\",\"width\":2840,\"height\":1218},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/2014\\\/02\\\/17\\\/building-search-engine-by-elasticsearch\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Building Search Engine by ElasticSearch\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/#website\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/\",\"name\":\"Maziyar PANAHI\",\"description\":\"Generative AI Leader in Healthcare\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\",\"name\":\"Maziyar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g\",\"caption\":\"Maziyar\"},\"description\":\"Big Data engineer\",\"sameAs\":[\"https:\\\/\\\/multivac.iscpif.fr\"],\"url\":\"https:\\\/\\\/iscpif.fr\\\/maziyar\\\/author\\\/maziyar-panahi\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building Search Engine by ElasticSearch","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/","og_locale":"en_US","og_type":"article","og_title":"Building Search Engine by ElasticSearch","og_description":"I’ve started to work with ElasticSearch for a while now. I gotta say it’s a powerful open source for building distributed real-time search engines and analytics engines. It also uses Shards and Replica in distributed machines to make your architecture reliable and scalable. It’s not just a simple full-text search engine even though it does … Continue reading \"Building Search Engine by ElasticSearch\"","og_url":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/","og_site_name":"Maziyar PANAHI","article_published_time":"2014-02-17T03:53:36+00:00","article_modified_time":"2015-11-16T09:20:10+00:00","og_image":[{"width":2840,"height":1218,"url":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","type":"image\/png"}],"author":"Maziyar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Maziyar","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#article","isPartOf":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/"},"author":{"name":"Maziyar","@id":"https:\/\/iscpif.fr\/maziyar\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf"},"headline":"Building Search Engine by ElasticSearch","datePublished":"2014-02-17T03:53:36+00:00","dateModified":"2015-11-16T09:20:10+00:00","mainEntityOfPage":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/"},"wordCount":297,"image":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","keywords":["AWS","EC2","ElasticSearch","Information Retrieving","MongoDB","Node.js","Search Engine","twitter"],"articleSection":["Geeky"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/","url":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/","name":"Building Search Engine by ElasticSearch","isPartOf":{"@id":"https:\/\/iscpif.fr\/maziyar\/#website"},"primaryImageOfPage":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#primaryimage"},"image":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","datePublished":"2014-02-17T03:53:36+00:00","dateModified":"2015-11-16T09:20:10+00:00","author":{"@id":"https:\/\/iscpif.fr\/maziyar\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf"},"breadcrumb":{"@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#primaryimage","url":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","contentUrl":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","width":2840,"height":1218},{"@type":"BreadcrumbList","@id":"https:\/\/iscpif.fr\/maziyar\/2014\/02\/17\/building-search-engine-by-elasticsearch\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/iscpif.fr\/maziyar\/"},{"@type":"ListItem","position":2,"name":"Building Search Engine by ElasticSearch"}]},{"@type":"WebSite","@id":"https:\/\/iscpif.fr\/maziyar\/#website","url":"https:\/\/iscpif.fr\/maziyar\/","name":"Maziyar PANAHI","description":"Generative AI Leader in Healthcare","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/iscpif.fr\/maziyar\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/iscpif.fr\/maziyar\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf","name":"Maziyar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/286798afc2f09c77b0ab1a0a60150f989d903d23d84c3eedfb8ebd1da500e5b1?s=96&d=mm&r=g","caption":"Maziyar"},"description":"Big Data engineer","sameAs":["https:\/\/multivac.iscpif.fr"],"url":"https:\/\/iscpif.fr\/maziyar\/author\/maziyar-panahi\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Marvel.png","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p6jMg3-1bU","jetpack-related-posts":[{"id":5231,"url":"https:\/\/iscpif.fr\/maziyar\/2015\/11\/28\/big-data-what-i-am-thankful-this-thanksgiving\/","url_meta":{"origin":4582,"position":0},"title":"Big Data: What I am thankful this Thanksgiving!","author":"Maziyar","date":"November 28, 2015","format":false,"excerpt":"Big Data platform at ISCPIF: This list represents what I am thankful to\u00a0this Thanksgiving Elasticsearch You know for search! This is the heart and the soul of our discovery and curiosity. The first step of every data analytics is finding the right data. Behind every great data discovery there must\u2026","rel":"","context":"In "iscpif"","block_context":{"text":"iscpif","link":"https:\/\/iscpif.fr\/maziyar\/category\/iscpif\/"},"img":{"alt_text":"big data at iscpif","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/11\/header.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":5154,"url":"https:\/\/iscpif.fr\/maziyar\/2015\/07\/01\/indexing-200m-tweets-into-our-search-engine\/","url_meta":{"origin":4582,"position":1},"title":"Indexing 200m tweets into our search engine cluster","author":"Maziyar","date":"July 1, 2015","format":false,"excerpt":"Indexing 200m tweets into our search engine cluster from Maziyar on Vimeo.","rel":"","context":"In "Geeky"","block_context":{"text":"Geeky","link":"https:\/\/iscpif.fr\/maziyar\/category\/geeky\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/07\/Processes.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":5278,"url":"https:\/\/iscpif.fr\/maziyar\/2015\/12\/03\/big-data-how-to-scale-from-zero-to-billions\/","url_meta":{"origin":4582,"position":2},"title":"Big Data: HOW TO SCALE FROM ZERO TO BILLIONS!","author":"Maziyar","date":"December 3, 2015","format":false,"excerpt":"How Big Data platform scaled from zero to billions of data within 6 months at ISCPIF (CNRS). This talk contains our use of Elasticsearch, MongoDB, Redis, RabbitMQ and scalable\/high available Web services built over Big Data architecture. This presentation was presented at Universit\u00e9 Paris-Sud, LAL, B\u00e2timent 200 organized by ARGOS.\u2026","rel":"","context":"In "iscpif"","block_context":{"text":"iscpif","link":"https:\/\/iscpif.fr\/maziyar\/category\/iscpif\/"},"img":{"alt_text":"real-time data analytics","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/12\/Screen-Shot-2015-12-03-at-12.23.21.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":4594,"url":"https:\/\/iscpif.fr\/maziyar\/2014\/03\/11\/some-of-the-stats-during-oscars-2014\/","url_meta":{"origin":4582,"position":3},"title":"Some of the stats during Oscars 2014","author":"Maziyar","date":"March 11, 2014","format":false,"excerpt":"","rel":"","context":"In "Geeky"","block_context":{"text":"Geeky","link":"https:\/\/iscpif.fr\/maziyar\/category\/geeky\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/minutes.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":4616,"url":"https:\/\/iscpif.fr\/maziyar\/2014\/07\/25\/building-a-real-time-news-analytics\/","url_meta":{"origin":4582,"position":4},"title":"Building a real-time news analytics","author":"Maziyar","date":"July 25, 2014","format":false,"excerpt":"It is always interesting to find out what exactly is happening around the world as they are happening. This feature of knowing something as it happens is so called \"realtime\" or \"near realtime\" because of the network latency, delay for processing the data, delay for visualizing and streaming the data,\u2026","rel":"","context":"In "Geeky"","block_context":{"text":"Geeky","link":"https:\/\/iscpif.fr\/maziyar\/category\/geeky\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/Screenshot-2014-06-12-17.45.22.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":4693,"url":"https:\/\/iscpif.fr\/maziyar\/2015\/05\/29\/lets-build-this-thing\/","url_meta":{"origin":4582,"position":5},"title":"Let’s build this thing!","author":"Maziyar","date":"May 29, 2015","format":false,"excerpt":"","rel":"","context":"In "Daily"","block_context":{"text":"Daily","link":"https:\/\/iscpif.fr\/maziyar\/category\/daily\/"},"img":{"alt_text":"IMG_0498","src":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/IMG_0498-640x640.jpg?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/IMG_0498-640x640.jpg?resize=350%2C200 1x, https:\/\/i0.wp.com\/iscpif.fr\/maziyar\/wp-content\/uploads\/sites\/5\/2015\/06\/IMG_0498-640x640.jpg?resize=525%2C300 1.5x"},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/posts\/4582","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/comments?post=4582"}],"version-history":[{"count":1,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/posts\/4582\/revisions"}],"predecessor-version":[{"id":5147,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/posts\/4582\/revisions\/5147"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/media\/5085"}],"wp:attachment":[{"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/media?parent=4582"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/categories?post=4582"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/iscpif.fr\/maziyar\/wp-json\/wp\/v2\/tags?post=4582"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}