{"id":220,"date":"2017-09-24T14:53:44","date_gmt":"2017-09-24T13:53:44","guid":{"rendered":"https:\/\/iscpif.fr\/bigdata\/?p=220"},"modified":"2017-10-14T10:53:58","modified_gmt":"2017-10-14T09:53:58","slug":"multivac-simple-twitter-analytics-part-two","status":"publish","type":"post","link":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/","title":{"rendered":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II"},"content":{"rendered":"<p>In the <a href=\"https:\/\/iscpif.fr\/bigdata\/2017\/09\/simple-twitter-analytics-by-multivac-data-science-lab\/\" target=\"_blank\" rel=\"noopener\">first part<\/a> we saw some simple analytics and text extraction over source of each tweets from my Twitter archive. We worked with both Spark Scala APIs and Spark SQL in Apache Zeppelin notebooks hosted by <a href=\"https:\/\/multivac.iscpif.fr\" target=\"_blank\" rel=\"noopener\">Multivac<\/a>.<\/p>\n<p>In part II we are going to do the followings:<\/p>\n<ul>\n<li>Extracting hashtags from content of each tweets<\/li>\n<li>Make some visualisations with the help of Spark SQL<\/li>\n<li>Extracting accounts that I have mentioned during last 7 years<\/li>\n<li>We see my original tweets vs retweeted tweets evolution<\/li>\n<\/ul>\n<p>As you remember in Part I, we used tweets.csv file which comes with very little metadata. Usually, tweet&#8217;s JSON has lots of fields such as <strong>Entities\u00a0<\/strong>that contains hashtags, mentions, URLs, media, etc. But since we don&#8217;t have this field in our minimal tweets.csv file we are going to extract hashtags from my tweets content.<\/p>\n<p>Last time for removing HTML tags we used Spark UDF, but this time we are going to use a function that already exists in Spark MlLib<strong>.\u00a0<\/strong>We are going to use a function called &#8220;<a href=\"https:\/\/spark.apache.org\/docs\/latest\/ml-features.html#tokenizer\" target=\"_blank\" rel=\"noopener\">RegexTokenizer<\/a>&#8220;:<\/p>\n<blockquote><p><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/scala\/index.html#org.apache.spark.ml.feature.RegexTokenizer\">RegexTokenizer<\/a>\u00a0allows more advanced tokenization based on regular expression (regex) matching. By default, the parameter \u201cpattern\u201d (regex, default:\u00a0<code>\"\\\\s+\"<\/code>) is used as delimiters to split the input text. Alternatively, users can set parameter \u201cgaps\u201d to false indicating the regex \u201cpattern\u201d denotes \u201ctokens\u201d rather than splitting gaps, and find all matching occurrences as the tokenization result.<\/p><\/blockquote>\n<p>Here is how we use it to extract Hashtags from the content of my Tweets:<\/p>\n<pre class=\"prettyprint\">import org.apache.spark.ml.feature.RegexTokenizer\r\n\r\nval hashtagsDF = new RegexTokenizer()\r\n.setGaps(false)\r\n.setPattern(\"#(\\\\w+)\")\r\n.setMinTokenLength(2)\r\n.setInputCol(\"text\")\r\n.setOutputCol(\"hashtags\")\r\n.transform(newTweetDF)<\/pre>\n<p><!--more--><\/p>\n<p>A little of explanation:<\/p>\n<ul>\n<li><strong>hashtagsDF<\/strong>: Is going to be a DataFrame with a new column called &#8220;hashtags&#8221; that contains the results of our\u00a0RegexTokenizer<\/li>\n<li><strong>setPattern<\/strong>: This is where we set our regex pattern<\/li>\n<li><strong>setInputCol<\/strong>: It&#8217;s where we get the input which is the &#8220;text&#8221; of my tweets<\/li>\n<li><strong>setOutputCol<\/strong>: What should be the name of our output column<\/li>\n<li><strong>transform<\/strong>:\u00a0By Apache Spark definition &#8220;Scaling, converting, or modifying features&#8221;. That&#8217;s why we pass &#8220;newTweetDF&#8221; which is our Tweets DataFrame.<\/li>\n<\/ul>\n<p>Let&#8217;s take a look at our new DataFrame and its schema<\/p>\n<pre class=\"prettyprint\">hashtagsDF.select(\"hashtags\").show(false)\r\n+---------------------------------------------------------------------------------------+\r\n|hashtags                                                                               |\r\n+---------------------------------------------------------------------------------------+\r\n|[]                                                                                     |\r\n|[#bigdata, #ml]                                                                        |\r\n|[]                                                                                     |\r\n|[#bigda]                                                                               |\r\n|[]                                                                                     |\r\n|[]                                                                                     |\r\n|[]                                                                                     |\r\n|[]                                                                                     |\r\n|[]                                                                                     |\r\n|[#legislatives2017]                                                                    |\r\n|[#journalism, #journalists, #fakenews, #legislatives, #politoscope]                    |\r\n|[#macron, #gouvernementmacron, #gouvernementphilippe, #ministres, #politoscope]        |\r\n|[#innovativesshs, #cnrsinnovation]                                                     |\r\n|[#legislatives2017, #macron]                                                           |\r\n|[#premierministre, #edouardphilippe, #macron, #legislatives2017, #politoscope]         |\r\n|[#premierministre, #edouardphilippe, #macronpresident, #legislatives2017, #politoscope]|\r\n|[#premierministre, #edouardphilippe, #macronpresident, #legislatives2017]              |\r\n|[]                                                                                     |\r\n|[#innovativesshs, #cnrsinnovation]                                                     |\r\n|[]                                                                                     |\r\n+---------------------------------------------------------------------------------------+\r\nonly showing top 20 rows\r\n<\/pre>\n<pre class=\"prettyprint\">hashtagsDF.printSchema\r\nroot\r\n |-- tweet_id: string (nullable = true)\r\n |-- in_reply_to_status_id: string (nullable = true)\r\n |-- in_reply_to_user_id: string (nullable = true)\r\n |-- timestamp: string (nullable = true)\r\n |-- source: string (nullable = true)\r\n |-- text: string (nullable = true)\r\n |-- retweeted_status_id: string (nullable = true)\r\n |-- retweeted_status_user_id: string (nullable = true)\r\n |-- retweeted_status_timestamp: string (nullable = true)\r\n |-- expanded_urls: string (nullable = true)\r\n |-- cleanSource: string (nullable = true)\r\n |-- hashtags: array (nullable = true)\r\n |    |-- element: string (containsNull = true)\r\n<\/pre>\n<p>OK! So the the column &#8220;hashtags&#8221; is an array pf strings and it can be also empty as I\u00a0 may have not mentioned any hashtag in my tweet.<\/p>\n<p>How many unique hashtags I have used and what are the top 20s:<\/p>\n<pre class=\"prettyprint\">val hastagsFreq = hashtagsDF.select(explode($\"hashtags\").as(\"value\")).groupBy(\"value\").count\r\nhastagsFreq.count()\r\n\r\nres15: Long = 747<\/pre>\n<pre class=\"prettyprint\">hastagsFreq.sort($\"count\".desc).show(20, false)\r\n\r\n+-------------------+-----+\r\n|value              |count|\r\n+-------------------+-----+\r\n|#bigdata           |76   |\r\n|#aws               |73   |\r\n|#politoscope       |62   |\r\n|#mongodb           |53   |\r\n|#mwc14             |42   |\r\n|#presidentielle2017|38   |\r\n|#nodejs            |36   |\r\n|#ec2               |36   |\r\n|#nowplaying        |34   |\r\n|#nowwatching       |28   |\r\n|#eccs13            |27   |\r\n|#macron            |21   |\r\n|#hadoop            |18   |\r\n|#ios               |18   |\r\n|#elasticsearch     |16   |\r\n|#mahmeri           |16   |\r\n|#redis             |16   |\r\n|#analytics         |16   |\r\n|#paris             |15   |\r\n|#terradata         |11   |\r\n+-------------------+-----+\r\nonly showing top 20 rows<\/pre>\n<p>Now let&#8217;s create a temporary view to run some SQL queries for visualisations:<\/p>\n<pre class=\"prettyprint\">hashtagsDF.select(explode($\"hashtags\").as(\"hashtag\")).createOrReplaceTempView(\"MyHashtags\")\r\n\r\n\/\/The SQL block\r\n%sql\r\nSELECT hashtag, count(*) as Count FROM MyHashtags\r\nGROUP BY hashtag\r\nORDER BY Count DESC\r\nLIMIT ${limit=20}<\/pre>\n<p>As you have noticed there is a variable inside the SQL statement &#8220;${limit=20}&#8221;. This will create an input field for easily pass some values:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-231\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24.png\" alt=\"\" width=\"2036\" height=\"1110\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24.png 2036w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-300x164.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-768x419.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-1030x562.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-1500x818.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-705x384.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-17.54.24-450x245.png 450w\" sizes=\"auto, (max-width: 2036px) 100vw, 2036px\" \/><\/a><\/p>\n<p>As nice and helpful as this could be, I still want to see the evolution of my hashtags through the time. So let&#8217;s run this SQL statement and see the visualisations:<\/p>\n<pre class=\"prettyprint\">SELECT *\r\nFROM(\r\n    SELECT *, ROW_NUMBER() OVER(PARTITION BY t.YEAR ORDER BY t.TotalCount DESC) as rank\r\n    FROM(\r\n        SELECT year(timestamp) as YEAR, hashtag, count(1) as TotalCount\r\n        FROM MyHashtags\r\n        GROUP BY hashtag, year(timestamp)\r\n    )t\r\n)WHERE rank &lt;= 10<\/pre>\n<p>This will show top 10 hashtags in each year:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-239\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43.png\" alt=\"\" width=\"2576\" height=\"1610\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43.png 2576w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-300x188.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-768x480.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-1030x644.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-1500x938.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-705x441.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-24-23.58.43-450x281.png 450w\" sizes=\"auto, (max-width: 2576px) 100vw, 2576px\" \/><\/a><\/p>\n<p>This tells me I used to tweet about #aws a lot back during 2013 and 2014. To dig deeper we can use Area Chart with Stream option and select #aws:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-241\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35.png\" alt=\"\" width=\"2576\" height=\"1308\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35.png 2576w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-300x152.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-768x390.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-1030x523.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-1500x762.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-705x358.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-00.04.35-450x228.png 450w\" sizes=\"auto, (max-width: 2576px) 100vw, 2576px\" \/><\/a><\/p>\n<p>It&#8217;s time to do the same thing with my @mentions. First we create a new DataFrame which contains a extracted mentions:<\/p>\n<pre class=\"prettyprint\">val mentionsDF = new RegexTokenizer()\r\n.setGaps(false)\r\n.setPattern(\"@(\\\\w+)\")\/\/anything starts with @\r\n.setMinTokenLength(2)\r\n.setInputCol(\"text\")\r\n.setOutputCol(\"mentions\")\/\/ extracted mentions\r\n.transform(newTweetDF)<\/pre>\n<p>I am interested in how many unique accounts have I ever mentioned:<\/p>\n<pre class=\"prettyprint\">val mentionsFreq = mentionsDF.select(explode($\"mentions\").as(\"value\")).groupBy(\"value\").count\r\nmentionsFreq.count()\r\n\r\nres108: Long = 1044\r\n<\/pre>\n<p>Since the result is a DataFrame I can display the top 10 accounts I have ever mentioned:<\/p>\n<pre class=\"prettyprint\">mentionsFreq.sort($\"count\".desc).show(10, false)\r\n<!--?prettify linenums=true?--><\/pre>\n<pre class=\"prettyprint\">+-----------+-----+\r\n|value      |count|\r\n+-----------+-----+\r\n|@verge     |368  |\r\n|@scoopit   |358  |\r\n|@cultofmac |168  |\r\n|@techcrunch|144  |\r\n|@iscpif    |106  |\r\n|@engadget  |104  |\r\n|@mashable  |101  |\r\n|@gracegynda|92   |\r\n|@thenextweb|87   |\r\n|@cnet      |72   |\r\n+-----------+-----+\r\nonly showing top 10 rows<\/pre>\n<p>However, it is more interesting to see the top 10 mentions of each year:<\/p>\n<pre class=\"prettyprint\">SELECT *\r\nFROM(\r\n    SELECT *, ROW_NUMBER() OVER(PARTITION BY t.YEAR ORDER BY t.TotalCount DESC) as rank\r\n    FROM(\r\n        SELECT year(timestamp) as YEAR, mention, count(1) as TotalCount\r\n        FROM MyMentions\r\n        GROUP BY mention, year(timestamp)\r\n    )t\r\n)WHERE rank &lt;= 10<\/pre>\n<p>This is a visualisation of my top mentions by year:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-244\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png\" alt=\"\" width=\"3172\" height=\"2000\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png 3172w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-300x189.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-768x484.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-1030x649.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-1500x946.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-705x445.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46-450x284.png 450w\" sizes=\"auto, (max-width: 3172px) 100vw, 3172px\" \/><\/a><\/p>\n<p>This chart shows me when and how much I mentioned @iscpif and @cnrs:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-245\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png\" alt=\"\" width=\"3166\" height=\"1636\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png 3166w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-300x155.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-768x397.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-1030x532.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-1500x775.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-705x364.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12-450x233.png 450w\" sizes=\"auto, (max-width: 3166px) 100vw, 3166px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>Well that makes sense since I joined ISC-PIF\/CNRS in 2014 \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the first part we saw some simple analytics and text extraction over source of each tweets from my Twitter archive. We worked with both Spark Scala APIs and Spark SQL in Apache Zeppelin notebooks hosted by Multivac. In part II we are going to do the followings: Extracting hashtags from content of each tweets [&hellip;]<\/p>\n","protected":false},"author":65,"featured_media":245,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[5],"tags":[21,23,17,24,25,22],"class_list":["post-220","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-multivac","tag-multivac","tag-scala","tag-spark","tag-sql","tag-twitter","tag-zeppelin"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog<\/title>\n<meta name=\"description\" content=\"Let&#039;s see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog\" \/>\n<meta property=\"og:description\" content=\"Let&#039;s see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data Blog\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-24T13:53:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-10-14T09:53:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png\" \/>\n\t<meta property=\"og:image:width\" content=\"3166\" \/>\n\t<meta property=\"og:image:height\" content=\"1636\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Maziyar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Maziyar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/\"},\"author\":{\"name\":\"Maziyar\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\"},\"headline\":\"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II\",\"datePublished\":\"2017-09-24T13:53:44+00:00\",\"dateModified\":\"2017-10-14T09:53:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/\"},\"wordCount\":582,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.26.12.png\",\"keywords\":[\"Multivac\",\"Scala\",\"Spark\",\"SQL\",\"Twitter\",\"Zeppelin\"],\"articleSection\":[\"Multivac\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/\",\"name\":\"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.26.12.png\",\"datePublished\":\"2017-09-24T13:53:44+00:00\",\"dateModified\":\"2017-10-14T09:53:58+00:00\",\"description\":\"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#primaryimage\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.26.12.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.26.12.png\",\"width\":3166,\"height\":1636},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-two\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#website\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\",\"name\":\"Big Data Blog\",\"description\":\"by ISC-PIF engineers\",\"publisher\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\",\"name\":\"Big Data Blog\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/iscpif-bigdata.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/iscpif-bigdata.png\",\"width\":2048,\"height\":2048,\"caption\":\"Big Data Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\",\"name\":\"Maziyar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"caption\":\"Maziyar\"},\"description\":\"Big Data engineer\",\"sameAs\":[\"https:\\\/\\\/multivac.iscpif.fr\"],\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/author\\\/maziyar-panahi\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog","description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/","og_locale":"en_US","og_type":"article","og_title":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog","og_description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","og_url":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/","og_site_name":"Big Data Blog","article_published_time":"2017-09-24T13:53:44+00:00","article_modified_time":"2017-10-14T09:53:58+00:00","og_image":[{"width":3166,"height":1636,"url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","type":"image\/png"}],"author":"Maziyar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Maziyar","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#article","isPartOf":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/"},"author":{"name":"Maziyar","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf"},"headline":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II","datePublished":"2017-09-24T13:53:44+00:00","dateModified":"2017-10-14T09:53:58+00:00","mainEntityOfPage":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/"},"wordCount":582,"commentCount":0,"publisher":{"@id":"https:\/\/iscpif.fr\/bigdata\/#organization"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","keywords":["Multivac","Scala","Spark","SQL","Twitter","Zeppelin"],"articleSection":["Multivac"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/","url":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/","name":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II - Big Data Blog","isPartOf":{"@id":"https:\/\/iscpif.fr\/bigdata\/#website"},"primaryImageOfPage":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#primaryimage"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","datePublished":"2017-09-24T13:53:44+00:00","dateModified":"2017-10-14T09:53:58+00:00","description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","breadcrumb":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#primaryimage","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","width":3166,"height":1636},{"@type":"BreadcrumbList","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-two\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/iscpif.fr\/bigdata\/"},{"@type":"ListItem","position":2,"name":"Simple Twitter Analytics by Multivac \u201cData Science Lab\u201d \u2013 Part II"}]},{"@type":"WebSite","@id":"https:\/\/iscpif.fr\/bigdata\/#website","url":"https:\/\/iscpif.fr\/bigdata\/","name":"Big Data Blog","description":"by ISC-PIF engineers","publisher":{"@id":"https:\/\/iscpif.fr\/bigdata\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/iscpif.fr\/bigdata\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/iscpif.fr\/bigdata\/#organization","name":"Big Data Blog","url":"https:\/\/iscpif.fr\/bigdata\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/logo\/image\/","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/iscpif-bigdata.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/iscpif-bigdata.png","width":2048,"height":2048,"caption":"Big Data Blog"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf","name":"Maziyar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","caption":"Maziyar"},"description":"Big Data engineer","sameAs":["https:\/\/multivac.iscpif.fr"],"url":"https:\/\/iscpif.fr\/bigdata\/author\/maziyar-panahi\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.26.12.png","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p93Nq6-3y","jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/220","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/comments?post=220"}],"version-history":[{"count":20,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/220\/revisions"}],"predecessor-version":[{"id":286,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/220\/revisions\/286"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/media\/245"}],"wp:attachment":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/media?parent=220"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/categories?post=220"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/tags?post=220"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}