{"id":118,"date":"2017-09-21T20:23:17","date_gmt":"2017-09-21T19:23:17","guid":{"rendered":"https:\/\/iscpif.fr\/bigdata\/?p=118"},"modified":"2017-10-14T10:50:19","modified_gmt":"2017-10-14T09:50:19","slug":"multivac-simple-twitter-analytics-part-one","status":"publish","type":"post","link":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/","title":{"rendered":"Simple Twitter Analytics by Multivac &#8220;Data Science Lab&#8221; &#8211; Part I"},"content":{"rendered":"<p>I am going to analyse all of my Tweets since 2010 by using\u00a0&#8220;<a href=\"https:\/\/iscpif.fr\/2017\/09\/introducing-multivac-data-science-lab\/\" target=\"_blank\" rel=\"noopener\">Multivac DSL<\/a>&#8220;.\u00a0 This tutorial will be in three parts. In first part I am going to do the followings:<\/p>\n<ul>\n<li>Read tweets in CSV format and convert them to Spark DataFrame<\/li>\n<li>Make some visualisations for better understanding<\/li>\n<li>Clean the &#8220;source&#8221; field<\/li>\n<li>Visualise the source of each tweet to find useful insights<\/li>\n<\/ul>\n<p>First, I need to login to Apache Zeppelin hosted by <strong>Multivac<\/strong> with my ISC-PIF LDAP account. This allows me to have my Spark Notebooks run everything over Multivac <strong>Hadoop<\/strong> cluster in our cloud. Now it&#8217;s time to get our hands on some tweets. For this, I am going to request my Twitter archive from my Twitter profile. Once this request has been sent it will be processed by Twitter and you will receive an email with a link to download.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-150\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.15.36.png\" alt=\"Twitter archive\" width=\"744\" height=\"253\" \/><\/p>\n<p><!--more--><\/p>\n<p>This is what&#8217;s inside the compressed directory that I downloaded:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-152\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613.png\" alt=\"Tweets in CSV format\" width=\"757\" height=\"283\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613.png 1301w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613-300x112.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613-768x287.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613-1030x385.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613-705x263.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-17.17.34-e1506094050613-450x168.png 450w\" sizes=\"auto, (max-width: 757px) 100vw, 757px\" \/><\/a><\/p>\n<p>You can see Twitter has already prepared a nice HTML page to browse your tweets:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-193\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29.png\" alt=\"Twitter UI for your tweets\" width=\"760\" height=\"729\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29.png 2498w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-300x288.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-768x737.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-1030x988.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-36x36.png 36w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-1500x1439.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-705x676.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-22-17.30.29-450x432.png 450w\" sizes=\"auto, (max-width: 760px) 100vw, 760px\" \/><\/a><\/p>\n<p>But we will focus on one particular file named &#8220;tweets.csv&#8221; which contains the content of your tweets with some minimal metadata.\u00a0Spark has the ability to easily read CSV or JSON files as\u00a0<a href=\"https:\/\/spark.apache.org\/docs\/latest\/sql-programming-guide.html\" target=\"_blank\" rel=\"noopener\">Spark DataFrame<\/a>.<\/p>\n<p>First, let&#8217;s take a look at our Spark version in our cluster:<\/p>\n<pre class=\"prettyprint\">spark.version\r\nres0: String = 2.2.0.cloudera1<\/pre>\n<p>Now here&#8217;s how you can read CSV file into DataFrame:<\/p>\n<pre class=\"prettyprint\">val tweetDF = spark.read.format(\"csv\")\r\n.option(\"header\", \"true\") \/\/file comes with headers, use them as the name of each column\r\n.option(\"charset\", \"UTF-8\")\/\/the tweets are in UTF-8 so we just making sure of it\r\n.option(\"qoute\", null)\/\/this is important cause some thweets contain double quotations and this messes the rows in SQL\r\n.option(\"mode\", \"DROPMALFORMED\")\/\/forget the corrupted rows\r\n.load(\"\/tmp\/tweets.csv\")\/\/ load the CSV file<\/pre>\n<p>Now the \u201c<strong>tweetDF<\/strong>\u201d variable contains our Tweets DataFrame. Let\u2019s take a look at schema and display 20 rows of this table:<\/p>\n<pre class=\"prettyprint\">tweetDF.printSchema\r\nroot\r\n |-- tweet_id: string (nullable = true)\r\n |-- in_reply_to_status_id: string (nullable = true)\r\n |-- in_reply_to_user_id: string (nullable = true)\r\n |-- timestamp: string (nullable = true)\r\n |-- source: string (nullable = true)\r\n |-- text: string (nullable = true)\r\n |-- retweeted_status_id: string (nullable = true)\r\n |-- retweeted_status_user_id: string (nullable = true)\r\n |-- retweeted_status_timestamp: string (nullable = true)\r\n |-- expanded_urls: string (nullable = true)\r\n<\/pre>\n<pre class=\"prettyprint\">tweetDF.count \/\/ how many rows exist in this DataFrame\r\nres2: Long = 5590\r\n\r\ntweetDF.show \/\/display 20 rows\r\n+------------------+---------------------+-------------------+--------------------+--------------------+--------------------+-------------------+------------------------+--------------------------+--------------------+\r\n|          tweet_id|in_reply_to_status_id|in_reply_to_user_id|           timestamp|              source|                text|retweeted_status_id|retweeted_status_user_id|retweeted_status_timestamp|       expanded_urls|\r\n+------------------+---------------------+-------------------+--------------------+--------------------+--------------------+-------------------+------------------------+--------------------------+--------------------+\r\n|895576868539314176|   895572859447332864|           48308882|2017-08-10 09:25:...|\"&lt;a href=\"\"http:\/...|@G3rf0x @netflix ...|               null|                    null|                      null|                null|\r\n|894556881552908288|                 null|               null|2017-08-07 13:52:...|\"&lt;a href=\"\"http:\/...|Indexing +100M @n...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|890613499495370754|   881867404875358209|         3890708435|2017-07-27 16:42:...|\"&lt;a href=\"\"http:\/...|@GeoMaisonobe @Em...|               null|                    null|                      null|                null|\r\n|884406660458053632|                 null|               null|2017-07-10 13:39:...|\"&lt;a href=\"\"https:...|RT @multivacsuppo...| 884406059489783809|      818419470473498624|      2017-07-10 13:36:...|                null|\r\n|883250853238497282|   882218484742201344|         3890708435|2017-07-07 09:06:...|\"&lt;a href=\"\"http:\/...|@GeoMaisonobe @Em...|               null|                    null|                      null|                null|\r\n|881900338323881985|   881867404875358209|         3890708435|2017-07-03 15:40:...|\"&lt;a href=\"\"http:\/...|@GeoMaisonobe @Em...|               null|                    null|                      null|                null|\r\n|881899963189534720|   881867404875358209|         3890708435|2017-07-03 15:38:...|\"&lt;a href=\"\"http:\/...|@GeoMaisonobe @Em...|               null|                    null|                      null|                null|\r\n|881858875561046017|   881857513888600064|         3890708435|2017-07-03 12:55:...|\"&lt;a href=\"\"http:\/...|@GeoMaisonobe @Em...|               null|                    null|                      null|                null|\r\n|867330936693829633|   867330008322375681|          572447555|2017-05-24 10:46:...|\"&lt;a href=\"\"http:\/...|@auberiemrc @mult...|               null|                    null|                      null|                null|\r\n|865897556433358848|                 null|               null|2017-05-20 11:50:...|\"&lt;a href=\"\"http:\/...|Reconfiguration d...|               null|                    null|                      null|https:\/\/politosco...|\r\n|865519971635314689|                 null|               null|2017-05-19 10:50:...|\"&lt;a href=\"\"http:\/...|We are studying F...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864846964470538241|                 null|               null|2017-05-17 14:15:...|\"&lt;a href=\"\"http:\/...|Que dit Twitter \u00e0...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864732912792608768|                 null|               null|2017-05-17 06:42:...|\"&lt;a href=\"\"http:\/...|L'\u00e9quipe ISC-PIF ...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864362990526545921|                 null|               null|2017-05-16 06:12:...|\"&lt;a href=\"\"http:\/...|RT @chavalarias: ...| 864263262313041921|                97267781|      2017-05-15 23:36:...|https:\/\/politosco...|\r\n|864151163989610500|                 null|               null|2017-05-15 16:11:...|\"&lt;a href=\"\"http:\/...|\"Graph of mention...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864107860036341761|                 null|               null|2017-05-15 13:19:...|\"&lt;a href=\"\"http:\/...|\"Who's talking ab...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864106463777755136|                 null|               null|2017-05-15 13:13:...|\"&lt;a href=\"\"http:\/...|\"What does French...|               null|                    null|                      null|https:\/\/twitter.c...|\r\n|864091533968015360|   864088698970939396|          392755065|2017-05-15 12:14:...|\"&lt;a href=\"\"https:...|@ISCPIF everythin...|               null|                    null|                      null|                null|\r\n|863302644449849347|                 null|               null|2017-05-13 07:59:...|\"&lt;a href=\"\"http:\/...|ISC-PIF\/CNRS is p...|               null|                    null|                      null|https:\/\/multivac....|\r\n|862540789729153024|                 null|               null|2017-05-11 05:32:...|\"&lt;a href=\"\"http:\/...|RT @dadoonet: w00...| 862222612977635328|                51172224|      2017-05-10 08:27:...|https:\/\/twitter.c...|\r\n+------------------+---------------------+-------------------+--------------------+--------------------+--------------------+-------------------+------------------------+--------------------------+--------------------+\r\nonly showing top 20 rows\r\n<\/pre>\n<p>As you can see there is column named &#8220;<strong>source<\/strong>&#8221; that indicates the medium which I posted my Tweets (iPhone, Web, third-party applications, etc.).<\/p>\n<p>This is how it looks like:<\/p>\n<pre class=\"prettyprint\">tweetDF.groupBy(\"source\").count.sort($\"count\".desc).show(false)\r\n+------------------------------------------------------------------------------------------------------------------------------+-----+\r\n|source                                                                                                                        |count|\r\n+------------------------------------------------------------------------------------------------------------------------------+-----+\r\n|\"&lt;a href=\"\"http:\/\/www.flipboard.com\"\" rel=\"\"nofollow\"\"&gt;Flipboard&lt;\/a&gt;\"                                                         |1721 |\r\n|\"&lt;a href=\"\"http:\/\/foursquare.com\"\" rel=\"\"nofollow\"\"&gt;Foursquare&lt;\/a&gt;\"                                                           |608  |\r\n|\"&lt;a href=\"\"http:\/\/twitter.com\/download\/iphone\"\" rel=\"\"nofollow\"\"&gt;Twitter for iPhone&lt;\/a&gt;\"                                      |434  |\r\n|\"&lt;a href=\"\"http:\/\/instagram.com\"\" rel=\"\"nofollow\"\"&gt;Instagram&lt;\/a&gt;\"                                                             |376  |\r\n|\"&lt;a href=\"\"http:\/\/twitter.com\"\" rel=\"\"nofollow\"\"&gt;Twitter Web Client&lt;\/a&gt;\"                                                      |375  |\r\n|\"&lt;a href=\"\"http:\/\/www.scoop.it\"\" rel=\"\"nofollow\"\"&gt;Scoop.it&lt;\/a&gt;\"                                                               |365  |\r\n|\"&lt;a href=\"\"http:\/\/bufferapp.com\"\" rel=\"\"nofollow\"\"&gt;Buffer&lt;\/a&gt;\"                                                                |358  |\r\n|\"&lt;a href=\"\"https:\/\/dev.twitter.com\/docs\/tfw\"\" rel=\"\"nofollow\"\"&gt;Twitter for Websites&lt;\/a&gt;\"                                      |355  |\r\n|\"&lt;a href=\"\"http:\/\/www.apple.com\"\" rel=\"\"nofollow\"\"&gt;iOS&lt;\/a&gt;\"                                                                   |301  |\r\n|\"&lt;a href=\"\"http:\/\/www.pulse.me\"\" rel=\"\"nofollow\"\"&gt;Pulse News&lt;\/a&gt;\"                                                             |117  |\r\n|\"&lt;a href=\"\"http:\/\/itunes.apple.com\/us\/app\/bbc-news\/id364147881?mt=8&amp;uo=4\"\" rel=\"\"nofollow\"\"&gt;BBC News on iOS&lt;\/a&gt;\"              |93   |\r\n|\"&lt;a href=\"\"https:\/\/vimeo.com\"\" rel=\"\"nofollow\"\"&gt;Vimeo&lt;\/a&gt;\"                                                                    |71   |\r\n|\"&lt;a href=\"\"http:\/\/www.apple.com\/\"\" rel=\"\"nofollow\"\"&gt;OS X&lt;\/a&gt;\"                                                                 |57   |\r\n|\"&lt;a href=\"\"https:\/\/about.twitter.com\/products\/tweetdeck\"\" rel=\"\"nofollow\"\"&gt;TweetDeck&lt;\/a&gt;\"                                     |47   |\r\n|\"&lt;a href=\"\"http:\/\/www.linkedin.com\/\"\" rel=\"\"nofollow\"\"&gt;LinkedIn&lt;\/a&gt;\"                                                          |39   |\r\n|\"&lt;a href=\"\"http:\/\/spotify.com\"\" rel=\"\"nofollow\"\"&gt;Spotify&lt;\/a&gt;\"                                                                 |32   |\r\n|\"&lt;a href=\"\"http:\/\/itunes.apple.com\/us\/app\/feedly\/id396069556?mt=8&amp;uo=4\"\" rel=\"\"nofollow\"\"&gt;feedly on iOS&lt;\/a&gt;\"                  |29   |\r\n|\"&lt;a href=\"\"http:\/\/www.foxnews.com\"\" rel=\"\"nofollow\"\"&gt;Fox News&lt;\/a&gt;\"                                                            |25   |\r\n|\"&lt;a href=\"\"http:\/\/klout.com\"\" rel=\"\"nofollow\"\"&gt;Post with Klout&lt;\/a&gt;\"                                                           |18   |\r\n|\"&lt;a href=\"\"http:\/\/itunes.apple.com\/us\/app\/google-currents\/id459182288?mt=8&amp;uo=4\"\" rel=\"\"nofollow\"\"&gt;Google Currents on iOS&lt;\/a&gt;\"|17   |\r\n+------------------------------------------------------------------------------------------------------------------------------+-----+\r\nonly showing top 20 rows<\/pre>\n<p>As it can be seen, it&#8217;s not much of a help. It&#8217;s even hard to read with all the HTML tags let along to be any use for any aggregation. So I am going to clean up the HTML tag with a simple regex wrapped around Spark UDF:<\/p>\n<pre class=\"prettyprint\">\/\/define a UDF to replace HTML tags and lowercase the content\r\ndef extractTweetSourceUDF = udf { tweet: String =&gt;\r\n  tweet.replaceAll(\"&lt;[^&gt;]*&gt;\", \"\").toLowerCase()\r\n}<\/pre>\n<p>You can read more about Spark UDF <a href=\"https:\/\/blog.cloudera.com\/blog\/2017\/02\/working-with-udfs-in-apache-spark\/\" target=\"_blank\" rel=\"noopener\">here<\/a>, but simply it&#8217;s a function that can be applied over DataFrame (one or more columns).<\/p>\n<p>Now let&#8217;s run the UDF over our DataFrame:\u00a0<strong>tweetDF<\/strong><\/p>\n<pre class=\"prettyprint\">val newTweetDF = tweetDF\r\n.filter(\"source IS NOT null\")\/\/make sure the source is not empty\r\n.withColumn(\"cleanSource\", extractTweetSourceUDF(col(\"source\")))\/\/apply the UDF and create a new column name cleanSource to contain the results\r\n<\/pre>\n<p>Displaying our new column:\u00a0<strong>cleanSource<\/strong><\/p>\n<pre class=\"prettyprint\">newTweetDF.select(\"cleanSource\").show(false)\r\n+--------------------+\r\n|cleanSource         |\r\n+--------------------+\r\n|\"twitter for iphone\"|\r\n|\"twitter web client\"|\r\n|\"twitter for iphone\"|\r\n|\"tweetdeck\"         |\r\n|\"twitter for iphone\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"twitter for iphone\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"twitter for iphone\"|\r\n|\"twitter for iphone\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"twitter web client\"|\r\n|\"tweetdeck\"         |\r\n|\"twitter for iphone\"|\r\n|\"twitter for iphone\"|\r\n+--------------------+\r\nonly showing top 20 rows<\/pre>\n<p>It&#8217;s better now! We can use some aggregations with this and see some insights.<\/p>\n<p>Let&#8217;s see the top 20 sources I have ever used to post my tweets:<\/p>\n<pre class=\"prettyprint\">newTweetDF.groupBy(\"cleanSource\").count.sort($\"count\".desc).show(false)\r\n+------------------------+-----+\r\n|cleanSource             |count|\r\n+------------------------+-----+\r\n|\"flipboard\"             |1721 |\r\n|\"foursquare\"            |608  |\r\n|\"twitter for iphone\"    |434  |\r\n|\"instagram\"             |376  |\r\n|\"twitter web client\"    |375  |\r\n|\"scoop.it\"              |365  |\r\n|\"buffer\"                |358  |\r\n|\"twitter for websites\"  |355  |\r\n|\"ios\"                   |301  |\r\n|\"pulse news\"            |117  |\r\n|\"bbc news on ios\"       |93   |\r\n|\"vimeo\"                 |71   |\r\n|\"os x\"                  |57   |\r\n|\"tweetdeck\"             |47   |\r\n|\"linkedin\"              |39   |\r\n|\"spotify\"               |32   |\r\n|\"feedly on ios\"         |29   |\r\n|\"fox news\"              |25   |\r\n|\"post with klout\"       |18   |\r\n|\"google currents on ios\"|17   |\r\n+------------------------+-----+\r\nonly showing top 20 rows<\/pre>\n<p>Let&#8217;s visualise this with built-in charts in Zeppelin. But before doing that I am going to create a temporary view in Spark so I can run SQL queries in another block:<\/p>\n<pre class=\"prettyprint\">newTweetDF.createOrReplaceTempView(\"MyTweets\")<\/pre>\n<p>Now I can run SQL statements by simply using %<strong>sql<\/strong>\u00a0in my notebooks:<\/p>\n<pre class=\"prettyprint\">%sql\r\nSELECT cleanSource, count(1) as TotalCount\r\nFROM MyTweets\r\nGROUP BY cleanSource\r\nORDER BY TotalCount DESC\r\nLIMIT 10<\/pre>\n<p>This is the results:<\/p>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-160 size-full\" title=\"Visualising by Multivac DSL Spark notebook\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47.png\" alt=\"Aggregating source of tweets\" width=\"3018\" height=\"1284\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47.png 3018w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-300x128.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-768x327.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-1030x438.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-1500x638.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-705x300.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.13.47-450x191.png 450w\" sizes=\"auto, (max-width: 3018px) 100vw, 3018px\" \/><\/a> <a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-159 size-full\" title=\"Visualising by Multivac DSL Spark notebook\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16.png\" alt=\"Aggregating source of tweets\" width=\"3012\" height=\"1234\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16.png 3012w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-300x123.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-768x315.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-1030x422.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-1500x615.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-705x289.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-19.12.16-450x184.png 450w\" sizes=\"auto, (max-width: 3012px) 100vw, 3012px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>OK! I don&#8217;t remember the last time I used &#8220;Flipboard&#8221; to share something on Twitter. Let&#8217;s see a histogram of my Twitter activity first before we go and see where this Flipboard thing is coming from:<\/p>\n<pre class=\"prettyprint\">%sql\r\nSELECT year(timestamp) as YEAR, count(*) as COUNT\r\nFROM MyTweets\r\nGROUP BY YEAR\r\nORDER BY YEAR DESC<\/pre>\n<p><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-162 size-full\" title=\"Visualising by Multivac DSL Spark notebook\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56.png\" alt=\"Twitter activity \" width=\"2006\" height=\"912\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56.png 2006w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-300x136.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-768x349.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-1030x468.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-1500x682.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-705x321.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-22.28.56-450x205.png 450w\" sizes=\"auto, (max-width: 2006px) 100vw, 2006px\" \/><\/a><\/p>\n<p>OK! I was very active during 2012 and 2013. Let&#8217;s see the same thing but instead of Twitter activity I am going to see the top sources I used to tweet from by year:<\/p>\n<p><span style=\"color: #000000;\"><a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-163 size-full\" title=\"Visualising by Multivac DSL Spark notebook\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52.png\" alt=\"Twitter activity by source of tweets\" width=\"2196\" height=\"1410\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52.png 2196w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-300x193.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-768x493.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-1030x661.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-1500x963.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-705x453.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.06.52-450x289.png 450w\" sizes=\"auto, (max-width: 2196px) 100vw, 2196px\" \/><\/a> <a href=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-165 size-full\" title=\"Visualising by Multivac DSL Spark notebook\" src=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18.png\" alt=\"Twitter activity by source of tweets by Multivac DSL\" width=\"2368\" height=\"1244\" srcset=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18.png 2368w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-300x158.png 300w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-768x403.png 768w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-1030x541.png 1030w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-1500x788.png 1500w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-705x370.png 705w, https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-13-23.15.18-450x236.png 450w\" sizes=\"auto, (max-width: 2368px) 100vw, 2368px\" \/><\/a><\/span><\/p>\n<p>Alright! Apparently I used to be a fan of &#8220;Flipboard&#8221; back in 2012 and after that my usage of this source dropped almost to zero by 2015.<\/p>\n<blockquote><p>Looking at my Twitter activity (I was very active during 2012-2013) and looking at the yearly source of my tweets can explain my surprise of why my top source of Tweeting is &#8220;Flipboard&#8221;.<\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<p>In the second part I am going to take a look at hashtags, accounts I mentioned, original tweets vs retweeted tweets and some word processing to understand what I have mostly focused when I was on Twitter.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I am going to analyse all of my Tweets since 2010 by using\u00a0&#8220;Multivac DSL&#8220;.\u00a0 This tutorial will be in three parts. In first part I am going to do the followings: Read tweets in CSV format and convert them to Spark DataFrame Make some visualisations for better understanding Clean the &#8220;source&#8221; field Visualise the source [&hellip;]<\/p>\n","protected":false},"author":65,"featured_media":244,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[5],"tags":[21,23,17,24,25,22],"class_list":["post-118","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-multivac","tag-multivac","tag-scala","tag-spark","tag-sql","tag-twitter","tag-zeppelin"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Simple Twitter Analytics by Multivac &quot;Data Science Lab&quot; - Part I - Big Data Blog<\/title>\n<meta name=\"description\" content=\"Let&#039;s see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Simple Twitter Analytics by Multivac &quot;Data Science Lab&quot; - Part I - Big Data Blog\" \/>\n<meta property=\"og:description\" content=\"Let&#039;s see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/\" \/>\n<meta property=\"og:site_name\" content=\"Big Data Blog\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-21T19:23:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-10-14T09:50:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png\" \/>\n\t<meta property=\"og:image:width\" content=\"3172\" \/>\n\t<meta property=\"og:image:height\" content=\"2000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Maziyar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Maziyar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/\"},\"author\":{\"name\":\"Maziyar\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\"},\"headline\":\"Simple Twitter Analytics by Multivac &#8220;Data Science Lab&#8221; &#8211; Part I\",\"datePublished\":\"2017-09-21T19:23:17+00:00\",\"dateModified\":\"2017-10-14T09:50:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/\"},\"wordCount\":611,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.23.46.png\",\"keywords\":[\"Multivac\",\"Scala\",\"Spark\",\"SQL\",\"Twitter\",\"Zeppelin\"],\"articleSection\":[\"Multivac\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/\",\"name\":\"Simple Twitter Analytics by Multivac \\\"Data Science Lab\\\" - Part I - Big Data Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.23.46.png\",\"datePublished\":\"2017-09-21T19:23:17+00:00\",\"dateModified\":\"2017-10-14T09:50:19+00:00\",\"description\":\"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#primaryimage\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.23.46.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Screenshot-2017-09-25-12.23.46.png\",\"width\":3172,\"height\":2000},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/2017\\\/09\\\/multivac-simple-twitter-analytics-part-one\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Simple Twitter Analytics by Multivac &#8220;Data Science Lab&#8221; &#8211; Part I\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#website\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\",\"name\":\"Big Data Blog\",\"description\":\"by ISC-PIF engineers\",\"publisher\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#organization\",\"name\":\"Big Data Blog\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/iscpif-bigdata.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/iscpif-bigdata.png\",\"width\":2048,\"height\":2048,\"caption\":\"Big Data Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/#\\\/schema\\\/person\\\/02ae0118261368c80d32bef0b75f5faf\",\"name\":\"Maziyar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"contentUrl\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2017\\\/09\\\/Maziyar-Panahi-120x120.png\",\"caption\":\"Maziyar\"},\"description\":\"Big Data engineer\",\"sameAs\":[\"https:\\\/\\\/multivac.iscpif.fr\"],\"url\":\"https:\\\/\\\/iscpif.fr\\\/bigdata\\\/author\\\/maziyar-panahi\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Simple Twitter Analytics by Multivac \"Data Science Lab\" - Part I - Big Data Blog","description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/","og_locale":"en_US","og_type":"article","og_title":"Simple Twitter Analytics by Multivac \"Data Science Lab\" - Part I - Big Data Blog","og_description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","og_url":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/","og_site_name":"Big Data Blog","article_published_time":"2017-09-21T19:23:17+00:00","article_modified_time":"2017-10-14T09:50:19+00:00","og_image":[{"width":3172,"height":2000,"url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","type":"image\/png"}],"author":"Maziyar","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Maziyar","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#article","isPartOf":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/"},"author":{"name":"Maziyar","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf"},"headline":"Simple Twitter Analytics by Multivac &#8220;Data Science Lab&#8221; &#8211; Part I","datePublished":"2017-09-21T19:23:17+00:00","dateModified":"2017-10-14T09:50:19+00:00","mainEntityOfPage":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/"},"wordCount":611,"commentCount":0,"publisher":{"@id":"https:\/\/iscpif.fr\/bigdata\/#organization"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","keywords":["Multivac","Scala","Spark","SQL","Twitter","Zeppelin"],"articleSection":["Multivac"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/","url":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/","name":"Simple Twitter Analytics by Multivac \"Data Science Lab\" - Part I - Big Data Blog","isPartOf":{"@id":"https:\/\/iscpif.fr\/bigdata\/#website"},"primaryImageOfPage":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#primaryimage"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#primaryimage"},"thumbnailUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","datePublished":"2017-09-21T19:23:17+00:00","dateModified":"2017-10-14T09:50:19+00:00","description":"Let's see how to use Multivac Data Science Lab for Twitter analytics. We are going to use interactive Spark notebook to analyse and visualise some tweets.","breadcrumb":{"@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#primaryimage","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","width":3172,"height":2000},{"@type":"BreadcrumbList","@id":"https:\/\/iscpif.fr\/bigdata\/2017\/09\/multivac-simple-twitter-analytics-part-one\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/iscpif.fr\/bigdata\/"},{"@type":"ListItem","position":2,"name":"Simple Twitter Analytics by Multivac &#8220;Data Science Lab&#8221; &#8211; Part I"}]},{"@type":"WebSite","@id":"https:\/\/iscpif.fr\/bigdata\/#website","url":"https:\/\/iscpif.fr\/bigdata\/","name":"Big Data Blog","description":"by ISC-PIF engineers","publisher":{"@id":"https:\/\/iscpif.fr\/bigdata\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/iscpif.fr\/bigdata\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/iscpif.fr\/bigdata\/#organization","name":"Big Data Blog","url":"https:\/\/iscpif.fr\/bigdata\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/logo\/image\/","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/iscpif-bigdata.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/iscpif-bigdata.png","width":2048,"height":2048,"caption":"Big Data Blog"},"image":{"@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/iscpif.fr\/bigdata\/#\/schema\/person\/02ae0118261368c80d32bef0b75f5faf","name":"Maziyar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","contentUrl":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Maziyar-Panahi-120x120.png","caption":"Maziyar"},"description":"Big Data engineer","sameAs":["https:\/\/multivac.iscpif.fr"],"url":"https:\/\/iscpif.fr\/bigdata\/author\/maziyar-panahi\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/iscpif.fr\/bigdata\/wp-content\/uploads\/sites\/25\/2017\/09\/Screenshot-2017-09-25-12.23.46.png","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p93Nq6-1U","jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/users\/65"}],"replies":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/comments?post=118"}],"version-history":[{"count":39,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/118\/revisions"}],"predecessor-version":[{"id":123,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/posts\/118\/revisions\/123"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/media\/244"}],"wp:attachment":[{"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/media?parent=118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/categories?post=118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/iscpif.fr\/bigdata\/wp-json\/wp\/v2\/tags?post=118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}