MuskSentiment is one of my favorite projects and I’m extremely proud of how it turned out. The project is my inaugural dive into data analytics, even though I didn’t really know what data analytics was at the time. It is also the first project with visual output, a significant step up from simple terminal output (which I still like to do!).
The project aims to take in a keyword, usually reflecting current trends (such as elections, celebrity drama, news, etc.). It then plots the sentiment of this keyword against time, demonstrating fluctuations in public opinion.
The project reveals — or rather, confirms — that Twitter is negative. Training and tuning the underlying TensorFlow model causes a lot of struggle, due to the fact that, on average, the inferred sentiment did not hover around the 0.5 (50%) mark. This leads me to the suspicion that the training data are biased or skewed. I even considered projecting the output values onto a custom scale, weighting each point to yield an average sentiment of 0.5. After further deliberation, I dismiss this idea, as such a modification would not accurately reflect public sentiment, assuming the training data are unbiased.
Turns out, Twitter just LOVES to be negative: the average sentiment for most queries hovers around 0.4, on a scale of 0 to 1. I should’ve known.
Note the sheer volume of tweets (xeets) about Ticketmaster — 43,000 tweets in the span of 9 hours. You might remember what happened that day.
While the program contains no graphical interface and all queries are submitted through a CLI (Command Line Interface), the computed trendlines were displayed with the Matplotlib Python library. Through the Twitter API, I pulled the requested number of tweets (usually 10-100k), relying on pagination to cater to Twitter’s generous request rate limits at the time (Long Live Twitter API). I also apply for the Elevated API tier reserved for special use cases, which increases my monthly tweet quota into the millions. The requested number of Tweets is denoted by the first number in the graph title and the actual returned number is in the parentheses (note: an older version of MuskSentiment did not display the actual number of returned results). Unfortunately, my implementation is limited by the scope of the Recent Search endpoint of the API, which returns Tweets up to a week old.
Each returned Tweet was ordered by date and assigned a binary sentiment, 0 or 1. The final output was produced incorporating a rolling mean that had a window width of totalReturnedTweets/13, with 13 being an arbitrary number that balanced a readable output with the desire for detail and precision.
Initially, the project relies on the OpenAI API, but it quickly becomes apparent that it is financially unfeasible to complete the requests at the desired volume. As a result, I switched to a TensorFlow classifier model trained on an existing dataset containing sentences and their assigned sentiment. Not all mention of OpenAI was erased, though — A snippet of code calculates the total amount of money saved in API costs by switching to TensorFlow. Right now, that number sits at $3,627.
Many of my personal projects originate from the desire of creating a service that, under my impression, did not exist and completely satisfied a need I had at the time. Also, having maximum control and knowledge of the service was incredibly beneficial. Therefore, creating the program myself, rather than seeking existing, partial solutions, was a no-brainer. I maintain a strong thirst for information surrounding current events and believe data on public sentiment is a crucial component required to grasp an entire situation.
At the time, the Twitter API had an incredible advantage over other data sources — it promptly provided a vast and relevant dataset with real public opinion, storing Tweets created as recently as thirty seconds prior to the API request, resulting in an unparalleled source for insight into realtime events.
So, does MuskSentiment live up to its standards, ideologies, goals, and motivations?
MuskSentiment is an astounding success. Every graph produced had an intriguing story attached to it, whether it be an Arsenal comeback, a celebrity birthday, or a Ticketmaster controversy. Queries with fewer than 43,000 tweets generated interesting results within seconds. Larger queries required manual rate limiting but produced a result in minutes, with live updates displaying progress.