Sentiment Analysis as a Trading Indicator

Alexander Özkan · March 28, 2020

I’d like to preface this by saying that I do not have a finance background in any form and this is simply an exploratory look.

I’d also like to say that this is a code heavy analysis, but you can skip to the bottom for the conclusion and plotting.

That aside, in order to see what impact the news has on the global stock markets we’ll need to gather some data.

We’ll start with the S&P 500 as our sample of stocks. I’ve sourced this dataset from It’s a little out of date, so we’ll need to verify that all of the company tickers/symbols still exist.

Let’s begin by getting some of the imports we need:

import pandas as pd
import matplotlib.pyplot as plt

The S&P data is in a .csv file, so we can import it and take a look using Pandas quite easily:

sp = pd.read_csv('constituents.csv')
(489, 3)
Symbol Name Sector
0 MMM 3M Company Industrials
1 AOS A.O. Smith Corp Industrials
2 ABT Abbott Laboratories Health Care
3 ABBV AbbVie Inc. Health Care
4 ACN Accenture plc Information Technology

As you can see we’ve got 492 rows and 3 columns, the symbol/ticker, name of the company, and sector the company operates in.

Let’s see what sectors comprise the S&P:

Consumer Discretionary        0.163599
Information Technology        0.141104
Financials                    0.139059
Industrials                   0.134969
Health Care                   0.122699
Real Estate                   0.065440
Consumer Staples              0.065440
Utilities                     0.057260
Energy                        0.057260
Materials                     0.047035
Telecommunication Services    0.006135
Name: Sector, dtype: float64
sector_counts = pd.value_counts(sp['Sector'].values, sort=True)
sector_pie = sector_counts.plot.pie(autopct='%.0f%%')
Text(0, 0.5, '')


I’ve normalised the values and we can see that consumer discretionary based companies are leading (at the time of this data), with 16% of the S&P 500 being companies in that sector.

Querying for Price and Volume

We’re going to need to get the daily price for each stock, as well as information about how many shares per day are being traded.

We can use a free API by Alpha Vantage for this. They’ll give us 500 free requests per day. Just enough for our use!

I’ve written a script to query the API for a 20 day historic list of the price the equity opened at, price it closed at, as well as the high/low and the volume for that day. This information is then inserted into an SQL database for later use.

All code is viewable here on my GitHub

The script was written to account for any no longer existing tickers (via reading the API errors), and the following were identified and removed:

    Symbol                            Name                  Sector
45    ANDV                        Andeavor                  Energy
68     BBT                BB&T Corporation              Financials
83    BF.B              Brown-Forman Corp.        Consumer Staples
133   CSRA                       CSRA Inc.  Information Technology
155    DPS         Dr Pepper Snapple Group        Consumer Staples
170   EVHC             Envision Healthcare             Health Care
210    GGP  General Growth Properties Inc.             Real Estate
277    LLL     L-3 Communications Holdings             Industrials
317    MON                    Monsanto Co.               Materials
330    NFX         Newfield Exploration Co                  Energy
367    PXD       Pioneer Natural Resources                  Energy
445    TWX                Time Warner Inc.  Consumer Discretionary
494    WYN               Wyndham Worldwide  Consumer Discretionary

Let’s take a look at the freshly gathered equity data:

import sqlite3

eq_db = sqlite3.connect('equities.db')
stock_info = pd.read_sql_query("SELECT * FROM equities_daily", eq_db)
stock_info = stock_info.set_index('ticker')

date open high low close volume
MMM 2020-03-26 131.79 136.38 130.61 136.18 6693932.0
MMM 2020-03-25 133.15 134.69 126.80 131.54 7740084.0
MMM 2020-03-24 122.29 133.45 121.00 132.72 9304832.0
MMM 2020-03-23 128.16 128.40 114.04 117.87 7920348.0
MMM 2020-03-20 138.07 139.24 122.71 124.89 9582251.0

Querying for News

A reliable source of news that can be queried by an application can get expensive quickly. Thankfully there’s a few free/low cost solutions.

For this project I’ll be using Similar to Alpha Vantage we get 500 free requests a day. This allows for a pretty granular search critera and has plenty of reliable news sources.

We’re only interested in the article’s published date, url, title, and description. In the same script as above on my GitHub we are parsing and inserting the relevant data to an SQL database for later use. For now it’s the previous 20 days worth of news.

Here’s a look at that news data:

news_db = sqlite3.connect('news.db')
news_info = pd.read_sql_query("SELECT * FROM news_daily", news_db)
news_info = news_info.set_index('ticker')

date url title description
MMM 2020-03-24 CVS plans to hire furloughed workers from its ... CVS plans to hire furloughed workers from its ...
MMM 2020-03-18 Biggest maker of face masks in U.S. is warning... The biggest maker of medical face masks in the...
MMM 2020-03-19 Ask HN: How should I invest $200K in this market? Comments
MMM 2020-03-03 Why Shares of 3M Are Down Today One of the better-performing industrials throu...
MMM 2020-03-10 It Is Not About 3M, It Is About You 3M is a great businesses. This makes investing...

Let’s take a closer look at something other than 3M:

news_info.loc['CSCO', 'title']
CSCO    Why Cisco Systems Stock Slumped 13.1% in February
CSCO    Cisco committing $225M to global coronavirus r...
CSCO    Why Cisco Stock Is Becoming Attractive Followi...
CSCO    Deep Dive: You can be ‘practically stealing’ q...
CSCO    Hedge Funds Have Never Been This Bullish On Ci...
CSCO    Deep Dive: These stocks may be your best choic...
CSCO    Raymond James and Cisco to Host a Tech Talk on...
CSCO    Deep Dive: These stocks soared the most after ...
CSCO    Deep Dive: Here are Thursday’s best-performing...
CSCO    Coronavirus school cancellations lead to educa...
CSCO                        Tech Is The Solution - Nasdaq
CSCO                 Tech Is The Solution - Yahoo Finance
CSCO    Deep Dive: These stocks took the biggest hit a...
CSCO    Oxbotica and Cisco to Solve Autonomous Vehicle...
CSCO    Oxbotica and Cisco to Solve Autonomous Vehicle...
CSCO    Microsoft Teams Adds 12M Customers In A Week A...
CSCO    Where Tech Stock Valuations Stand Following a ...
CSCO    Deep Dive: These U.S. stocks fell the most aft...
CSCO    Datadog Stock Finds Support Amid Coronavirus -...
CSCO                    Cisco begins new round of layoffs
Name: title, dtype: object

In case you haven’t seen the full code for how I’m sourcing this news, it’s worth noting that I’m searching for both the ticker and full company name when it comes to news.

This means that even for easily misunderstood tickers that we’re still getting accurate news:

news_info.loc['A', 'title']
A    You Have To Love Agilent Technologies, Inc.'s ...
A    Agilent Technologies to Adjourn Annual Meeting...
A    Agilent Technologies Announces Cash Dividend o...
A    Agilent Technologies Announces Webcasts for In...
A    Is Agilent Technologies Inc. (A) Going To Burn...
A    Kim Kardashian y Kylie Jenner dejan de vender ...
A    Agilent Introduces CrossLab Connect Services f...
A       Agilent Receives Two Scientists’ Choice Awards
A    Agilent Introduces Three New Microarrays for P...
A    Agilent and Visiopharm Co-promote Advanced Dig...
A    Bill Ackman Continues To Chip Away At Largest ...
A    Is Agilent Technologies Inc. (A) Going To Burn...
A    World Flow Cytometry Industry Outlook, 2020-20...
A    Hedge Funds Have Never Been This Bullish On Sq...
A    Worldwide Genomics Markets, 2020-2027 - Compre...
A    The global protein sequencing market is antici...
A    Asia Pacific Genomics and Proteomic Tools Mark...
A    Global Environmental Sensing and Monitoring Te...
A    Proteomics Industry Analysis, 2020-2026 - Outb...
A    Global DNA Sequencing Market (2020 to 2024) - ...
Name: title, dtype: object

Sentiment Analysis - VADER

Now that we’ve got a small bit of data compiled, we can start to explore it further and see if there’s any correlations.

The first problem we’ll face is choosing how to determine the sentiment of a news article. The field of natural language processing and sentiment analysis is massive and generally complex.

I’m going to start this project off by using a simplified method of sentiment analysis - the Valence Aware Dictionary and sEntiment Reasoner (VADER) method. This is a rule-based/lexicon technique. It’s particularly good for social media related content, but we’ll test it out on news headlines now.

Normally for sentiment analysis you would remove stopwords (filler words of no value) from the text you are processing, but in the case of VADER it is advantageous to leave them in.

Let’s define our sentiment analysis function:

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA

def determineSentiment(title):
    Calculates the weighted sentiment of a piece of text
    Returns a string which is the determined score of sentiment

    # VADER Polarity Score of Sentiment
    sia = SIA()
    results = []

    pol_score = sia.polarity_scores(title)
    pol_score['news_text'] = title

    ## Check the compound result of the analysis
    ## Tolerances are near mirrors of those used in academia (though more sensitive to negativity)
    compound = results[0]['compound']
    if compound >= 0.05:
        return 'Positive'
    elif compound >= -0.02 and compound < 0.05:
        return 'Neutral'
    elif compound < -0.02:
        return 'Negative'

Now let’s pass in some titles and see how it classifies them:

positive_title = news_info.loc['A', 'title'][0]
print("Title: \"{}\" has sentiment: {} \n".format(positive_title, determineSentiment(positive_title)))

neutral_title = news_info.loc['F', 'title'][0]
print("Title: \"{}\" has sentiment: {} \n".format(neutral_title, determineSentiment(neutral_title)))

negative_title = news_info.loc['GPS', 'title'][2]
print("Title: \"{}\" has sentiment: {} \n".format(negative_title, determineSentiment(negative_title)))
Title: "You Have To Love Agilent Technologies, Inc.'s (NYSE:A) Dividend" has sentiment: Positive 

Title: "Factbox: Ford and General Motors' electric vehicle plans" has sentiment: Neutral 

Title: "The Ratings Game: Gap, Banana Republic at risk as coronavirus gives shoppers one more reason to avoid the mall" has sentiment: Negative 

As you can see from the three above examples, it is capable of identifying particularly clear sentiment. However, it isn’t too good with the nuances of financial language as it is not context aware:

unclear_title = news_info.loc['GPS', 'title'][7]
print("Title: \"{}\" has sentiment: {}".format(unclear_title, determineSentiment(unclear_title)))
Title: "The Gap Inc. (GPS): These Hedge Funds Caught Flat-Footed" has sentiment: Neutral

Now let’s modify our sentiment analysis function to begin checking batches of news:

def determineSentiment(title):
    Calculates the weighted sentiment of a piece of text.
    Returns a float 'compound' which is the aggregate score of positive, 
    negative and neutral sentiment

    # VADER Polarity Score of Sentiment
    sia = SIA()
    results = []

    pol_score = sia.polarity_scores(title)
    pol_score['news_text'] = title

    return results[0]['compound']

And let’s create a function to compile the sentiment of all news for a given ticker:

def generateSentimentDict(ticker):
    Generates a dictionary with a key of YYYY-MM-DD 
    and a value of the aggregate news sentiment for that day
    daily_sentiment = {}

    articles = news_info.loc[ticker, 'title']
    for article in articles:
        sentiment = determineSentiment(article)
        #if sentiment >= 0.05 or sentiment <= -0.05:
        # Determine the date that the article was published on
        row_with_date = news_info.loc[news_info['title'] == article]
        date = row_with_date['date'][0]

        # Add the calculated sentiment to the dict of dates
        if date in daily_sentiment:
            daily_sentiment[date] += sentiment
            daily_sentiment[date] = sentiment

    return daily_sentiment

Now it’s time to gather all of our combined data and begin to plot and investigate things further.

First let’s generate a dictionary of the news sentiment for the last month in regards to the Boeing Company (BA):

boeing_sentiment = generateSentimentDict('BA')
{'2020-03-10': 0.5574, '2020-03-17': -0.3182, '2020-03-23': -1.0513, '2020-03-11': 0.5994, '2020-03-06': 0.0, '2020-03-12': -0.296, '2020-03-18': -0.6249, '2020-03-21': 0.6597, '2020-03-03': 0.296, '2020-03-20': 0.296, '2020-03-15': 0.0, '2020-03-16': 0.0, '2020-03-24': 0.0, '2020-03-04': -0.0258, '2020-03-05': 0.0, '2020-03-26': 0.34, '2020-03-25': 0.1796, '2020-02-27': 0.0}

We can now plot the sentiment as a time series:

# Sort items from dict in ascending time and plot
date,sentiment = zip(*sorted(boeing_sentiment.items()))
plt.ylabel('Sentiment Score')
[<matplotlib.lines.Line2D at 0x7f73eb37d7d0>]


Let’s plot Boeing’s share price:

from datetime import datetime

# Pull in our 20 day pricing information earlier compiled
stock_info = pd.read_sql_query("SELECT date, close FROM equities_daily WHERE ticker == 'BA'", eq_db)

# Convert the date to the datetime format for plotting
stock_info['date'] = pd.to_datetime(stock_info['date'], format="%Y/%m/%d")

date = stock_info['date']
price = stock_info['close']
plt.ylabel('Share Price')
plt.plot(date, price, color='red')
[<matplotlib.lines.Line2D at 0x7f73eb228710>]


Now let’s overlay the two time series:

# Pull in our 20 day pricing information earlier compiled
date = stock_info['date']
date = date.iloc[1:]
price = stock_info['close']
price = price.iloc[1:]

# Convert sentiment from unordered dict to datetime ordered dataframe
sentiment_df = pd.DataFrame(boeing_sentiment.items(), columns=['date', 'sentiment'])
sentiment_df = sentiment_df.sort_values(by='date', ascending=False)
sentiment = sent_pd['sentiment']

fig, ax1 = plt.subplots()

color = 'tab:red'
ax1.set_ylabel('Share Price', color=color)
ax1.plot(date, price, color=color)
ax1.tick_params(axis='y', labelcolor=color)
ax1.tick_params(axis='x', labelrotation=45)

ax2 = ax1.twinx()

color = 'tab:blue'
ax2.set_ylabel('News Sentiment', color=color)
ax2.plot(date, sentiment, color=color)
ax2.tick_params(axis='y', labelcolor=color)



Interestingly, there seems to be a common trend between the two data sets.

However, there seems to be a strange spike in sentiment between 2020-03-17 and 2020-03-21.

After delving into the sentiment values generated in the dictionary above we can see that on 2020-03-21 the sentiment was 0.6597 and on 2020-03-20 it was 0.296.

Let’s find those titles:

news_info_ba = news_info.loc['BA']
print(news_info_ba.loc[news_info_ba['date'] == '2020-03-20']['title'][0])
print(news_info_ba.loc[news_info_ba['date'] == '2020-03-21']['title'][0])
Boeing halts dividend, share buyback program amid coronavirus pandemic - MarketWatch
Outside the Box: Coronavirus bailouts are coming: Here’s the smart way to help businesses and workers

“Boeing halts dividend, share buyback program amid coronavirus pandemic - MarketWatch”

  • This is questionable in sentiment, although marked as strongly positive. In reality I’d argue this is negative as a halting of dividends indicates a poor cash situation.

“Outside the Box: Coronavirus bailouts are coming: Here’s the smart way to help businesses and workers”

  • This seems like an opinion piece, and after further investigation it is. It’s a positive sentiment but not clearly relevant.


It would seem that there is indeed use in analysing news as a trading indicator. However I do not think it should be the only factor when considering a trade.

If those two articles were removed from the dataset we’d end up with a nearly perfect overlap of our lines. However manually intervening would be a poor choice as it would be impractical in a production system. This problem should really be fixed by using a better sentiment analysis method.

There’s many ways to improve on this:

  • The analysis was during one of the most volatile markets we’ve had in many months/years
  • The analysis was a very small period in length (~ one month)
  • VADER is optimized for the sentiment of Tweets, not financial news headlines
  • News is generally lagging with respect to the price, and markets may not always respond to the news how we think they might

Ideally I’d like to have access to more news, and certainly consider training something like a Naïve Bayes model for my sentiment determiner.

To train such a model would require a considerable amount of labeled data, which would have taken a long time to compile and hand label.

As well as that, I think this strategy could work if paired with other signals to determine a longer term sentiment. Perhaps a modified turtle strategy that uses news as one of its indicators, only trading when the sentiment is positive for multiple consecutive days (or vice versa).

Thanks for reading!

Twitter, Facebook