Weekly Update (1/29/19)

with No Comments

For the past week I did:

  • Completed data collection from Alpha Vantage API
  • Started working on neural network implementation
  • Met with my advisor and went over my main argument for my project

This week I will do:

  • Keep working/complete on neural network implementation
  • Complete the outline

Weekly Update (1/22/19)

with No Comments

For the past week I did:

  • Obtained data from Alpha Vantage API
  • Updated the project timeline for the semester.
  • Prepared a topic presentation
  • Met with my advisor and went over my updated project timeline.

This week I will do:

  • Read, experiment, and implement neural network 

Weekly update

with No Comments

I have been reading more recent papers on the topic, and I came across with the paper recently published (Oct 22nd, 2018).

The paper is called “Predicting Chinese Stock Market Price Trend Using Machine Learning Approach.”

 

In summary, they implement four well-known machine learning algorithms to predict the price movement of Shangai Stock Exchange 50 index stocks after 30 days.
It is very simple but unique that they tried to predict stock movements after 30 days from the prediction day instead of just predicting tomorrow’s movement.
I think this approach is more realistic in actual investment because investors usually hold stocks for a certain period of days (except for short-term investors).
However, the shortcoming of their paper is that they only predicted whether prices will go up or down. Therefore, they only evaluated their method by directional accuracy.
Thus, I was thinking of predicting stock “prices” 30 days later instead of just predicting up or down for my project (movement).
In that case, I will be able to evaluate machine learning algorithms for long-term investment not only by directional accuracy but also closeness and trading simulation.
Given that the paper was recently published on ACM (I believe it is a credible source), I do not think anyone has done it before.

Weekly update

with No Comments

I reviewed the papers I read for my literature review, and also I collected more papers to read for my topic proposal.

Since I would like to focus on the evaluation of existing machine learning predictive models, I do not plan to come up with new algorithms.

One thing that puzzles me with all the papers I read is that they all claimed that they performed well with high prediction accuracy. But my question here is that if they really work, then why don’t people use those algorithms earn a profit?

I think there are problems with evaluation methods in these types of research and I would like to focus on that in my project.

Because the paper by Patel et al. is close to what I want to do, I am thinking of implementing the algorithms (ANN, SVM, random forest, and used in the paper and focusing more on the evaluation methods.

The paper “Natural language based financial forecasting: a survey (2017)” surveys different evaluation methods for stock prediction models. Although the paper focuses on NLP on stock prediction, the evaluation methods can be applied for my project as well.

Weekly update

with No Comments

After reading more about literature on stock prediction with sentiment analysis, I came to the conclusion that it is very difficult to obtain text data that is relevant for my research for free. I looked at Twitter streaming API and thought of start collecting data from now to next semester (3 to 4 months worth of Tweets). But the problem with Twitter streaming API is that it has request limits for free accounts and I will only be able to collect a small fraction of data.

Thus, I decided to work on the project, evaluating machine learning models for stock prediction.
The problem with previous studies is that each researcher used a different evaluation method, and thus, it becomes a challenge to come to a definite conclusion about which approach is the best. Moreover, evaluating a strategy with one criterion does not give an accurate assessment of a strategy. For example, an algorithm may have high directional accuracy, but it does not mean that the strategy can make a greater profit.
Therefore, I will backtest each machine learning algorithm with three different backtesting metrics: accuracy, closeness, and trading simulation.

Weekly update

with No Comments

For the past week, I worked on literature reviews on two topics: stock prediction with natural language processing and stock prediction with machine learning.

One of the most important findings is that each researcher used different datasets to predict different prediction targets. Thus, it becomes a challenge to come to a definite conclusion on which approach is the best. For future research, backtesting different approaches on the same datasets over multiple stocks or indexes will be necessary to fairly compare the performance of different algorithms.

Weekly update

with No Comments

For the past week, I did the second reading for the 15 papers for the annotated bibliography and also prepared the presentation for the class.

I narrowed down my topics into two: stock prediction with sentiment analysis and stock prediction with machine learning. Major findings are the followings:

  1. Researchers use different sources for texts (twitter, news article, other social media etc)
  2. Researchers use different prediction targets (market movement, individual stocks, etc)

I will be working on literature reviews for this week.

Weekly update

with No Comments

After skimming through 15 papers over three different topics, I am still most interested in the topic of “generate sentiment-based stock trading signals through NLP.” Since last week, I started taking a Coursera course called applied text mining in Python. I believe the skills I will acquire in this course will be helpful for my senior capstone project next semester if I decide to do the topic above. I also contacted a CS senior from last year who have done his senior capstone project in NLP. He kindly shared his senior capstone paper as well as some helpful resources for my research.

Weekly Update

with No Comments

For the past week, I did the following:
– Found research papers on my topics
– Skimmed through several papers and gained the better understandings of the topics
– After skimming through papers on my possible three topics, I am most interested in the topic “generate sentiment-based stock trading signals through NLP.”

For this week, I plan to do the followings:
– Start reading papers with greater care
– Start building an annotated bibliography

Weekly Update

with No Comments

I talked with Dave about my senior project ideas as I had a concern that former students have done similar research previously.
The followings are some of the takeaways from the discussion:

  • Finding a good niche within a field is a key. I should utilize my background in finance to do so.
  • Having broad ideas where I could differentiate my project before delving into research is helpful.
  • But I do not need to stick to the initial idea if I find something better along the way.

For this week, I plan to do the followings:

  • Read several papers on my three ideas and get the better understanding of what have done in the field.
  • Brainstorm some niche fields where I could apply ML into the stock prediction

 

Project ideas

with No Comments

Idea 1: A web application that lets users test the performance of systematic trading strategies with user selected parameters.
The purpose of this application is to let users try out different inputs and test how his/her trading strategy would have worked on historical data. For trade signals, I am thinking of using simple moving average and several machine learning methods. The result will be shown visually in graphs and tables.
Through this project, I will be able to learn the technical aspects of machine learning as well as how to build an interactive web application.

Idea 2: A Study of different machine learning methods in stock price predictions.
In this project, I aim to learn several machine learning methods (probably several different neural network methods) and analyze the performance of different strategies. Unlike idea 1 above, I do not intend to make a web application for this project. Thus, I plan to go deeper in learning the technical parts of machine learning methods.

Idea 3: Stock trend prediction through sentiment analysis using Natural Language Processing.
In this project, I aim to learn the fundamental of NLP, generate sentiment-based trade signals, and test the performance of the strategy. Similar to the idea 2 above, I do not intend to make a web application for this project. Thus, I will go deeper in learning technical of NLP.