Weekly update (03/06/19)

with No Comments

For the past week, I worked on implementing the trading simulation. It has been taking more than what I thought but I plan to complete it by the end of this week.


I was not able to work as much as I hoped due to the other commitments I had before the spring break. But as I plan to stay on campus over the break, I will try to complete the implementation part as much as possible and will complete the initial draft due during the break.

Weekly update (02/26/19)

with No Comments

For the past week, I primarily worked on finishing up the partial draft of my paper.

I reread some of the relevant papers for my research and modified the related work section so that the connection between my research and each cited work is more clear.

Also in my topic proposal, I had a huge list of related works. However, in my partial draft for the capstone paper, I only cited the papers which are most important for my research and modified them to fit into the new format.

Weekly update (2/19/19)

with No Comments

For the past week I:

  • Limited the training data to consider only events after the crises in 2008
  • Tried to predict the value for after 1 to 20 days and examined how accuracies changed
  • Started working on trading simulation (Bollinger band)

This week I will:

  • Complete the partial draft, which is due Feb 25th
  • Complete implementing the trading simulation

Weekly Update (2/13/19)

with No Comments

For the past week I:
– Implemented MLP ‘Regressor’ instead of classifier. The accuracy increased significantly.
– Wrote functions to generate financial indicators from close prices
– Implemented lag

This week I will do:
– Limit the training data to consider only events after the crises in 2008.
– prediction after 5 days is with accuracy +-2.5%, I will predict 10 days and see how accurate the predictions are.
– selection approach for simulator. When simulator is going to decide to sell or buy?
– explanation / justification of your design choices for simulator.

Weekly Update (2/6/19)

with No Comments

For the past week I:

  • Completed the outline
  • Worked on neural network implementation
  • Thought about trading strategy

This week I will do:

  • Complete on MLP implementation
  • Start working on the paper (background, motivation)

Weekly Update (1/29/19)

with No Comments

For the past week I did:

  • Completed data collection from Alpha Vantage API
  • Started working on neural network implementation
  • Met with my advisor and went over my main argument for my project

This week I will do:

  • Keep working/complete on neural network implementation
  • Complete the outline

Weekly Update (1/22/19)

with No Comments

For the past week I did:

  • Obtained data from Alpha Vantage API
  • Updated the project timeline for the semester.
  • Prepared a topic presentation
  • Met with my advisor and went over my updated project timeline.

This week I will do:

  • Read, experiment, and implement neural network 

Weekly update

with No Comments

I completed the 2nd draft of my topic proposal.

In particular, I added and fixed the followings:

  • A table that summarises what evaluation methods are used in previous work
  • Added reference links to all the tables and figures
  • Fixed all the comments I received on my first draft

Weekly update

with No Comments

After reviewing my literature review and the papers I have read previously in detail, I found out that not many researchers predict stock prices but instead they just predict the price trend (whether up or down).
Thus, I have been reading more papers and trying to find the detail implementation of two machine learning algorithms (ANN and SVR).
For the coming week, I will be reading more papers and improving my topic proposal.

Weekly update

with No Comments

I have been reading more recent papers on the topic, and I came across with the paper recently published (Oct 22nd, 2018).

The paper is called “Predicting Chinese Stock Market Price Trend Using Machine Learning Approach.”

 

In summary, they implement four well-known machine learning algorithms to predict the price movement of Shangai Stock Exchange 50 index stocks after 30 days.
It is very simple but unique that they tried to predict stock movements after 30 days from the prediction day instead of just predicting tomorrow’s movement.
I think this approach is more realistic in actual investment because investors usually hold stocks for a certain period of days (except for short-term investors).
However, the shortcoming of their paper is that they only predicted whether prices will go up or down. Therefore, they only evaluated their method by directional accuracy.
Thus, I was thinking of predicting stock “prices” 30 days later instead of just predicting up or down for my project (movement).
In that case, I will be able to evaluate machine learning algorithms for long-term investment not only by directional accuracy but also closeness and trading simulation.
Given that the paper was recently published on ACM (I believe it is a credible source), I do not think anyone has done it before.

Weekly update

with No Comments

I worked on the first draft of my topic proposal for the past week.

I decided to evaluate stock predictive models by ANN and SVM based on three different metrics: accuracy, closeness, and trading simulation.

Five years of data of two stock price indices (S&P 500 and DJIA) and two stocks (Apple and Amazon) from January 2013 to December 2018 is used in the project. All the data is obtained from Alpha Vantage API. The data collected from Alpha Vantage API is already available in an organized CSV format. The collected data will be pre-processed to generate ten technical financial indicators used as inputs into ANN and SVM model.

For the coming week, I will start learning the basics of ANN.

Weekly update

with No Comments

I reviewed the papers I read for my literature review, and also I collected more papers to read for my topic proposal.

Since I would like to focus on the evaluation of existing machine learning predictive models, I do not plan to come up with new algorithms.

One thing that puzzles me with all the papers I read is that they all claimed that they performed well with high prediction accuracy. But my question here is that if they really work, then why don’t people use those algorithms earn a profit?

I think there are problems with evaluation methods in these types of research and I would like to focus on that in my project.

Because the paper by Patel et al. is close to what I want to do, I am thinking of implementing the algorithms (ANN, SVM, random forest, and used in the paper and focusing more on the evaluation methods.

The paper “Natural language based financial forecasting: a survey (2017)” surveys different evaluation methods for stock prediction models. Although the paper focuses on NLP on stock prediction, the evaluation methods can be applied for my project as well.

Weekly update

with No Comments

After reading more about literature on stock prediction with sentiment analysis, I came to the conclusion that it is very difficult to obtain text data that is relevant for my research for free. I looked at Twitter streaming API and thought of start collecting data from now to next semester (3 to 4 months worth of Tweets). But the problem with Twitter streaming API is that it has request limits for free accounts and I will only be able to collect a small fraction of data.

Thus, I decided to work on the project, evaluating machine learning models for stock prediction.
The problem with previous studies is that each researcher used a different evaluation method, and thus, it becomes a challenge to come to a definite conclusion about which approach is the best. Moreover, evaluating a strategy with one criterion does not give an accurate assessment of a strategy. For example, an algorithm may have high directional accuracy, but it does not mean that the strategy can make a greater profit.
Therefore, I will backtest each machine learning algorithm with three different backtesting metrics: accuracy, closeness, and trading simulation.

Weekly update

with No Comments

For the past week, I worked on literature reviews on two topics: stock prediction with natural language processing and stock prediction with machine learning.

One of the most important findings is that each researcher used different datasets to predict different prediction targets. Thus, it becomes a challenge to come to a definite conclusion on which approach is the best. For future research, backtesting different approaches on the same datasets over multiple stocks or indexes will be necessary to fairly compare the performance of different algorithms.

Weekly update

with No Comments

For the past week, I did the second reading for the 15 papers for the annotated bibliography and also prepared the presentation for the class.

I narrowed down my topics into two: stock prediction with sentiment analysis and stock prediction with machine learning. Major findings are the followings:

  1. Researchers use different sources for texts (twitter, news article, other social media etc)
  2. Researchers use different prediction targets (market movement, individual stocks, etc)

I will be working on literature reviews for this week.

Weekly update

with No Comments

After skimming through 15 papers over three different topics, I am still most interested in the topic of “generate sentiment-based stock trading signals through NLP.” Since last week, I started taking a Coursera course called applied text mining in Python. I believe the skills I will acquire in this course will be helpful for my senior capstone project next semester if I decide to do the topic above. I also contacted a CS senior from last year who have done his senior capstone project in NLP. He kindly shared his senior capstone paper as well as some helpful resources for my research.

Weekly Update

with No Comments

For the past week, I did the following:
– Found research papers on my topics
– Skimmed through several papers and gained the better understandings of the topics
– After skimming through papers on my possible three topics, I am most interested in the topic “generate sentiment-based stock trading signals through NLP.”

For this week, I plan to do the followings:
– Start reading papers with greater care
– Start building an annotated bibliography

Weekly Update

with No Comments

I talked with Dave about my senior project ideas as I had a concern that former students have done similar research previously.
The followings are some of the takeaways from the discussion:

  • Finding a good niche within a field is a key. I should utilize my background in finance to do so.
  • Having broad ideas where I could differentiate my project before delving into research is helpful.
  • But I do not need to stick to the initial idea if I find something better along the way.

For this week, I plan to do the followings:

  • Read several papers on my three ideas and get the better understanding of what have done in the field.
  • Brainstorm some niche fields where I could apply ML into the stock prediction

 

Project ideas

with No Comments

Idea 1: A web application that lets users test the performance of systematic trading strategies with user selected parameters.
The purpose of this application is to let users try out different inputs and test how his/her trading strategy would have worked on historical data. For trade signals, I am thinking of using simple moving average and several machine learning methods. The result will be shown visually in graphs and tables.
Through this project, I will be able to learn the technical aspects of machine learning as well as how to build an interactive web application.

Idea 2: A Study of different machine learning methods in stock price predictions.
In this project, I aim to learn several machine learning methods (probably several different neural network methods) and analyze the performance of different strategies. Unlike idea 1 above, I do not intend to make a web application for this project. Thus, I plan to go deeper in learning the technical parts of machine learning methods.

Idea 3: Stock trend prediction through sentiment analysis using Natural Language Processing.
In this project, I aim to learn the fundamental of NLP, generate sentiment-based trade signals, and test the performance of the strategy. Similar to the idea 2 above, I do not intend to make a web application for this project. Thus, I will go deeper in learning technical of NLP.