Bringing Innovative Load Balancing to NGINX

with No Comments

Senior Capstone: Bringing Innovative Load Balancing to NGINX


Load balancing remains an important area of study in computer science largely due to the increasing demand on data centers and webservers. However, it is rare to see improvements in load balancing algorithms implemented outside of expensive specialized hardware. This research project is an attempt to bring these innovative techniques to NGINX, the industry leading open source load balancer and webserver. In addition to implementing a new, native NGINX module, I have developed a simple workflow to benchmark and compare the performance of available load balancing algorithms in any given production environment. My benchmarks indicate that it is possible to take advantage of more sophisticated load distribution techniques without paying a significant performance cost in additional overhead.

My Software Architecture


Final Project

with No Comments



My Senior Project will be based on improving the accuracy of machine learning algorithms by using various statistical methods to correctly pre-process a dataset and then feed it into a machine learning algorithm.

The software part of my project will be based on designing a platform that works with R studio to run these tests on a dataset and then feed it to a machine learning algorithm and then analyze the results. This software recommends the series of ‘knobs’ that can turned on the dataset to better it for the algorithm. My paper will be based on the results that I found and whether or not pre-processing made a difference in the performance of these algorithms.

Update April 13

with No Comments

Steadily working on the prose of the survey paper. Got the feedback from Charlie and incorporating those. No major changes yet. Reading some of the failed cases of p2p/mesh mobile/wireless implementation to understand and find problems with appropriate scope.

Update April 6, 2017

with No Comments

I’m looking into different implementation of peer to peer technologies and trying to understand what and where they lack in comparison to traditional implementation. Reading about firebase API as well, which powers app like firechat.

Survey Paper

with No Comments

4/12 Continuing Work. Going to finish explaining current sources before I add new ones.

4/5 Finishing prose in first draft, but the outline is done. Most of the work is gonna be in the area of explaining each of the technical sources.

Steadily working on my paper. I’ve been looking at the paper on linguistics especially; its good material. I’m thinking of separating it into two parts: Structural Analyis, and Signal Analysis. I think i have sources for both.

Survey Paper

with No Comments

Topic 1 : Data Mining Analysis and Prediction 

Week 1: March 12- 18

Went into more detail on the annotated bibliographies, organized the order in which would will help my paper and best fit the flow of ideas.

Week 2:   March 19 – 25

Looked into what the data mining tools and application that are mentioned in the papers. Checked if they could fall within the scope of the work that I want to do. Created the overall outline for my paper including how the major topics and methodology would progress. Created the workflow diagram and looked at other previously done Survey paper. Latex seems like a good tool to use in addition to Zotero to create templates.

Watched some TED talk videos on the topic:

Aaron Koblin: Visualizing ourselves … with crowd-sourced data

Hans Rosling: The best stats you’ve ever seen

Mathias Lundø Nielsen :How to Monetize Big Data

Week 3: March 26 – 31

Started connecting major topics in terms of how they fit the block structures for my paper and compiled paragraphs on some topics. Looked into other previously done works mentioned in the papers regarding Data Mining and the tools used in those research.


Week 4: April 1- 7 

Building on the outline and creating diagrams mentioned. Mostly going through papers to build on the brief few sentences mentioned for each topic.

Week 5: April 7 – 14

Worked on the second draft. Added more content to the paper, removed a couple of subtopic.

Week 6: April 14 -21 

Finished up the survey paper with all necessary topics and figures and diagrams as well as the conclusion.

Survey Paper

with No Comments

Week 1 March 30th

I have gone ahead and read more carefully about the chosen topic from my bibliography. Furthermore, I’m looking for some more papers to add to the bibliography list. I also have the general outline for the survey paper ready, and waiting to add more details.


Week 2 April 6th

  • I have skimmed through a few more papers to try to find something to add to the bibliography.
  • Try to improve the overall structure of the paper, haven’t been able to
  • Read carefully the papers already in the bibliography. Some of which doesn’t fit in the current context and needs to be replaced/removed.

Week 3 April 12th

  • Sit and think of some other projects that might befit this
  • Find papers about Cloud computing on mobile devices
  • Find some more paper about 3D augmented rendering on mobile devices
  • Find a good tutorial on beginner augmented reality (place a 3D cube on a spot with a mark)

Week 4 April 19th

  • Add a few articles to the reference list
  • Update cloud computing
  • Finish up the 2nd draft

Week 5 April 26th

  • Continue the paper
  • Looking at AR examples

Survey Paper

with No Comments

Week 1 (3/29):

  1. I have found a reasonable amount of papers to read.
  2. I have refined the topic. The previous topic was about analyzing and find the relationship between weather pattern and oil stock price. I have decided to broaden the topic to find the relationship between two general entities (so not just weather pattern and oil stock price) by developing an algorithm for similarity scoring and matching.
  3. I have a basic outline for the survey paper jotted down.

Week 2 (4/5):

  1. A lot of readings, especially about SVM (Support Vector Machine)
  2. Work on adding some more text in the introduction part
  3. Read, read, and read.
  4. Explore the term artificial neural networks, etc.

Week 3 (4/12):

  1. Read about different approach to pattern matching problem in other area of study (BLAST, etc.)
  2. Read about AI neural networks, it’s confusing.
  3. Start adding meat to the outline of the survey paper.

Week 4 (4/19):

  1. Watch the MIT’s Opencourseware Intro to AI
  2. Modular Neural Network – Reading
  3. SVM readings

Week 5 (4/26):

  1. Continue the course
  2. Re-learn calculus

Annotated Bibliography

with No Comments


1/ Real Time Face Detection

M. Sharif, K. Ayub, D. Sattar, M. Raza

Real Time Face Detection method by changing the RGB color space to HSB color space and try to detect the skin region. Then, try to detect the eye of the face in the region. The first step is face detection, second step is face verification. Time is crucial since it is real time. The published time is 2012.

2/ FaceNet: A Unified Embedding for Face Recognition and Clustering:

A paper describing the method for face recognition using FaceNet system. FaceNet is a system by Google that allows high accuracy and speed in face detection mechanism. The accuracy of FaceNet system is 99.63% on the widely used Labeled Faces in the Wild (LFW) dataset.

Two Implementations of FaceNet for Face Recognition:


Virtual Space:

1/ Towards Massively Multi-User Augmented Reality on Handheld Devices

Develop a framework for implementing augmented reality interface on hand-held devices. There’s a graphical tool for developing graphical interfaces called PocketKnife, a software renderer called Klimt and a wrapper library that provides access to network sockets, threads and shared memory. In the end, they develop several AR games with the framework, such as the invisible train game.



1/ Financial time series forecasting using support vector machines

Using support vector machine and compare the results with other methods of forecasting. The upper bound C and the gamma kernel parameter play an important role in the performance of SVMs. The prediction performance may be increased if the optimum parameters of SVM are selected.

C parameter is the parameter for how small will the hyperplane of largest minimum margin be.



Annotated Bibliographies

with No Comments


T1-  Data Mining, analysis and prediction 

Topp, N., & Pawloski, B. (2002). Online Data Collection. Journal of Science Education and Technology, 11(2), 173-178.

This paper touches on the history online data collection, some brief review of the more recent progress and work that is being done as well as how a database connected to the Internet collects data. It also presents a brief insight into where these methods might head towards in the future. Overall, this is a short 7-page article to give a good insight and a starting point as well good references.


Hand, D., Blunt, G., Kelly, M., & Adams, N. (2000). Data Mining for Fun and Profit. Statistical Science, 15(2), 111-126.

This is a more detailed paper regarding the different tool, models, patterns and quality of data mining. Even though it was written in 2000 is very useful is terms of getting a broader idea of model building and pattern detection. It looks at statistical tools and their implementation as well as the challenges to data mining through well explained examples and graphs.


Edelman, B. (2012). Using Internet Data for Economic Research. The Journal of Economic Perspectives, 26(2), 189-206.

Economist have always been keen to collect and analyze data for their research and experimentation. This paper introduces how data scraping has been employed by companies and businesses to extract data for their use. It is an excellent paper that combines data scraping with data analysis and where and how it has been used. It sets the foundation for data analysis and lists various other good papers in the particular field.



Buhrmester, M., Kwang, T., & Gosling, S. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5.

Amazon’s Mechanical Turk helps bring together a statistician’s dream of data collection and an economist’s love for data analysis. It has proved to be an excellent platform to conduct research in not only economics but also psychology and other social sciences. This is a very short 4 page paper that looks at the mechanical Turk, what it has helped research and conclude and how it has been used to obtain high quality inexpensive data. This paper is significant in a sense that it is an application of the above-mentioned tools of collection, analysis and possibly prediction.


T2- A more informed Earlham : Interactive Technology for Social change

1/ Vellido Alcacena, Alfredo et al. “Seeing Is Believing: The Importance of Visualization in Real-World Machine Learning Applications.” N.p., 2011. 219–226. Web. 20 Feb. 2017.

2/ “And What Do I Do Now? Using Data Visualization for Social Change.” Center for Artistic Activism. N.p., 23 Jan. 2016. Web. 20 Feb. 2017.

3/ Valkanova, Nina et al. “Reveal-It!: The Impact of a Social Visualization Projection on Public Awareness and Discourse.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM, 2013. 3461–3470. ACM Digital Library. Web. 20 Feb. 2017. CHI ’13.


T3– CS for all : Learning made easy.

1/  Muller, Catherine L., and Chris Kidd. “Debugging Geographers: Teaching Programming To Non-Computer Scientists.” Journal Of Geography In Higher Education 38.2 (2014): 175-192. Academic Search Premier. Web. 20 Feb. 2017

2/ Rowe, Glenn, and Gareth Thorburn. “VINCE–An On-Line Tutorial Tool For Teaching Introductory Programming.” British Journal Of Educational Technology 31.4 (2000): 359. Academic Search Premier. Web. 20 Feb. 2017.

3/  Cavus, Nadire. “Assessing The Success Rate Of Students Using A Learning Management System Together With A Collaborative Tool In Web-Based Teaching Of Programming Languages.” Journal Of Educational Computing Research 36.3 (2007): 301-321. Professional Development Collection. Web. 20 Feb. 2017.

Annotated Bibliography

with No Comments
  1. Fake news:
    1. Shao, Chengcheng, et al. “Hoaxy: A platform for tracking online misinformation.” Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 2016.
    2. Castillo, Carlos, et al. “Know your neighbors: Web spam detection using the web topology.” Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2007.
    3. Boididou, Christina, et al. “Challenges of computational verification in social multimedia.” Proceedings of the 23rd International Conference on World Wide Web. ACM, 2014
  2. Peer-to-peer platforms:
    1. Daswani, Neil, Hector Garcia-Molina, and Beverly Yang. “Open problems in data-sharing peer-to-peer systems.” International conference on database theory. Springer Berlin Heidelberg, 2003.
    2. Ripeanu, Matei. “Peer-to-peer architecture case study: Gnutella network.” Peer-to-Peer Computing, 2001. Proceedings. First International Conference on. IEEE, 2001.
    3. Hinz, Lucas. “Peer-to-peer support in a personal service environment.” Master of Science Thesis, Uppsala University, Uppsala, Sweden (2002).
  3. Browser Fingerprinting (possibility of going into cyber-security and related branches):
    1. Eckersley, Peter. “How unique is your web browser?.” International Symposium on Privacy Enhancing Technologies Symposium. Springer Berlin Heidelberg, 2010.
    2. Nikiforakis, Nick, et al. “Cookieless monster: Exploring the ecosystem of web-based device fingerprinting.” Security and privacy (SP), 2013 IEEE symposium on. IEEE, 2013.
    3. Acar, Gunes, et al. “FPDetective: dusting the web for fingerprinters.” Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security. ACM, 2013.

Final Abstracts list

with No Comments

T1–  ::Data Mining, analysis and prediction 

This survey paper will first look at the tools used to gather and store data from user and other domains. It will then look at how, in the past, others have worked with data to make co-relations and predictions. It will then look attempt to look at publicly available data and try to find correlation with other market data. Our focus here will be to see the extent to which one data can be abstractly analyzed and linked to others and with what degree of certainty. It will involve working with a lot of data and analyzing it to find trends and patterns and possibly making predictions.


Topic 2 – CS for Social Change and Sustainability


Every year the different branches of campus such as Health Services, facilities, Public Safety, ITS and the registrar’s office send out emails to students that are lengthy reports which no one ever reads. Earlham facilities keep records on energy consumption that the students seldom look at and every now and then there are issues around campus that divides the student body but students rarely get to vote on.

To address these problems I suggest a mobile survey app that allows students to vote on issues as well as view various data from departments around the campus. These survey results and data will also be dynamically displayed on screens around the campus. It would involve learning and implementing graphic interface tools as well as visualization programs. If we link this through quadratics (as is done for student government voting), we can make sure that only Earlham students get to vote and each student gets to vote only once.

The ability to view data and trends on key statistics across from these departments would certainly help the students in a better-informed position and in a place to bring change.


T3 – CS for all

As I see my Econ professors struggle with STATA (a simple tool to work with data through commands), I cannot help but draw parallels on how it first felt to learn programming. Reality is that most people without a CS background have difficulty in learning these new tools and softwares. Softwares, most of which are outdated in their use, but, are still taught to students who usually resort to memorizing them to pass midterms. I think that it would be very helpful if we as CS students can help discover, learn, teach as well as document these softwares and help other departments. I propose an interactive interface like Code-academy where students are given tutorials that go progressively forward in complexity. Co-ordination from these departments would be essential to understand their needs and create an interface catered to help their students learn from scratch.


{ possible additions could be log-in mechanism via moodle to ensure students are spending the amount of time they should be taking these interactive courses”}



Annotated Bibliographies

with No Comments
I/ Econ Simulation Game
1) “Educational Video Game Design: A Review of the Literature – Semantic Scholar.” N.p., n.d. Web. 16 Feb. 2017.
2) Squire, Kurt. “Changing the Game: What Happens When Video Games Enter the Classroom.” Innovate: journal of online education 1.6 (2005): n. pag. Web. 16 Feb. 2017.
3) —. “Video Games in Education.” International Journal of Intelligent Simulations and Gaming 2 (2003): 49–62. Print.
II/ Social Network Data Mining
1) Cheong, France, and Christopher Cheong. “Social Media Data Mining: A Social Network Analysis Of Tweets During The 2010-2011 Australian Floods.” (2011): n. pag. Web. 16 Feb. 2017.
2) Kempe, David, Jon Kleinberg, and Éva Tardos. “Maximizing the Spread of Influence Through a Social Network.” Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: ACM, 2003. 137–146. ACM Digital Library. Web. 16 Feb. 2017. KDD ’03.
3) Wu, X. et al. “Data Mining with Big Data.” IEEE Transactions on Knowledge and Data Engineering 26.1 (2014): 97–107. IEEE Xplore. Web.
III/ Connection between stocks
Zager, Laura A., and George C. Verghese. “Graph Similarity Scoring and Matching.” Applied Mathematics Letters 21.1 (2008): 86–94. ScienceDirect. Web.

Top 3s

with No Comments

Project Idea:

Real-time visualisation of point clouds through android device

Project Idea:

P2P Git: a decentralised version of git version control

Project Idea:

Automatic exposure, shutter speed, ISO, and aperture algorithm implementation for digital cameras.


Capstone Abstracts – v1

with No Comments

Abstract 1

Recently I became interested in P2P messaging and/or protocols. While these protocols can offer security and prevent wiretapping (for example, bitmessaging), there are some serious drawbacks. For one, decentralization is difficult to achieve while maintaining the advantages of a centralized server, which provides major shares of benefits of client-server model. Even if decentralization is achieved, the architectures turns out to be not so well for scalability. I haven’t identified what exactly I am going to work on, but focusing on an aspect that makes the P2P protocols more robust is my motivation behind the project.


Abstract 2

It’s a widespread belief that fake news has played a noteworthy roles in shaping the voters pick for the US presidential candidate in the election cycle 2016. Fact checking, and thus weeding out fake news is one of the most difficult challenges that technology can take on; however, it’s unlikely for a set of algorithm to match the accuracy of a human fact checker, as of today. In this paper, we examine how natural language processing can help finding patterns in dubious claim as opposed to stories that are factually consistent. Employing artificial intelligence agent, we are able to show that a “true story” is supported by several sources and report the same event/fact, while a fake news story is likely reported from a single source and gets circulated. In addition to that, we’ll also examine how AI can be used to detect the extent to which a story is verifiable, which is a key characteristic of a credible story.

Abstract 3

When a device is connected to the internet, a combination of several data points uniquely identify a machine, which is known as browser fingerprinting. Advertisers and marketers use cookies in order to target potential customers, and it is very easy to abuse those tools and it leaves any device connected to the internet vulnerable to attacks. We’ll investigate the uniqueness of browser fingerprinting briefly and examine the impact of a single data point in determining the uniqueness of a fingerprint. In doing so, we’ll analyse the privacy aspect of an user and ways to achieve the security, anonymity and how the anonymity impacts the connectivity of a device.

Capstone Abstracts- V1

with No Comments


T1 – One data predicts another

This survey paper will look at publicly available data and try to find correlation with other market data. For example, it would study how weather patterns or viral news stories could correlate to stock prices for certain stocks. It will try to see to what extent one data can be abstractly analyzed and linked to others with what degree of certainty. It will involve working with a lot of data and analyzing it to find trends and patterns.

Possible Ideas to build on: 

— > Will be looking into Behavioral economics and how certain events follow another. Using this, I will look for places to extract co-related data. 

— > Will involve a fair bit of learning STATA to work on data and derive co-relations. Some statistical modeling would be helpful. 

—> Stock market data is usually well kept however similar day to day data is rarely seen in other places. One possible topic being finding co-relations is to look in unusual places within the stock markets. for example: Boeings stocks might be brought down by President Trump’s tweets but what other markets have shown unusual reactions to his tweets. Perhaps a comparison of market changes with key words in tweets of with the most popular people on twitter on that area. 


T2- Computers, data and everything else.

This survey paper will look at how the trends and tools of data analysis have changed within the stock markets and particularly with the field of Economics. Infamously labelled “the dismal science”, economist are only now able to collect and manipulate data to prove their theories. It will look at how data analysis because of modern computing is affecting other fields.

Possible Ideas to build on: 

—> Databases used in the stock markets and how they have eased day to day operations. 

—> Other popular mass scale data collection tools and how development in computing has changed their workings. { This would be more of a history digging up, I would look up how and why the successors were picked over their predecessors.}} 

—> Some bits of this project could be used on the first idea. 


T3 – Data Mining

This survey paper looks at how and what data is being extracted from users and in what ways companies are storing and profiting from it. It looks at targeted advertisements, cyber security, the algorithms working in the background and the databases that sell our data.

Possible Ideas to build on: 

—> Look into tools of data mining. The use of cookies and pop up ads and data extraction from search bars. How are these companies getting smarter every day, what loopholes in are they employing.  How they create a virtual personal of you based on what they know about you so far. 

—> Learn how the government has in the past used data from social security and taxes to analyze various sociological aspects. Where else has such data analysis existed within the computer science. How can the two be related ? 


Capstone Abstracts – v1

with No Comments

I/ Sometimes lectures and text books can be too “dry” for students to get excited about a subject, specifically economics. At the same time, researchers have found the potential of games in education, especially when used as an introduction to new concepts. EconBuild is a game that simulates different aspects of economics that we normally encounter in our economics intro classes, proving students a platform to practice what they learn in class. The game can help students to enforce the most fundamental elements of economics such as demand and supply, stock market, etc.

II/ In this day and age, more and more businesses choose to expand their brand using social networks, thus leading to the fact that social media users continue to provide advertisement, positive and negative. In order to become competitive, it is necessary for a company to establish its online present as well as analyze its component’s dominance. Using a Hadoop based approach to reduce the size of database, we can gather and analyze information about a company on social media and predict certain trends to help with its growth.

III/ Stock market is usually unpredictable. There is no particular rule that it obeys to, which is why investing in stock is considered a risky business. Many people have tried to analyze particular trends in order to guess whether the stock price would rise or not. However there hasn’t been a lot of software that analyze the relationship between different related stocks. Using support vector machine approach, combining with graph similarity scoring and matching algorithm, we can establish relationships between different stocks, thus open the possibility of being able to predict particular stock trends.

Capstone Abstracts – v1

with No Comments

This paper will describe a project created using support vector machines (SVM) to predict stock price. Since the method is support vector machines, the data must be labeled, which fits what needed for stock evaluation. Stock’s information comes from its financial statements, which are all labeled. In this particular project, the version of SVM is a machine called least square support vector machines, which are used for regression analysis. The language being used is Python with scikit-learn, which has SVM implemented in the library.

This paper will describe a project using augmented reality (AR). AR is a live direct or indirect view of a physical, real world environment augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. For this particular project, I will use Swift to implement a iOS app to provide users a augmented reality graphical view with supplemented GPS information. The application will take the user’s location and give additional information about the POI around the areas on the phone when the POI shows up.

This paper will describe a project using Machine Learning for Real-time Face Detection and Recognition using the mobile’s camera and compare the result to college’s student database. The paper will allow people to connect easily by knowing the name, location and mobile number with just a look on the phone. The program will run on iOS and Android using Cordova as a base.

“Hello World”

with No Comments

This means that I have my WordPress page up and running. I will be posting updates on this site on my Senior Capstone project. That also made me suddenly realize my college career has begun its ending! whoops.

Last Program Update

with No Comments

Yesterday, I just changed the name of the executable file to “konductor” simply because I wanted the command to be a little more descriptive over just “main,” and all related files have also been updated to reflect the change. Unless something else happens, this will probably be my last program update for the semester.

Meanwhile, today is all about finishing my 2nd draft so that I can work on other projects due tomorrow.

Progress on Paper and Program

with No Comments

Today, I tried getting rid of the sleep function from OpenCV that watches when a key has been pressed so that the program can quit safely. This is the only sleep function left in the program that could possibly interfere with the timekeeping functions in the program, and keyboard control is not necessarily a key part of the program either. However, I couldn’t find any other alternatives in the OpenCV reference; I was looking for a callback function that is invoked when a window is closed, but that doesn’t seem to exist in OpenCV.

Nevertheless, I have been continuing to work on my paper and add the new changes to it. Although, I don’t know if I’ll have time to finish testing the program before the poster is due on Friday.

Work Update [12/1 – 12/4]

with No Comments

I have been extensively working on improving the front end of Robyn’s web interface and the Natural Language Processing aspect of Robyn. Trying to give some finishing touches to Robyn before the demo this Wednesday.

Demo Video and Small Update

with No Comments

Before I forget, here’s the link to the video of the demo I played in my presentation. This may be updated as I continue testing the program.

Today’s update mainly just consisted of a style change that moved all of my variables and functions (except the main function) to a separate header file so that my main.c file looks all nice and clean. I may separate all of the functions into their own groups and header files too as long as I don’t break any dependencies to other functions or variables.

Progress Update

with No Comments

Over Thanksgiving break I have learned how to compile and modify the software for ffmpeg in Cygwin, a virtual Unix environment.  This makes the process of modifying and compiling the code easier.  Furthermore, I have been developing and working on implementing algorithms to use the conclusions from my experiment better choose how many keyframes to allocate for different types of videos.  I have also made the full results of my experiments available here.

Work Log[11-27 to 11-30]

with No Comments

I did not do much work during Thanksgiving since I was travelling a lot. However, after coming back from the break I have spent a good amount of time everyday working on my senior sem project. I have especially been working making a web interface for the python script of Robyn. Since I had not done much work with python web frameworks, I had to learn quite a bit about these. After looking at several options like Django, CGI, Flask and Bottle, I decided to use Bottle since it has a relatively low barrier to learning and since it is light weight, which is sufficient for Robyn and makes the system easier to setup as well. Today I finished the plumbing for the web interface, as well as the front end for Robyn.

Now, for the next couple of days I will work on the NLP part of Robyn.

Post-Presentation Thoughts

with No Comments

Today’s program update was all about moving blocks of code around so that my main function is only six lines long, while also adding dash options (-m, -f, -h) to my program as a typical command would.

Looking through the XKin library for a little bit and after having done my presentation, I’ll soon try to implement XKin’s advanced method for getting hand contours and see if that helps make the centroid point of the hand, and thus the hand motions, more stable. In the meantime, though, my paper is in need of major updates.

Before Testing

with No Comments

I forgot that I also needed to compose an actual orchestral piece to demo my program with, so that’s what I did for the entire evening. I didn’t actually compose an original piece, but rather just took Beethoven’s Ode to Joy and adapted it for a woodwind quintet (since I don’t have time to compose for a full orchestra). Soon, I’ll actually test my program and try to improve my beat detection algorithm wherever I can.

I also tried adding the optional features into my program for a little bit, but then quickly realized they would take more time than I have, so I decided to ditch them. I did, however, add back the drawing of the hand contour (albeit slightly modified) to make sure that XKin is still detecting my hand and nothing but my hand.

Core Program Complete!

with No Comments

Yesterday was a pretty big day as I was able to implement FluidSynth’s sequencer with fewer lines of code than I originally thought I needed. While the program is still not completely perfect, I was able to take the average number of clock ticks per beat and use that to schedule MIDI note messages that do not occur right on the beat. I may make some more adjustments to make the beat detection and tempo more stable, but otherwise, the core functionality of my program is basically done. Now I need to record myself using the program for demo purposes and measure beat detection accuracy for testing purposes.

Two other optional things I can do to make my program better are to improve the GUI by adding information to the window, and to add measure numbers to group beats together. Both serve to make the program and the music, respectively, a little more readable.

Learning to Schedule MIDI Events

with No Comments

The past few days have been pretty uneventful, since I’m still feeling sick and I had to work on another paper for another class, but I am still reading through the FluidSynth API and figuring out how scheduling MIDI events work. FluidSynth also seems to have its own method of counting time through ticks, so I may be able to replace the current timestamps I have from <time.h> and replace them with FluidSynth’s synthesizer ticks.

Configuring JACK and Other Instruments

with No Comments

Today, I found out how to automatically connect the FluidSynth audio output to the computer’s audio device input without having to manually make the connection in QjackCtl every time I run the program, and how to map each instrument to a specific channel. The FluidSynth API uses the “fluid_settings_setint” and “fluid_synth_program_select” functions, respectively, for such tasks. Both features are now integrated into the program, and I also allow the users to change the channel to instrument mapping as they see fit as well.

Now, the last thing I need to do to make this program a true virtual conductor is to incorporate tempo in such a way that I don’t have to limit myself and the music to eighth notes or longer anymore. Earlier today, Forrest also suggested that I use an integer tick counter of some kind to subdivide each beat instead of a floating point number. Also, for some reason, I’m still getting wildly inaccurate BPM readings, even without the PortAudio sleep function in the way anymore, but I may still be able to use the clock ticks themselves to my advantage. Although, a simple ratio can convert clock ticks to MIDI ticks easily, but I still need to figure out how I can trigger note on/off messages after a certain number of clock ticks have elapsed.

Another Feature To Add

with No Comments

Yesterday, I changed the velocity value of every offbeat to make them the same as the previous onbeat since the acceleration at each offbeat tends to be much smaller and the offbeat notes would be much quieter than desired. This change, however, also made me realize that my current method of counting beats may be improved by incorporating tempo calculations, which I forgot about until now and is not doing anything in my program yet. The goal would be to approximate when all offbeats (such as 16th notes) would occur using an average tempo instead of limit myself to trigger notes precisely at every 8th note. While this method would not be able to react to sudden changes in tempo, this could be a quick way for my program to be open to all types of music and not just those limited to those with basic rhythms.

Late Minor Update

with No Comments

Yesterday, I only had time to make relatively small changes to my program, but I did write more of the readme in more detail explaining how to use the program. Even more will be explained as I finish developing my program.

Also, I figured out that the reason for the bad points showing up so often was that part of the table was mistaken to be part of the hand, so I moved the Kinect camera a little higher, and sure enough, that issue was fixed. Now the only two important issues I have left to figure out are how to change the instrument that is mapped to each channel, and how the user will be allowed to change said mapping to fit the music. I may also try to figure out how to automatically connect the FluidSynth audio output to the device output of the computer, but I’m not completely sure if this is OS-specific.

Getting FluidSynth Working

with No Comments

In a shorter amount of time that I thought, I was not only able to add FluidSynth to my program, but also able to get working MIDI audio output from my short CSV files, although I had to configure JACK in such a way that all of the necessary parts are connected as well. For some reason, though, I’m seeing a lot more stuttering on the part of the GUI lately and the hand position readings aren’t so smooth anymore, and the beats suffer as a result. I’ll try figuring out the cause soon, but now I have a paper to finish.

Daily Work Update [11-13]

with No Comments

I have been working quite a bit on my program. I set up what I think will be my primary database for info related to diseases. I have also been updating my github to reflect the updated state of my code. I ran into issues where the sql script I was using was based of MySQL and had many syntax that is incompatible with SQLite3 which is my db server. After working on it and through several processes I was able to fix the script and create the DB.

Minor Improvements

with No Comments

I briefly removed the timestamps from my program, but I didn’t notice any change in performance any more, so I just left them in the program as before. I also made my program a little more interesting by playing random notes instead of looping through a sequence of notes, and changed the beat counter to increment every eighth note instead of every quarter note. The latter change will be important when I finally replace PortAudio with FluidSynth.

I also played around with the VMPK and Qsynth programs in Linux to refresh myself on how MIDI playback works and how I can send MIDI messages from my program to one of these programs. Thanks to that, I now have a better idea of what I need to do with FluidSynth to make Kinect to MIDI playback happen. I also plan to have a CSV file that stores the following values for the music we want to play:

beatNumber, channelNumber, noteNumber, noteOn/Off

The instruments that correspond to each channel will have to be specified beforehand too.

More Experimentation

with No Comments

The past few days have been really rough on me, as I attended the Techpoint Xtern Finalist reception all day yesterday, all while being sick with a sore throat and cold from the freezing weather recently. On a positive note, I used my spare time to continue writing my rough draft, so there wasn’t too much time lost.

Back to my program, I’ve added a number of features/improvements to it, the first one being adding timestamps for every recorded position in the hand so that I could actually calculate my velocity and acceleration using time. I did notice that the “frame rate” of the program dropped as a result, so I may try to reduce the number of timestamps later and minimize their usage. I also made sure that all of the points that fall outside the window are discarded to lessen the effect of reading points nowhere near the hand. This also means checking the distance between two consecutive points and discarding the last point if the distance is above some impossible value. I also use distance to make sure there are no false positives when the hand is still, so that the hand has to move a certain distance in order to register a beat. A number of musical functions have also been added for future use, such as converting a MIDI note number to the corresponding frequency of the sound.

Tempo Tracking and More Searching

with No Comments

Yesterday, as suggested by Forrest, I added the ability to calculate the current tempo of the music in BPM based on the amount of time in between the last two detected beats. It doesn’t attempt to ignore any false positive readings, and it doesn’t take into account the time taken up by the sleep function, but it’s a rough solution for now.

Now, the next major step I am hoping to take with this program is to use the beats to play some notes through MIDI messages. I am searching for libraries that will allow me to send MIDI note on/off messages to some basic synthesizer, and FluidSynth looks to be a decent option so far.

Daily work log 11/8/16

with No Comments

Today I open up the extension cord to see what I was going to be working with. I expected two solid pieces of copper instead, I found many very skinny pieces.  Will need to consult with kyle about how to go about working from this point on.

Daily Work Log [11-7] and Design

with No Comments

So after playing around and exploring for a bit, I have finally chosen my final set of tools for the project. I will be using Python, AIML, SQLite with Py3kAIML and sqlite3 libraries. I was able to finish the plumbing and now have a very basic bot that can listen to the user, fetch data from the SQLite database and print the result. Now that I have the main tools I will be using, the design of the system will be the following:


Fun with PortAudio and Next Steps

with No Comments

Today, I added the ability to change the volume of the sound based on the acceleration value, or how quickly the hand is moved, as well as change the frequency of the sound and thus change the note being played using a simple beat counter. I also noticed that the beat detection works almost flawlessly while I make the conductor’s motions repeatedly, which is a good sign that my threshold value is close to the ideal value, if there is one.

Now that the beat detection is working for the most part, the next thing I need to do is to figure out how to take these beats and either: a) convert them to MIDI messages, or b) route them through JACK to another application. Whichever library I find and use, it has to be one that doesn’t involve a sleep function that causes the entire program to freeze for the duration of the sleep.

November 2 – November 7 Weekly Update

with No Comments

November 2 – November 4

Since I figured that understanding Android development would take more time than I expected, I decided to speed up the development process by using Cordova as my development platform. I installed Cordova on my computer and started integrating the Wikitude API into it.

November 5 – November 6

I took an actual campus tour on the family weekend to see how the guides walk the guests through the campus of Earlham and see from a visitor’s perspective. It is really helpful and I collected some information for the application.

November 7

I started working with GPS locations and Image recognitions.

Audio Implementation

with No Comments

I added the PortAudio functions necessary to enable simple playback as well as revised my beat detection algorithm to watch for both velocity and acceleration. My first impressions of the application so far is that the latency from gesture to sound is pretty good, but I noticed that the program freezes while the sound is playing (due to the Pa_Sleep function which controls the duration of the sound), which freezes the GUI, but could potentially mess up the velocity and acceleration readings as well. False positives or true negatives in the beats can also occur depending on the amount of threshold set, and the detection algorithm still needs more improvement to prevent them as much as possible.

Current Design and Next Steps

with No Comments

Here’s the current version of the flow chart of my program design, although it will surely be revised as the program is revised.


I’ve also been thinking about how exactly the tracking of velocity and acceleration is going to work. At a bare minimum, I believe what we specifically want to detect is when the hand moves from negative velocity to positive velocity along the y-axis. A simple switch can watch when the velocity was previously negative and triggers a beat when the velocity turns positive (or above some threshold value to prevent false positives) during each iteration of the main program loop. The amount of acceleration at that point, or how fast the motion is being done, can then determine the volume of the music at that beat.

Work log 10/29/16-11/4/16

with No Comments

I have spent the last week split up between 3 different tasks: Starting to chart the twitter ER diagram, following the O’Reilly Social Media Mining book to continue to learn about harvesting through APIs, and reopening my database systems textbook to remind myself how views work.

After speaking with Charlie last week, we discussed the possibility of using views to select relevant tables from the larger Facebook and Twitter structures to create a model that was easily modifiable and a combination of the two existing models, rather than trying to force the models themselves together into a new, heavily set model.

Work on the Twitter model is coming along, and I hope to be done by the end of this week.

Daily work log 11/2/16

with No Comments

Went to home depot and found a very help person who was knowledgeable in electronics.  I have decided to use an extension cord as the backbone of my non-invasive device.  I will plug both the power source of the Arduino and the washing machine into the extension cord, then open up the extension cord to read the voltage of the washing machine. I am working and talking to kyle about different technique with resistors to bring the voltage down to a readable amount and then using the Arduino’s built in 0-5V reader to read the voltage.   Began working with online bread board and Arduino simulators to begin testing.

Quick Update

with No Comments

Small update today since I have other assignments I need to finish.

I implemented a simple modified queue that stores the last few recorded positions of the hand in order to quickly calculate acceleration. I also learned a bit more about the OpenCV drawing functions and was able to replace drawing my hand itself on the screen with drawing a line trail showing the position and movement of the hand. Those points are all we care about and that makes debugging the program a little bit easier.

Update 11/2/16

with No Comments

I have placed some of my early data in various spreadsheets.  I am continuing the process of collecting data, and am ready to use my the information I have so far and observations I have made to start the first draft of my paper.

Update 10/30/16

with No Comments

I have been gathering more data to find the optimal number of keyframes for various types of videos.  The videos with larger file sizes take a long time to compress.

Daily work log 11/1/16

with No Comments

Couldn’t get OpenCV to install properly on my laptop so I asked the CS admins to create a cluster account for me. By sshing to the cluster I will be able to use OpenCV. Tomorrow I expect to get the facedetect sample to compile and run. From there I can start working on implementing the emotion code.

Compiling the Program

with No Comments

After a good amount of online searching and experimentation, I finally got my Makefile to compile a working program. There is no audio output for my main program yet, but I am going to try out a different beat detection implementation that bypasses the clunky gesture recognition (namely tracking the position of the hand and calculating acceleration), and hopefully, it will result in simpler and better performance.

Daily Work Log 10/31/16

with No Comments

Just wrapping the wires around the cord, doesn’t work, neither does attaching the wires to the prongs of the plug.  I am thinking about going to an electrician or home depot to find someone who knows where to measure the voltage

Testing PortAudio

with No Comments

I don’t know what took me so long to do it, but I finally installed PortAudio so that I can actually use it in my prototype program. To make sure it works, I ran one of the example programs, “paex_sine”, which plays a sine wave for five seconds, and got the following output:

esly14@mc-1:~/Documents/git/edward1617/portaudio/bin$ ./paex_sine
PortAudio Test: output sine wave. SR = 44100, BufSize = 64
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
bt_audio_service_open: connect() failed: Connection refused (111)
Play for 5 seconds.
ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred
Stream Completed: No Message
Test finished.

I’m not entirely sure what is causing the errors to appear, but the sine wave still played just fine, so I’ll leave it alone for now unless something else happens along the way.

Now that I have all the libraries I need for my prototype program, all I need to do next is to make some changes to the demo program to suit my initial needs. I’ll also need to figure out how to compile the program once the code is done and then write my own Makefile.

1 5 6 7 8 9