I have been working on editing my final draft for my proposal and updating it with new information that I discussed with Xunfei about the direction of the project. I have been adding information on the different directions I can take this project.

Poster used to present Capstone

Submitted Proposal Draft 2. Worked on presentation. I have decided that instead of just generating the geometry of the levels it will be a lot more interesting and compelling to also figure out how to generate the puzzles within the level. This will be tricky but I can think of a few ways to to this. The first and easiest one would have some chunks fully contain a simple puzzle, while this would be effective and ensure complete-ablility it would get repetitive fairly quick and not be very challenging. The second method would be to have meta data contained within each chunk, and have a more developed chunk selection process. such as this chunk contains a 4 block vertical jump, so before there is another chunk like this there must be a chunk that contains a crate. the last method that I thought of would be another algorithm that would loop over the geometry of the level and place puzzles like that. this would be quite tricky but it would simplify the generation of the geometry.

I have finally submitted the first draft of my proposal for PCG using ORE in my video game. I am excited to keep working on it, Initial results are promising.

I have been continuing work on my first draft of the proposal, progress is slow but I will eventually get there. I have been doing a little work to try and improve my simplified ORE algorithm but I think I will have to postpone this work so that I can focus on my proposal.

I have designed a simplified ORE algorithm. I created a chunk library using ascii characters, only 15 different chunks. It is a simplified ORE algorithm because each chunk has 2 anchors, so there is only ever one place for the extension to take place. It also randomly selects the chunks so there is no selection process. While this algorithm is simple, I have been able to generate levels with great diversity. One of the parameters I feed it is the number of chunks I want to stitch together. I find that 40 looks the best and that the more chunks you have the more variety. since there is only ever one extension point, it does not allow for branching which is one facet that make a game more enjoyable.

This week I spent working on getting my poster ready for printing. Other than that I also spent several hours preparing my paper for the evaluation draft due Friday. Not there yet but making progress.

my literary review has definitely guided which direction I want to take my project in. While doing research into the different kinds and application of PCG There are a lot of cool examples of the use of tile based such as in spelunky or rhythm based in others. I am most intrigued by Occupancy Regulated Extension(ORE), an algorithm that looks at where a play can exist in a level and branches out based on a predetermined library of level segments. This methods seems like the most applicable to my video game, or at least the geometry and it gives the users some control, what level segments go into the library. I have yet to find source code for it however.

I spent most of this week writing the most recent draft, and then putting together a first draft of the poster. Spent some time debugging, but my focus was on the paper and poster.

I have finally finished my literary review, and it has been super cool finding more examples of games that use PCG. I think It would be really neat if I could figure out a way to implement an algorithm in a way to make my game infinite! Its also amazing how far PCG has come from Rouge which was basically the first to now, were some games are much more complex and intricate.

This past week, I spent countless hours working on my Final Paper.  I also spent a considerable amount of time working to correct errors and bugs in my code inside of MaxMSP.

This week I continued debugging my code. I now have everything I need working working. It’s not as good as I think it can be, but it is fully operational. Spent a little time starting paper revisions and that’s next up.

This week ive been working on revising my paper according to the feedback i got and suggestions from Xunfei.

Ive been trying to polish out any remaining functionality issues and format it better for presentations.

Still polishing the paper. Been revising section by section and created a new diagram for the paper.

Need to fix an issue with the citations.

Will make time to try and tweak the experiment for the paper to see if I can get different result.

1. The spring break week dealt with mapping MIDI control into the acoustics of the three-dimensional space, which I have successfully done.
2. I have also successfully added an FIR based buffer into the program as well to track analytics. You use buffir~ object when you need a finite impulse response (FIR) filter that convolves an input signal with samples from an input buffer. With that, you can look at an FIR filter response graph in MaxMSP which calculates the filter response of the FIR in the main sound patch for a given number of samples (coefficients)

Created and ran experiment. Noted the results down.

Finished the first draft of the Capstone paper. Currently revising and polishing it.

This week I turned in a new draft of my paper and fixed a few bugs. I am now ready to collect most of the data I’ll need.

This week I have dealt with working on improving my final paper and working on the MIDI mapping and audio of my project inside MaxMSP. I primarily worked on getting the audio format correctly to be able to loop and connect each file with the object associated in the three-dimensional space within MaxMSP.  I secondly worked on the formatting of my paper; it needed to flow from topic to topic more also recognizing why it was relevant to the project as a whole.

This week I have been working on my paper both revising the submitted partial draft and adding the portions that were not done yet.

I have also done some work on my project itself. I have been doing a lot of refactoring. I made it so all the variations I am testing can be turned on and off via toggle buttons and all things that need to be initialized to run the patch are accessible in the main patch window instead of having to open the subpatches. I also revised portions of my project that had large repeated sections of code. 

This week I

-implemented feedback from the first draft

-started writing the rest of the paper

-did some minor debugging and commenting

Built a functioning, testable (not yet accurate) Neural Network that takes in my input of heuristic data and output a direction.

Since the puzzle states were strings, I wanted the output to be the value added to the index of the “_” to move to a new position but negative values cannot be used as targets or labels in neural network so instead I coded them using values 0-3 to represent each move.

I created a function that takes in the output of the neural network and converts it to a move for the puzzle.

Also know which specific layers I need to experiment with make my neural network be accurate.

I have continued working on my annotated bibliography, but I have not finished it yet. I am finding it hard to focus on and be motivated for a project that I can not physically work on. I am still missing a couple sources for the annotated bibliography and I have been a bit lazy about looking for them. I think I will start by looking at other papers that I have found citation’s.

This week mainly dealt with the writing of my final paper. My paper was mainly a mostly complete draft that worked for my initial goal and vision of SoundStroll 2.0. However, in its finality, it will be called SonoSpatial Walk and and it will be its completely own project. The added changes allow for generation of objects through MIDI, and hopefully other things will be allowed with the source code that was completed changed by myself as well. Hopefully, with the object generation, I will be allowed to work on the sound properties that play and loop through triggers. Also with the sound properties, I’m hoping I can resynthesize sound in real time using Fourier Transformations in order to complete change the sound as well.

Over this week I worked on finishing a first draft of my paper. Also I worked on debugging code and trying to install some packages.

This week I:

-worked on my paper draft

-worked on fixing bugs in my software

I have been a little bit behind on synthesizing my articles into the annotated bibliography, I mostly have just been trying to collect more and more articles. The gesture control Idea and the educational app idea seem a little difficult to complete in a semester of work so I think I will probably continue forward with the PCG for my video game, It is something I am definitely passionate about, I cant say the say for the others.

-I created a large variation to my algorithm that spacializes the visualization based on the ratios just intonation ratio instead of being directly correlated to the frequencies. This involved implenting a new module that calculates the just intonation ratio and scales the sine wave visuals to that.

-I created tables to use in my paper draft and started turning my outline into my draft.

This week, while I was not able to work on the coding aspect to my project, I did receive a synthesizer from Professor Forrest Tobey, and have been working extensively in MaxMSP to get it to react to my program.

Also, I have been working on my paper to get it ready for the 02/25/19 deadline. 

I’m still trying to get the data over. I’ve been working on it for like 2 to 5 hours everyday but errors keep arriving. I’ve also researched about algorithms and have a better idea of my project scope.

I tested that the last of my dependencies is on the cluster and functional. Started implementing the second major piece of my approach, SURF. Taking a little longer than expected but not a major delay.

Changed a piece of my pipeline and created some issues. Working on fixing those, and hopefully improving the overall functionality in the process.

Managed to have Keras, an open source network library, installed in Jupyter.

Currently focusing on building a sample neural network, adjusting the data into a format that can be used by the network, iterating on the architecture required for the Neural Network and writing a draft of the first few sections of the Capstone paper.

This week, I was able to realize partial spatialzation in MaxMSP using the new tool that I had found previously, HOA Library. Right now I am trying to get my object driver to work to be able to create multiple figures. Also, Forrest wanted me to map the knobs of a MIDI keyboard to Max and the patch.

I have started collecting papers on my three ideas.

There are many different sources that have used gesture controlled navigation in the past so I think it will be hard to differentiate from them, but most of them seem to have to use another device. There are a couple that are just controlled by a camera, this would be a neat way to solve this problem, and wouldn’t require people to buy anything.

For my second Idea of PCG for my video game, I have gathered sources where people use pcg to remake games or even make new ones. None of them seem super similar to what I would like to implement. I have also found a few good overviews and lit reviews of the subject. These will come in handy.

The papers I found for my last idea, of an educational app for ultimate frisbee, mostly just include the theorys and methods for educational applications. these include apps such as duo lingo and other language learning sources. I figure a lot of this could be mapped over to any educational application. It seems like a lot of the problems in this field deal with retention.

I have been trying to get the sensor to give me significant results and have had a lot of trouble with that. I mostly did research and experimentation to try and get it to work.

This week I finished my implementation of the SIFT algorithm, starting to work on implementing SURF next.

Also looking ways to organize images to improve accuracy. Currently planning to get keypoints, then organize photos based on similar keypoints rather than locality.

This week I did the following tasks:

-Met with Dave about the structure of my outline and my experimental results section

-Met with Forrest to get feedback on my work.

-Began to work on a new variation of my algorithm based on Forrest’s feedback

-Read chapter 5 in writing for computer science

I decided to pivot on my ideas a little bit because 2 of them were already solved and I didn’t feel that I had anything to add to the field. I have found papers and starting to read them on my new ideas, ideas listed below.

Idea 1 : Gesture Controlled Mouse and Keyboard

Description: The Idea is to fully replace the keyboard and mouse of a computer by using a 2D camera to track hand motions. I really like the idea of incorporating the feature of swipe to text as seen on some mobile phones, this could possibly increase typing speed. 

Idea 2 : Procedural Level generation for 2D platform game

Description: Last semester I built a Game in Unity with 2 friends, we have continued to work on our game with the intention that one day we sell it. A large part of this process will be the generation of new levels. One idea is to develop an algorithm that procedurally generates the levels. While procedurally generated games are fairly common, procedurally generated platformers are far less common. There are a few challenges involved that I think would make this interesting. 

Idea 3: Educational app for learning the rules of Ultimate Frisbee

Description: My other new idea, is to create an app that is maybe similar to other educational apps such as Duo-lingo, but with the intention of teaching the rules of ultimate frisbee. Since it is Self-refereed sport it is much more important that the players know the rules themselves. However these rules in the rule book are very wordy and hard to follow sometimes, this would provide an easily accessible way to learn the rules without having to sit down and read a book.

This week I made a first draft of an outline for the paper. Also worked on implementing the first of three algorithms I plan to use in my research, Scale Invariant Feature Transformation. While not completely finished I have most of the framework complete, and am just about where on schedule.

This week offered me a chance to get into detail with spatialization libraries:

• Jamoma – C++ and Max/MSP general purpose audio and spatialization toolset with standard interfaces, requires custom installation depending on the version of Max though. This library is needed to use most of Tom’s Max patches.
• NASA Slab – (Older) open source project for testing spatial auditory displays, requires registration to NASA open source archive.
• CICM Higher order ambisonics library – SuperCollider(under development), CSound, Pd, Faust, oFx, Max/MSP and C++ archive of useful ambisonics stuff, renders down to binaural well but computationally quite intensive. This library is required to use most of Tom’s Max patches.
• Sound Field Synthesis Toolbox for MATLAB – Python version exists as well btw. Sound Field Synthesis Toolbox (SFS) for Matlab/Octave gives you the possibility to play around with sound field synthesis methods like Wave Field Synthesis (WFS), or near-field compensated Higher Order Ambisonics (NFC-HOA). There are functions to simulate monochromatic sound fields for different secondary source (loudspeaker) setups, time snapshots of full band impulses emitted by the secondary source distributions, or even generate Binaural Room Scanning (BRS) stimuli sets to simulate WFS with the SoundScape Renderer (SSR).
• MIAP – Max/MSP objects, not spatial audio per se but pretty cool. More spatial parameter space exploration, though the binaural example is in the pack.
• Octogris – OSX DAW 8 channel spatialization plugin
• Spatium – Plugin (AU), Max/MSP, Standalones (OSX). Modular open source software tools for sound spatialization: renderers, interfaces, plugins, objects. Got some nice processing based physic interactions for spatial control.
• ambiX – Ambisonics spatialization Plugin compatible with Reaper, Ardour, MaxMSP, Bidule or as standalone applications with Jack.
• HOA – Higher Order Ambisonics (HOA) resources for Pure Data and Max from Paris Nord university.
• ATK – Ambisonic Toolkit for Reaper and SuperCollider.
• Sonic Architecture – resources for ambisonics in Csound and the blue environment from Jan Jacob Hofmann.
• Iannix- a graphical open source sequencer for digital art. It requires sound making software or hardware connected to the sequencer. The sequencer sends instructions (e.g.OSC) and allows to create 2D and 3D scores programming the behavior of sliders and triggers.
• Zirkonium – tool from zkm to spatialize music.
• NPM – web audio classes for Ambisonic processing FOA and HOA.
• [omnitone] (https://github.com/GoogleChrome/omnitone) – spatial audio on the web – by Google.

Adapted from: https://github.com/darkjazz/qm-spatial-audio/wiki/Open-source-free-spatialisation-tools-around-the-web

What I decided to go for was was HOA Library, because HoaLibrary is a collection of C++ and FAUST classes and objects for Max, PureData and VST destined to high order ambisonics sound reproduction. It won “Le Prix du Jeune chercheur,” awarded by the AFIM in 2013. This library is free, open-source and made available by CICM, the research center of music and computer science of the Paris 8 University. WIth using that, I know that I can make a lot of edits to it, and many people have used it in a concert or installation setting. Now that it’s decided, I can work towards the connection between MIDI and the spatialization of objects.

This week I accomplished several things:

-Reconfigured the way my patch combines matrices to avoid issues I was having w crossfade

-implenented horizontal movement after note press

-Organized parts of my project into sub-patches and cleaned up some stuff

-Implemented envelopes connected to sound and video out

-Created an outline for my paper 

This week I talked to Dave about the delay on my database. I am waiting to hear back from Charlie andor Craig so I can start working on building the database and connecting the dots. Until then I have read up on some more algorithms, specifically Reddits and ELO’s. I am also writing up my outline for CS488 and cleaning up some paragraphs from last years paper.

Built a prototype testing agent for the Capstone that should in theory take in a file of problem states and go through them with each Heuristic(Neural Network not yet ready) then output the solution size, number of nodes visited and number of nodes that are waiting for a visit. Haven’t gotten to properly test this one so will need to make time for that.

Tried writing a high level outline of my Capstone.

Further deliberated on my 3 ideas, I really enjoy the idea about replacing the keyboard and mouse set up with a gesture recognition system from a 2D camera, most gesture recog systems require another device that one must buy, but cameras are already built into most things. My other favorite is the personal budgeting app, I am interested in this one because it applies to my life, It would be software that I would use to help me save money.

This week, I finally got my own laptop so I redownloaded all the necessary software and libraries for my gesture recognition system. Using the OpenCV library in Python, I was finally able to use the camera to detect the hand by separating the background with the foreground via thresholding. Then, I used findContours to identify the hand within a region of interest (roi). Finally, I made a copy of the the frame containing the roi and displayed the video in binary black and white. My code can now detect the hand with a static background.

In terms of implementation, I have created 4 different heuristics functions for my Capstone that take in a state and output a value based on how good the state is. with 4 being inadmissible. Technically 5 as 2 heuristics are additions to a single heuristic.

Modified my admissible heuristic function so it outputs a move (left=-1,right=+1,up=-4,down=+4) based on what move a search algorithm would have performed if it is was taking an action in that state.

Created my training agent function that outputs a file contains vectors of these heuristic outputs. One per state

Did some research into activation functions and Neural Network types to figure out what initial design I should go with for my Network.

This week I worked on developing a script that takes a directory and determines if the contents are all images of the same type, and if they aren’t converts them to jpgs. I also meet with my adviser and starting working on an outline that will be due next week.

This week dealt with the focus of the FFT and other conversion formulas. In recognizing that the Fast Fourier Transform is an algorithm that computes the discrete Fourier transform (DFT) of a sequence, or its inverse (IDFT), you understand how that mainly relates to Fourier analysis and converting a signal from its original domain (often time and/or space) to a representation in the frequency domain and vice versa. With my current project, I am trying to understand how this relates between the signal processing and MIDI, and how that will allow me to transfer the signals into the shapes that I will be making in Jitter and MaxMSP. In understanding the multiple ways in which the transforms work I can use the signals that I get by decomposing a sequence of values into components of different frequencies, which will then generate multiple sounds, and that hopefully can be somewhat composed into a song or piece that is made through the patch. Currently SoundStroll 2.0, but it might end up being a whole different concept and Max patch when it’s done. In understanding what changes are being made, the new product could be thought of as a completely different three-dimensional audiovisual spatialization software.  My main work has been with working to create a different audiovisual projection whilst working with MIDI and other signals to get different spatialization in MaxMSP, this was concurrently added to my paper, and it was also worked on in MaxMSP and Jitter.

Here is what I worked on for my project this week:

-Worked with poly tool to process two simultaneous notes.

-Got it working but temporarily disabled the circle of fifths map table-Now midi messages are sent to two paths for separate notes.

-Using xfade to combine matrices is not good, colors are still evenly distributed throughout the whole matrix instead of following freq mapping.

-Patch is getting crowded need to figure out in and out tool to split up and organize project.

-Talked with Xunfei about getting Max installed on some Earlham laptops

Researched how image processing works (tutorials, examples using python and open-source software OpenCV). ~20 minutes on 1/19/19, ~30 minutes on 1/21/19

Made sure all necessary software was downloaded to my computer. ~30 minutes on 1/17/19

Adjusted timeline. ~15 minutes on 1/23/19

Found an advisor, and set up a meeting time to plan my project. Tested that the libraries I plan on using are available and functional on the cluster. Finally reviewed my project to make review the goals I’ve set myself are achievable and reasonable.

This first week, I tried to focus my project into something tangible. Initially, I was not sure about how I wanted to go about editing and making SoundStroll 2.0, and I had talks with advisors who told me that I tried to achieve too much in a short period and that I also needed to find a focus. Interestingly enough, I think that working with and creating a bandpass filter, in addition to other improvements, can produce different effects in SoundStroll 2.0, and I’m curious to see how that might change SoundStroll 2.0, regarding effects and three-dimensional objects that are spatialized with the signals.

In understanding what needs to be done for the project, I still have many main steps:

1. Completely rehaul and add a new more advanced spatialization toolset/visualizer to work with the updated software.

2. Add vocal recognition through a speech processing patch that’s made by myself to the updated Max project and get them to work together.

3. Add new additions that track the Fourier Transforms for analysis that allow the original and added vocal recognition through a speech processing patch.

4. Make sure that vocal, MIDI, and OpenGL components are working so that objects are spatialized through vocal component or the MIDI interface.

5. Potentially add virtual reality/VR component to the project to let you traverse the world.

In finality, I want SoundStroll 2.0 to be a logical successor to its former self. SoundStroll 2.0, or the new edition that I plan to create will use Fourier Transforms for vocal recognition through a speech processing application, which can also be analyzed for more profound findings audially or mathematically in relation to sound; a different, more advanced spatialization toolset to create a scene that you can traverse; and a more connected environment to create different ways to traverse with Max for Live, Ableton Live, and Reason, which will allow new sounds and objects to be triggered and spatialized through SoundStroll 2.0.

This week I looked over the specific dataset that I want to use for my project. I reached out to sources to obtain that dataset. Similarly, I read various researches and, checked existing experiments and projects to learn implementation of Convolutional Neural Network (CNN) in python using Tensorflow and Keras. Read documentations and went over tutorials to learn Tensorflow and OpenCv. Also, I setup Tensorflow in my local machine to work with OpenCv library.

Just sharing a helpful tutorial link below to understand about OpenCV

Link: https://www.lynda.com/Python-tutorials/OpenCV-Python-Developers/601786-2.html

The tangible work I have done this week on my project is finding the data sets and started setting up a work flow to automate the process. I created the connection between SQL and my simulation so that the simulation takes 2 arguments, a name of the database to store the timestamp data as well as the result data and the file path of a csv containing data. This sets up a database for the data with three tables, two of which contain results and one with the actual timestamp data. I have also started to take notes as I go so that I can refer to them when I start to write my paper.

I have gotten off to a relatively smooth start on my project. Xunfei is going to be my adviser and I have scheduled a meeting with her for tomorrow morning. I have prepared to present on my topic in class tomorrow. The majority of my time this week I spent starting my project implementation. I am still learning how to use Max/Jitter effectively but here are some of the things I have done:

-Map color values to MIDI notes.

-Map a saw waveform function over the x axis with scale relative to note frecuency

-Make the saw function scroll horizontally at a rate relative to frequency

-Crossfade the color and waveform matrices

With all of this together, my projects output visual output represents a single note at a time. Obviously it is a pretty rough version of what I want the eventually single note visualization to look like but the foundation of representing tone through color and pitch through wave frequency is functional and has no noticeable latency between note press and display.

First post of Capstone progress. First thing I did was review my Capstone proposal paper to re familiarize myself. Immediately I begun work on the work I outlined for Week 1 in my timeline.

I built a generator for random initial board states for an 11 sliding tile-puzzle. I chose to have each board state be a solved state that was then subjected to a random number of moves. This is ensure that each state is solvable without having to check first. Another choice I made was to have each state be represented as a string with each tile being represented by a hexadecimal number and position represented by its index in string.The blank space is represented by “_”. This was done because a number of states will be generated during the AI’s decision making process so the states should be as small and compact as possible.

I also started to look into the tools for creating the neural network. So far I am looking at Tensorflow, Keras in combination with Python.

Final Paper – crawl-o-matic-o-matic

Poster – ebramth15_poster

Gitlab with code – https://gitlab.cluster.earlham.edu/duckroller/crawl-o-matic-o-matic

Paper: https://portfolios.cs.earlham.edu/wp-content/uploads/2018/12/Detecting-textual-analogies-Rei.pdf

Poster: https://portfolios.cs.earlham.edu/wp-content/uploads/2018/12/CS488-Poster.pdf

Code: https://github.com/reirembeci/senior-capstone

Finished working on the final paper.

This week, I attempted to collect preliminary data, which made me realize that I would need an air compressor in order to obtain meaningful data. I added this to my proposal and made some modifications to improve it overall.

This week after receiving feedback from Xunfei on the second draft of my paper, I updated it with respect to her comments. I also followed up with Craig; he is going to purchase the equipment early next semester.

Got feedback for my second draft of my proposal. Ready to make these final changes and submit on time! Started installing the dependencies so that I can get familiar with the tools I’ll be using over the break. Not planning to commit too much effort into this in the midst of finals, but will redouble my efforts after the tests are over.

This week I received feedback on the second draft of my proposal from Xunfei. I spent time updating the proposal with respect to her feedback. I also met with Craig and decided on a set of RFID reader and tags. Craig is going to purchase them before the end of the semester.

This and the previous week, I spend my time trying to beat my way through a variety of problems related to one specific module of my project’s code. After talking over it with my adviser I have shifted both that piece of code and what should be taking my priority. The original model of my project was that, after the blockchain authenticated a ‘website’ so that it could access a user’s ‘cookie’ file, the functions would be passed over to the user from the website’s server to the user’s. My adviser suggested I simply change this so that the user has both the file and the functions, and that the website simply sends a signal if the blockchain transaction was approved. The actual process of the functions on the cookie file are not as important as the blockchain authentications, which is the center of the project.

The refocusing suggestion was also on everything related to my experiment, such as the details of my experimentation, and the discussion of further research. Leave it so that I have as much done as possible, with that window left for my experimental results.

This week I haven’t been able to work much because of some traveling. However, I was able to think more about my design decision including the RFID reader that I will buying. I have selected a device, and discussed with Xunfei. I am planning to meet with Craig and possibly purchase the equipment before the end of the semester.

This week I received feedback from Xunfei on my initial draft of proposal. I spent the last few days thinking more about design and trying to answer some questions that Xunfei raised. I tried referring back to my sources and it has been very helpful.

I met with my mentors 3 times to discuss the proposal’s feedback and what I can do to improve it. I also reached out to some folks whose previous work could help me. I decided on my end product as a result of that. I also decided to change a few things in my project based on the feedback I got. I will start with a simpler artist style to replicate and if that goes well, I will move to the more complex one. I will also most likely change my sensor from an “air quality” to a CO sensor because that will allow me to see more variation on campus, but I will have to obtain preliminary data before I can make a final call.

This week I did a lot of research into how jitter works and how it communicates with max and msp objects. In the process I found a few more helpful sources to replace some of the less relevant related works that I have. The past couple of days I did some programming on a max patch to familiarize myself with the basics of converting an audio signal into jitter output. I still have a lot to figure out but I was able to create a visual representation of a basic sine wave. I learned about audio sample rates and how to buffer an audio stream into a matrix.