Final Report

So finally it’s over. Today is the last date for submission of the GSoC project. This entire ride was a lot informative as well as an experience filled one. I thank Peragro organisation for accepting my GSoC project and my mentors Matthew Lagoe, Branko Majic and sueastside for helping me out fully throughout this project.

Below are the modules of the peragro to which i have contributed in. I will give a brief description of each module and what all work has been done during this summers.

Peragro-at : Peragro’s analyzer and transcoder framework, which is the standalone heart of Peragro. It has two main components.

• Analyzer - analyzes file contents for potential assets and extract features or metadata. command line to analyze :

  pt -v a file.blend

  Analyzer has a plug-in system that searches for an appropriate plugin for analyzing the given file type (blender file in our case) and prints the assets such as mesh, images , textures and their metadata.

• Transcoder - It converts a file type to usable asset. Transcoder also have a plugin system same as that of analyzer that allows to convert one file type to another and can be used to change the resolution, rotation, size etc.
  
  
  pt t file.blend  "Cube1-data(application/x-blender.mesh)" "image/png" --size=64,64

  The above command will transcode the 'Cube1-data' asset to 'png' format with size 64x64

My Contribution to peragro-at:

1. Audio feature extraction: My project started with creating a plugin that can extract audio features like tempo, global key, mood, gender etc. Essentia is an open-source library for audio analysis and audio-based music information retrieval which was used to extract features metadata from an audio file. Essentia’s pre-build command line feature extractor (streaming_extractor_music) was used to extract the low level features such as tempo, global key, Average loudness etc. For the high level features (such as mood, gender, genre etc) extraction, Essentia need to be build with gaia (which implement similarity measures and classification) to be able to run high-level models. For each high level feature like for say gender, there is a pre-trained model (gender.history file). Each model is basically a transformation history that maps a pool (a feature vector) of such lower-level descriptors produced by low-level extractor into probability values of classes on which the model was trained. For high level, "essentia_streaming_extractor_music_svm" was used as command line extractor which need to be configured with the models. Code:https://github.com/peragro/peragro-at/blob/master/src/damn_at/analyzers/audio/feature_extraction.py
   Conf_file:https://github.com/peragro/peragro-at/blob/master/src/damn_at/analyzers/audio/profile.conf
2. Docker : The next thing that need to be done was to make the project easy to install and use. So we decided to dockerize the project. The Docker image can be easily pulled from the docker hub as shown below:
   docker pull peragro/peragro-at

   and voila we are good to go. Now to analyze a file just use the command line:
   

   docker run --rm -v `pwd`:/peragro -ti peragro/peragro-at pt a file.blend

   Code: https://github.com/peragro/peragro-at/blob/master/Dockerfile
3. Added .fbx file support for 3D models Code: https://github.com/peragro/peragro-at/blob/master/src/damn_at/analyzers/mesh/analyzer_assimp.py
4. Resolved many pylint and other errors and added travis support. 

All of my Commits :https://github.com/peragro/peragro-at/commits/master?author=sagar-kohli

Peragro-index : ElasticSearch based indexing for the extracted meta-data. Indexing is done through pipeline command and directly through stored analyzed data.

My Contribution to peragro-index:

1. Command line indexing: Previous version of command line was too lengthy so I made it short. Command line for indexing analyzed data:
   pt a file.blend | pt index elastic
   

   Code: https://github.com/peragro/peragro-index/commit/c65cb2411305ccff49f655859337c9f165a3d1bf

   
   
2. Docker: Similar to peragro-at, I have also dockerized the peragro-index which will help the user not to install the dependencies along with peragro-at.
   
   Code : https://github.com/peragro/peragro-index/blob/master/Dockerfile

All of my commits : https://github.com/peragro/peragro-index/commits/master?author=sagar-kohli

Final Words:

I would like to thank my mentors Matthew Lagoe (Botanic), Branko Majic and sueastside for guiding me. They were really encouraging and helpful. I’ve had a wonderful journey so far. Ofcourse, this is not the end. My work for GSoC introduces so much more possibility and I would like to explore and continue contributing :)

Hi !! everyone

The second evaluation has just started, so let me just summarize, what have I done after the First evaluation.

In my last Blog I told you about the plugin that extracts the low level and highlevel features from the given audio file. Now this plugin together with the other existing ones form a part of the project(peragro whole project) which is called as Analyzers. So what an analyzer module does is, is it finds the right plugin for analyzing a given file of a particular mimetype(basically an audio file in our case). Then that plugin will output the various assests found in the given file and their relavent metadata. Then there is another module named as Transcoder that help you to convert a files assest of a particular mimetype to other relevant mimetypes. for more information on analyzers and transcoders see this (http://peragro.github.io/peragro-at/)

So this week I read the transcoder's code and learned how it works and fixed some of its bugs. In addition to that there were lots of landscape errors in the whole project, I had fixed the majors ones and will try to fix others also.I have also learned about how a cyclic import can cause a big problem to a project.

sagar-kohli

hi !! everyone

Its second week after the First evaluation and yes !! :) I have passed the First evaluation and can official continue my GSoC journey. A lot thanks to my mentors without whom it won't have been possible.

In my last blog I told you about a plug-in that extracts low-level features from an audio file using Essentia binaries. This week I have added high-level features extractor to the plug-in. For high-level extractor we need to build essentia with gaia to be able to run high-level models. Each model (a *.history file) is basically a transformation history that maps a pool (a feature vector) of such lower-level descriptors produced by low-level extractor into probability values of classes on which the model was trained.

"essentia_streaming_extractor_music_svm" extractor was used to extract high -level feature and "streaming_extractor_music" extractor binary was used to extract low-level features.

The following are the high-level features that can be extracted now:
  1. Mood
  2. Genre
  3. Danceability
  4. Voice or Instrumental
  5. Gender

In addition to this, I together with my mentor Botanic worked to improve travis-ci that now uses docker container to run tests.

sagar-kohli

Hi!  Thanks to all who went over my previous post.

So the First evaluation has started and this is my second blog. In the previous blog I had explained about my organization peragro and gave you a brief summary of what i will be doing this summer.

In this Blog i will tell you about what i have done so far.

I have made a plug-in that extracts features from an audio file using Essentia binaries. Essentia is a open-source library for audio analysis and audio-based music information retrieval. It contains an extensive collection of algorithms including audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors. The metadata (features) that can be extracted so far includes the following:
   1. Tempo
   2. italic name of tempo (tempo markings)
   3. beat count
   4. Chord
   5. Global key
   6. Average loudness
   7. Lossless

While installing Essentia we realized that our project has led to lots of dependencies and thats when my Mentors suggested to use docker containers and dockerize the project. And thats where I got to know about, what a docker is and its uses.

I would like to strongly thank my mentors who had always helped me whenever i got stuck and properly guiding me along. Got to learn many things from my Mentors and hope i will learn many more during this GSoC journey.

sagar-kohli

The start of my GSoC journey!

Hi !!

This is my first blog post in the many to come as a part of my GSoC journey with Peragro. Today I'll try to explain the project that I will be doing this summer.

Project Title    : Implementation of an audio search system.
Organization   : Python Software Foundation (PSF)
Sub Org.         : Peragro 
Mentors          : Matthew Lagoe, Sueastside, Branko Majic


 What is peragro?
The core principle of Peragro is to make a centralized 'asset server':

• To stimulate collaboration between artists, by the means of tasks e.g. a texture artist can pick up where a modeller left of.
• To take the job of exporting out of the artist's hand by entirely automating it.
• Allow to tag assets and make collections to easily manage assets.
• Share an art repository between projects to reuse common assets, tag or group assets to be more settings or project specific.
• Deliver and accept its content in the format requested/provided by the user as to support multiple tools so assets can be freely distributed between artist and each can use their program of choice.
• Deliver its content in the format requested by the engine, projects might use CS, Ogre or 2D, etc representations of assets.

 Aim of my project?
My project aims at providing an audio Information Retrieval system that would provide client API(s) for audio search and information retrieval. Search will be based on annotated text/descriptions or tags/labels associated with an audio as well as on low and high level features. Focus is to implement an AIR system that provides text based and content based search. Later we can add searches based on small recording and can get audio metadata and list of mixable audio. We can also create a module that will return results based on mood .

So in the First month of my project i'll be working on making a plugin that will extract the features from the audio files using essentia and gaia libraries. Then I will make module that will push the extracted meta data to elasticsearch which will then be used for searching.

So this is the brief summary of what i'll be doing this summer.

sagar-kohli