MyMediaLite + ServiceStack

The first milestone in my PhD project is nearly reached. For the continuation of my research it is necessary to have a recommender system that I can easily manipulate for research.

Since I have already been using the full MyMedia, this would be maybe the easiest way. However, the entire framework that was delivered by this project is somewhat of an overkill for my needs. This project however was forked into MyMediaLite, a lightweight version of the framework which mainly consists of the different recommendation engines.

The drawback of MyMediaLite however is that it is not readily usable as a web service. Enter ServiceStack… This library allows for easy deployment of a .NET application as a web service.

The combination resulted in a .NET application that can be deployed on a web server. Right now the only methods provided are the submitting of ratings and receiving recommendations, both as REST interfaces. Recommendations are returned as JSON object. To do this a number of things had to be changed, because of the lack of thread safety in MyMediaLite. The main change is that ratings are stored in a queue that is processed periodically. Without this addition the data was not consistent, as there is some mapping going on from external ID’s (used in the http requests) and internal ID’s (used in the recommender engine).

Yesterday and today I have been stresstesting the application through Blitz.io and the numbers are quite surprising. The report shows that apparently the application can take 250hits/s easily.

It does have some startup issues, so going from 0 to 250 hits instantaneously causes timeouts. Sadly I cannot try any higher loads with my Blitz account.

So for now the idea is creating a user interface around this application, such that research can continue. We will probably still be using the 10M dataset of GroupLens, as we have done most of our research on this dataset. But with full control of the application it is also time to think of new ideas like visualizing preferences and/or movies, similar to what I have done in my thesis. Time will tell what way to go…