Let’s talk about prices

Neopian market is very volatile. One proof of it is that itemdb updates about 1,800 prices every day. But how does itemdb sets its prices?

Starting with the basics

First of all we need data and data may come in different ways. Most of it comes from our incredible user base that uses the Item Data Extractor script – the script allows us to grab price info when you search for an item on  (Super?) Shop Wizard for example.

We also have trade lots. We can’t get this price data fully automatically so we need your help pricing and voting on pricing suggestions.

Thanks to you we get about 20,000 raw price reports every single day.

Processing

Now we have thousands of raw data points and we need to process it and that’s where the All Mighty Algorithm comes into play. Basically it is built with 3 things in mind:

1. Prioritize fresh and reliable data.

We want to always use the most recent data possible to calculate an item’s price. Also some sources are more reliable than others (SSW is far more reliable than Usershops for example). If we don’t have enough of those (fresh and reliable) we look for some older data or wait until we get more fresh ones.

2. Don’t get fooled by manipulators

Neopets is full of bots, weird price manipulation guilds and rich people. And so is the real world. So there is pretty good financial market grade math formulas to detect this and don’t let them contaminate our data. We apply a bunch of them and also manually review every inflation warning. Sometimes things do slip through so it is very useful to click the Report Wrong Price button so we can review and improve our defences.

3. Get the right price

This one is the hardest. We usually have two opposite user types: the ones wanting to sell (seeking higher prices!) and the ones trying to buy (looking for the lowest price). Our goal is to get the lowest stable price. Sometimes that will be the lowest at the time but sometimes that will be some middle ground between the lowest and some of the median. We like to think about itemdb prices not as a current price pinpoint but something like the order of magnitude of that item’s price.

Evolving

Our old algorithm was good, but it had some flaws:

• It waited too long before deciding a price in some cases. Even though it had “”a lot”” of data, it would still wait for fresher ones.
• Didn’t handle well some very edge price-manipulation cases
• No Auction House data

When we were developing itemdb we made a design decision on not considering Auction House data for price estimation – at the time we considered AH data to be not very reliable (tons of “Junk Auctions” and NF discounts). But much has changed ever since.

Auction History button represents 20% of our tracked interactions in the last 24hrs

Auction History became one of the most used features on the site. And frequently Trading Post prices (which used to be our only source for UB items) were very different and raised questions – “Why itemdb says this item costs X when the Auction History says otherwise”

So we rebuilt our pricing algorithm from the ground up – it now also considers Auction House prices alongside all other data sources we have. We added some protections to prevent junk auctions from polluting the data and more manipulation protections.

One big flaw of our previous code was mixing very old + fresh data in some scenarios (especially for old items). We now weight each price report based on things like price, data source, age, stock on sale. That allows us to have a greater control on the final estimated price.

We ran both old and new pricing algorithms for a week and made several adjustments to make sure everything was smooth. Our new code behaves about the same as the old one for buyable items (aka lower than 1M) but on unbuyables it exceeds the previous version – updating prices much faster!

 

This article reflects the state of itemdb at the time of writing. We are in constant development, so things may have changed since then.