Earning Platinum with warframe.market analyzer
One of the great things about Warframe is that you can unlock pretty much everything in the game without spending a penny and it won't take forever. Warframe is using premium currency (Platinum) that players can purchase from developers and use to speed things up. There is also trading available in-game, where players can interchange all kinds of items including Platinum. This allows earning "Plat" without the necessity of buying it. Trade chat is available; however, it is hard to browse through fast-moving wall of text in search of stuff that we need. This is why external sites, where sellers and buyers can contact each other, are gaining more and more popularity. The most popular one is https://warframe.market. Manually navigating this website when you want to track multiple items can be tedious and this is where our bot comes in. Our goal is to create an automated system that will grab HTML source of an item offers on warframe.market then parse it to find all relevant data, process it and then display it on an overlay. I managed to earn a couple thousand plat in 2 weeks, trading (flipping items) every day for a couple of hours in the "background" without actively playing the game.
Getting items' offers
Every single item on the market has its own unique URL. For the purpose of this post, I will be using famous Maiming Strike mod as an example. Its buy and sell offers can be accessed under https://warframe.market/items/maiming_strike. Let's take a look at page source (manually in a web browser for now). We are looking for username from the first listing to locate a list of offers in the HTML. This fragment looks like what we need (shortened in order to avoid obfuscation):
{"orders": [
{"user": {"reputation_bonus": 0, "reputation": 0, "region": "en", "status": "offline", "ingame_name": "VisibleGhost3", "id": "5629d9fbb66f835d64203df7", "avatar": null}, "creation_date": "iso", "region": "en", "last_update": "iso", "mod_rank": 0, "order_type": "sell", "quantity": 1, "id": "574028140f31392ff0edd151", "platform": "pc", "platinum": 1050, "visible": true},
{"user": {"reputation_bonus": 0, "reputation": 0, "region": "en", "status": "offline", "ingame_name": "Wongerboi", "id": "560348ddb66f83536ca1eadd", "avatar": null}, "creation_date": "iso", "region": "en", "last_update": "iso", "mod_rank": 0, "order_type": "buy", "quantity": 1, "id": "579efbbd0f31393850375f4c", "platform": "pc", "platinum": 50, "visible": true},
{"user": {"reputation_bonus": 0, "reputation": 0, "region": "en", "status": "offline", "ingame_name": "Wongerboi", "id": "560348ddb66f83536ca1eadd", "avatar": null}, "creation_date": "iso", "region": "en", "last_update": "iso", "mod_rank": 0, "order_type": "sell", "quantity": 1, "id": "57b2a0d7d3ffb650f4503c55", "platform": "pc", "platinum": 950, "visible": true},
{"user": {"reputation_bonus": 0, "reputation": 0, "region": "en", "status": "offline", "ingame_name": "Mokrez", "id": "571d17730f31390530831276", "avatar": null}, "creation_date": "iso", "region": "en", "last_update": "iso", "mod_rank": 0, "order_type": "buy", "quantity": 1, "id": "57e1135d0f31391942e95618", "platform": "pc", "platinum": 100, "visible": true}]}
It seems that every available listing is here and the actual web is applying all kinds of filters to present relevant data to an end-user (to filter online status, min/max mod rank or price, etc.). Now it's time for regular-expressions-fun, as we need to extract all the offers.
By analyzing multiple item page sources, I noticed that fields (by field I mean "field_name" : value pair) in single listing are in random order and user (compound field containing list of sub-fields within { } brackets) is not necessarily the first one; thus, we need to use following regex match to find all fields in a single offer.
Regex reg = new Regex(@"{([^{]*)""user"": {([^}]*)}([^}]*)}", RegexOptions.IgnoreCase);
Match m = reg.Match(htmlData);
Regular expressions are not for the faint of heart 😀 If you are not familiar with them take a look at this introduction https://regexone.com. To make things even harder there are multiple "standards"; however, for our purposes, we care about only one of them. Here is how I get website source and extract all offers:
List<ItemOffer> GetItemOffers(Request request)
{
List<ItemOffer> result = new List<ItemOffer>();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; // required for HTTPS websites (TLS version may vary between sites)
using (WebClient client = new WebClient())
{
try
{
string htmlCode = client.DownloadString("https://warframe.market/items/" + request.ItemName);
result = ParseOffers(htmlCode, request.ItemName);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
return result;
}
List<ItemOffer> ParseOffers(string htmlData, string itemName)
{
List<ItemOffer> result = new List<ItemOffer>();
Regex reg = new Regex(@"{([^{]*)""user"": {([^}]*)}([^}]*)}", RegexOptions.IgnoreCase);
Match m = reg.Match(htmlData);
while (m.Success)
{
ItemOffer entry = new ItemOffer();
entry.Fields = ParseFields(m.Groups[1].Value + m.Groups[3].Value);
entry.UserFields = ParseFields(m.Groups[2].Value);
result.Add(entry);
m = m.NextMatch();
}
return result;
}
Dictionary<string, string> ParseFields(string entryStr)
{
Dictionary<string, string> result = new Dictionary<string, string>();
Regex reg = new Regex(@"""([^:]+)"": ([^,}]+)", RegexOptions.IgnoreCase);
Match m = reg.Match(entryStr);
while (m.Success)
{
result[m.Groups[1].Value] = m.Groups[2].Value.Replace("\"", string.Empty); // remove '"' from string values
m = m.NextMatch();
}
return result;
}
Analysis
Since we managed to extract all offers data, now it's time to analyze it. What we're interested in, is finding the lowest seller and highest buyer along with price difference to second-best offer (to identify potential "special" deals when someone is selling extra cheap or buying super high). That is really simple. We need to filter out everybody who is not "in-game" (or whatever minimum online status you want to consider) and then sort all remaining sell offers ascending and buy offers descending.
Statistics
Not everybody is familiar with market trends, so warframe.market introduced statistics to help people figure out how particular item value changed over time. We could definitely use that information to better understand what is currently happening on the market. Here are stats for our MaimingStrike mod. Statistics are stored with an hour precision and I assume represent average transaction price from all transactions made within that hour. When no transactions were made there will be blank spot without data node.
First, we need to find chart data. This is basically the same procedure as with item offers, but this time we will be looking for '48hours' array of fields. Next, we will compute our own 24h average price using moving average values from the chart. Keep in mind that there is a node in chart only when at least one item was sold or bought within given hour.
List<ItemSaleStat> GetItemStats(Request request)
{
List<ItemSaleStat> result = new List<ItemSaleStat>();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
using (WebClient client = new WebClient())
{
try
{
string htmlCode = client.DownloadString("https://warframe.market/items/" + request.ItemName + "/statistics");
result = ParseData<ItemSaleStat>(htmlCode, @"""48hours"": \[([^\]]+)\]");
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
return result;
}
List<T> ParseData<T>(string htmlData, string pattern) where T : Data, new()
{
List<T> result = new List<T>();
Match dataMatch = new Regex(pattern, RegexOptions.IgnoreCase).Match(htmlData);
if (dataMatch.Success)
{
string data = dataMatch.Groups[1].Value;
Regex reg = new Regex(@"{([^}]*)}", RegexOptions.IgnoreCase);
Match m = reg.Match(data);
while (m.Success)
{
T entry = new T();
entry.Fields = ParseFields(m.Groups[1].Value);
result.Add(entry);
m = m.NextMatch();
}
}
return result;
}
Time used in nodes is in UTC standard. All we need to do is compute average from all nodes within last 24 hours timeframe.
class ItemSaleStat : Data
{
public DateTime GetDateTime() { return DateTime.ParseExact(Fields["datetime"], "yyyy-MM-ddTHH:mm:ss+00:00", CultureInfo.InvariantCulture); }
public float GetAvgPrice() { return float.Parse(Fields["avg_price"]); }
public float GetMovingAvgPrice() { return Fields.ContainsKey("moving_avg") ? float.Parse(Fields["moving_avg"]) : -1; }
}
public class AnalyzedSaleStats
{
public AnalyzedSaleStats(List<ItemSaleStat> stats)
{
List<ItemSaleStat> reverseStats = stats.Reverse<ItemSaleStat>().ToList();
if (stats.Count > 0)
{
float movingAvg = reverseStats.First().GetMovingAvgPrice();
CurrentAvg = (int)Math.Round(movingAvg != -1 ? movingAvg : stats.Last().GetMovingAvgPrice());
}
//all time is in UTC timezone
DateTime currentTime = DateTime.UtcNow;
DateTime timeLimitFor24H = currentTime + new TimeSpan(-24, 0, 0);
bool updated24h = false;
float sum = 0;
int entriesNum = 0;
for (int i = 0; i < reverseStats.Count; ++i)
{
DateTime statTime = reverseStats[i].GetDateTime();
if (statTime < timeLimitFor24H && !updated24h)
{
if (entriesNum > 0) Avg24h = (int)Math.Round(sum / entriesNum);
updated24h = true;
}
float avgPrice = reverseStats[i].GetMovingAvgPrice();
if (avgPrice > 0)
{
sum += reverseStats[i].GetMovingAvgPrice();
++entriesNum;
}
}
if (!updated24h && entriesNum > 0)
{
Avg24h = (int)Math.Round(sum / entriesNum);
}
}
public int CurrentAvg = -1;
public int Avg24h = -1;
}
Summary
Now, all we need to do is put all this information on an overlay. This is nothing new, so I will let myself skip this part and present the final result (along with some explanation).
Above techniques can be used to implement bots for all kinds of web browser games that are HTML based. As usual, the full source code is available for patrons on https://www.patreon.com/bottersgonnabot. This time I also included a compiled version for those of you who just want to try their trading skills. The analyzer is fully configurable through requests script, where you can define what items you want to track along with minimum mod rank (only for mods of course :D) and a refresh frequency (just please don't DoS the website with 5 seconds recurring requests :P). Here are contents of sample requests.txt file:
;item_name:min_rank:refresh_interval_in_seconds maiming_strike:0:120 ;argon_scope:0:100 primed_reach:10:90
Overlay's opacity and position can be configured through a command line. Here are all supported arguments:
-position_x <value> ;overlay x position on screen -position_y <value> ;overlay y position on screen -display_no <value> ;display(monitor) id to render overlay on - 0 is default -top_left ;automatically place overlay in top left corner of selected display -top_right ;automatically place overlay in top right corner of selected display -bottom_left ;automatically place overlay in bottom left corner of selected display -bottom_right ;automatically place overlay in bottom right corner of selected display -opacity <value> ;overlay opacity float between 0 and 1 -username <value> ;your username on warframe.market will be displayed differently when your offer is the best one -requests_file <path> ;full path to requests script -no_hotkeys ;disable analyzer hotkeys (as it may be conflicting with other programs for some people)
There is albo a bunch of hotkeys:
Ctrl+0-9 ;copy "want to buy" whisper message to the top seller to the clipboard so you can easily just paste it into Warframe chat Alt+0-9 ;copy "want to sell" whisper message to the top buyer to the clipboard so you can easily just paste it into Warframe chat Ctrl+Alt+S ;show/hide overlay Ctrl+Alt+O ;show offers for at least online/in-game users
"Investors smiling on that Capitalization!"