Earning potentially even more Platinum with riven.market analyzer
Trading rivens is rather difficult and time-consuming task. You need quiet experience to properly assess riven value and spend a lot of time in trading chat to find a buyer for a fair price. Fortunately, there are websites that make that task a little easier. One of them is riven.market (as of right now it supports only selling offers). Today we will implement an extension to warframe.analyzer for that site, that will allow us to quickly track prices for rivens we are interested in buying. This time we will be fetching the website and parsing HTML to extract all needed data. Check out my previous post for additional explanation related to downloading website source in C#. Patrons have both source and executable available for download.
Requests
We will be following the same pipeline as in the previous post. For each riven we are interested in there will be a request defined in a script file. I decided to use the following format:
;weapon:refresh_freq:max_rerolls:stats_requirements Akstiletto:100:-1:Damage+Multi Opticor:100:-1:CritChance+Damage|Multi Baza:100:-1:
Rules for stats requirements are pretty simple. Required stats are separated with '+'. In Akstiletto request I need a riven to have both damage and multishot stats. Sometimes I want to say that I need critical chance as one stat and damage or multishot as the second one like I did in Opticor request. In terms of precedence '|' is stronger than '+' as I first split by '+' and then by '|'. You can basically interpret '+' as a start of next stat requirement. Stats' names acronyms are those used by riven.market author and can be found in the page source, however using a few regular expressions I managed to compile following list:
Damage - Damage Multi - Multishot Speed - Fire Rate / Attack Speed Corpus - Damage to Corpus Grineer - Damage to Grineer Infested - Damage to Infested Impact - Impact Puncture - Puncture Slash - Slash Cold - Cold Electric - Electric Heat - Heat Toxin - Toxin ChannelDmg - Channeling Damage ChannelEff - Channeling Efficiency Combo - Combo Duration CritChance - Critical Chance Slide - Slide Attack Critical Chance CritDmg - Critical Damage Finisher - Finisher Damage Flight - Flight Speed Ammo - Ammo Max Magazine - Magazine Capacity Punch - Punch Through Reload - Reload Speed Range - Range StatusC - Status Chance StatusD - Status Duration Recoil - Weapon Recoil Zoom - Zoom
Now we can proceed to make an URL query and parse the result.
Fetching and parsing HTML
The first step is to download website source. At this point, we need to specify which weapon's rivens we are interested in.
public static List<RivenOffer> GetRivens(RivenRequest request)
{
List<RivenOffer> result = new List<RivenOffer>();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
using (WebClient client = new WebClient())
{
try
{
string htmlCode = client.DownloadString("https://riven.market/list?weapon=" + request.WeaponName + "&price=20000&stats=&field=price&sort=asc&page=1");
result = ParseRivenData(htmlCode);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
return result;
}
Now, we need some kind of HTML library to parse the result. I decided to use very popular HtmlAgilityPack. It comes as a NuGet package that can be easily installed from Visual Studio. For VS2017 go to Tools->NuGet Package Manager->Manage NuGet Packages for Soultion...
Our goal is to fill out offer structures.
public class Data
{
public Dictionary<string, string> Fields = new Dictionary<string, string>();
}
public class RivenOffer : Data
{
public string GetName() { return Fields["data-name"]; }
public string GetUser() { return Fields["data-user"]; }
public int GetPlatinum() { return int.Parse(Fields["data-price"]); }
public string GetPolarity() { return Fields["data-polarity"]; }
public int GetRerolls() { return int.Parse(Fields["data-rerolls"]); }
// 4th stat is negative
public string GetStat(int i) { string stat; Fields.TryGetValue("data-stat" + i, out stat); return (stat == null || stat == "- none -") ? "" : stat; }
public float GetStatValue(int i) { string amountStr; Fields.TryGetValue("data-stat" + i + "amount", out amountStr); return (amountStr != null && amountStr.Length > 0) ? float.Parse(amountStr) : 0; }
public OnlineStatus GetOnlineStatus() { string onlineStatusStr = Fields["data-onlinestatus"]; return onlineStatusStr == "offline" ? OnlineStatus.Offline : (onlineStatusStr == "online" ? OnlineStatus.Online : (onlineStatusStr == "ingame" ? OnlineStatus.InGame : OnlineStatus.Unknown)); }
//...
}
Here is single entry extracted and formatted manually from page source to figure out what we actually need to find.
<div class="riven" data-rank="8" data-wtype="Secondary" data-weapon="Sicarus" data-name="Croni-Hexaata" data-price="1200" data-polarity="naramon" data-stat1="StatusC" data-stat1amount="108.0" data-stat2="Damage" data-stat2amount="262.1" data-stat3="Speed" data-stat3amount="90.4" data-stat4="" data-stat4amount="0.0" data-mr="11" data-rerolls="12" data-user="4upufFkUS2gSv9BJWlLL39mbyvy1">
<div class="attribute lightbox">
<div class="lightbox wrapper">
<div class="lightbox preview"></div>
<div class="lightbox user">Seller: <span class="name">Prime_Prime</span></div>
<div class="lightbox user">Price: <span class="name">1200 <img src="/_modules/riven/img/platinum.png"></span></div>
</div>
</div>
<div class="attribute image"><i class="material-icons">zoom_in</i><div class="image container"></div>
</div>
<div class="attribute type">Secondary</div>
<div class="attribute weapon">Sicarus</div>
<div class="attribute name">Croni-Hexaata</div>
<div class="attribute stats"></div>
<div class="attribute price">1200 <img src="/_modules/riven/img/platinum.png"></div>
<div class="attribute polarity"><img src="/_modules/riven/img/naramon.png">Naramon</div>
<div class="attribute rank">8</div>
<div class="attribute mastery">11</div>
<div class="attribute rerolls"><img src="/_modules/riven/img/rerolls.png">12</div>
<div class="attribute seller" data-id="4upufFkUS2gSv9BJWlLL39mbyvy1">Prime_Prime</div>
<div class="attribute online ingame">done_all
<div class="hovermenu">
<a href="/profile?user=Prime_Prime"><i class="material-icons">person_outline</i>Profile</a>
<a href="javascript:void(0)" onclick="openConversation(this);"><i class="material-icons">chat</i>Message</a>
<a class="nomobile" href="javascript:void(0)" onclick="copyName(this);"><i class="material-icons">content_copy</i>Copy to Clipboard</a>
</div>
</div>
</div>
It looks like we need to get all divisions (<div> tags) having riven class. Fortunately, everything except a user data is already in attributes so we will just use them in offer data structure treating attribute name as a field key and its value as a field value :). Seller's name can be obtained by getting the inner text from the division with attribute seller class. To check his/her online status we need to verify a presence of division with attribute online/online ingame. Here is the code for doing all that:
static List<RivenOffer> ParseRivenData(string htmlData)
{
List<RivenOffer> result = new List<RivenOffer>();
HtmlAgilityPack.HtmlDocument html = new HtmlAgilityPack.HtmlDocument();
html.LoadHtml(htmlData);
List<HtmlAgilityPack.HtmlNode> rivenNodes = html.DocumentNode.Descendants().Where(x => (x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Equals("riven"))).ToList();
foreach (var node in rivenNodes)
{
RivenOffer rivenData = new RivenOffer();
foreach (var attr in node.Attributes)
{
rivenData.Fields[attr.Name] = attr.Value;
}
HtmlAgilityPack.HtmlNode userNode = node.Descendants().Where(x => x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Equals("attribute seller")).ToList().First();
rivenData.Fields["data-user"] = userNode.InnerText;
// online status
string onlineStatus = "offline";
if (node.Descendants().Where(x => x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Equals("attribute online ingame")).Count() > 0)
{
onlineStatus = "ingame";
}
else if (node.Descendants().Where(x => x.Name == "div" && x.Attributes["class"] != null && x.Attributes["class"].Value.Equals("attribute online")).Count() > 0)
{
onlineStatus = "online";
}
rivenData.Fields["data-onlinestatus"] = onlineStatus;
result.Add(rivenData);
}
return result;
}
Analyzing
Now when we have all the offers we need to run them through request's requirements.
public class RivenOffer : Data
{
//...
public bool HasStats(List<List<string>> stats)
{
UpdateStats();
foreach (var values in stats)
{
bool statCriteriaMet = false;
foreach (string stat in values)
{
if (Stats.Contains(stat))
{
statCriteriaMet = true;
break;
}
}
if (!statCriteriaMet)
return false;
}
return true;
}
private void UpdateStats()
{
if (Stats == null)
{
Stats = new List<string>();
for (int i = 1; i <= 3; ++i)
Stats.Add(GetStat(i));
NegativeStat = GetStat(4);
}
}
private List<string> Stats;
private string NegativeStat;
}
public class RivenRequest
{
public string WeaponName;
public string DisplayItemName;
public int MaxRerolls;
public List<List<string>> ReqStats = new List<List<string>>();
public int RefreshFreq; // in seconds
public Stopwatch TimeSinceUpdate = new Stopwatch();
public string GetId() { return WeaponName + MaxRerolls; }
}
public class AnalyzedRivenRequest
{
public AnalyzedRivenRequest(RivenRequest request, List<RivenOffer> offers)
{
Req = request;
AllOffers = new List<RivenOffer>(offers);
ReAnalyze();
}
public void ReAnalyze()
{
List<RivenOffer> offers = new List<RivenOffer>(AllOffers);
AllOffers = offers.FindAll(x => (Req.MaxRerolls < 0 || x.GetRerolls() <= Req.MaxRerolls) && x.HasStats(Req.ReqStats)).OrderBy(x => x.GetPlatinum()).ToList();
if (AllOffers.Count() > 0)
{
TopSeller = AllOffers.First();
if (AllOffers.Count() > 1)
{
PriceGapToNextSeller = TopSeller.GetPlatinum() - AllOffers.ElementAt(1).GetPlatinum();
PriceGapToNextSellerPct = PriceGapToNextSeller / AllOffers.ElementAt(1).GetPlatinum();
}
else
{
PriceGapToNextSeller = PriceGapToNextSellerPct = float.MaxValue;
}
}
}
public RivenOffer TopSeller;
public float PriceGapToNextSeller;
public float PriceGapToNextSellerPct;
public List<RivenOffer> AllOffers;
public RivenRequest Req;
}
We are basically doing the very same thing we did with items' offers in warframe.market analyzer, except for additional stats requirements check.
As usual, I will skip visualization step, if you are interested please take a look at source code. Here is final result:
And when you want to track both markets it looks like this:
The full source code is available for patrons on https://www.patreon.com/bottersgonnabot. Enjoy!