A word on debugging
No matter how much we try to create a "bugless" code, sooner or later the time will come when we have to debug it. Over the years I came up with a few practices that helped me a lot in some dark times. These are not revolutionary in any way and most of you are already using them. However for those who are new to this kind of stuff, they may save them some time... and sanity. All discussed techniques are not specific to bot programming, I'm using them on daily basis during game development.
Supervised
Techniques from this category of are usually used when we are actively debugging in search for a reproducible bug.
Let's say we are checking colors of bunch of pixels to determine game state and we are not getting expected results. We could break after every step and analyze contents of whatever data structures we use to store the results... or we could visualize that data to actually see where it fails. It's much more natural way of analyzing data for humans. When developing game we have full access to code so we implement all kinds of debug renderers to draw stuff on screen. When it comes to bot development that is not the case so we have to use overlays.
Overlaying
Overlay is just a transparent window that is being displayed on top of a game window. There is no connection between these two, so it's pretty safe to use (unless some anti-cheating system is making screenshots and someone is analyzing them). Basically we need to create a window with bunch of hack-ish flags to make it transparent and click/keytrough. Then copy position and dimensions from a game window and implement some helper functions so we can easily draw basic shapes and render text. I'm using both GDI and DirectX based overlays. Here is example of GDI one.
using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Windows.Forms;
public class BaseOverlay : Form
{
[StructLayout(LayoutKind.Sequential)]
public struct RECT
{
public int Left; // x position of upper-left corner
public int Top; // y position of upper-left corner
public int Right; // x position of lower-right corner
public int Bottom; // y position of lower-right corner
}
[DllImport("user32.dll", SetLastError = true)]
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);
[DllImport("user32.dll", SetLastError = true)]
public static extern bool GetWindowRect(IntPtr hWnd, out RECT lpRect);
[DllImport("user32.dll")]
public static extern int SetWindowLong(IntPtr hWnd, int nIndex, int dwNewLong);
[DllImport("user32.dll")]
public static extern long GetWindowLong(IntPtr hWnd, int nIndex);
//https://msdn.microsoft.com/en-us/library/windows/desktop/ms633545(v=vs.85).aspx
// alternate options for hWndInsertAfter
public static IntPtr HWND_BOTTOM = (IntPtr)1;
public static IntPtr HWND_NOTOPMOST = (IntPtr)(-2);
public static IntPtr HWND_TOP = (IntPtr)0;
public static IntPtr HWND_TOPMOST = (IntPtr)(-1);
// options for wFlags
public static int SWP_NOSIZE = 0x0001;
public static int SWP_NOMOVE = 0x0002;
public static int SWP_NOZORDER = 0x0004;
public static int SWP_SHOWWINDOW = 0x0040;
public static int SWP_HIDEWINDOW = 0x0080;
[DllImport("user32.dll", SetLastError = true)]
public static extern bool SetWindowPos(IntPtr hWnd, IntPtr hWndInsertAfter, int x, int Y, int cx, int cy, int wFlags);
public BaseOverlay(string wnd_name)
{
m_WndName = wnd_name;
Name = "Overlay";
Text = "Overlay";
FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
TransparencyKey = Color.Maroon;
BackColor = Color.Maroon;
TopMost = true;
DoubleBuffered = true;
Paint += new PaintEventHandler(Render);
update_timer.Enabled = true;
update_timer.Interval = 5;
update_timer.Tick += new EventHandler(Update);
SetWindowLong(Handle, -20, (int)GetWindowLong(Handle, -20) | 0x00000020); // click/key through
SetStyle(ControlStyles.SupportsTransparentBackColor, true);
}
protected virtual void Render(object sender, PaintEventArgs e)
{
e.Graphics.CompositingQuality = CompositingQuality.GammaCorrected;
e.Graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.AntiAlias;
e.Graphics.Clear(Color.Maroon);
}
protected virtual void Update(object sender, EventArgs e)
{
HookToWnd();
Refresh();
}
protected System.Drawing.Point ConvertToPoint(Vec3 p)
{
return new System.Drawing.Point((int)p.X, (int)p.Y);
}
protected void RenderLine(Graphics g, Pen pen, Vec3 start, Vec3 end)
{
g.DrawLine(pen, ConvertToPoint(start), ConvertToPoint(end));
}
protected void RenderPoint(Graphics g, Pen pen, Vec3 pos, float radius)
{
System.Drawing.Point p = ConvertToPoint(pos);
g.FillRectangle(new SolidBrush(pen.Color), pos.X, pos.Y, 1, 1);
}
protected void RenderCircle(Graphics g, Pen pen, Vec3 pos, float radius)
{
const int LINES_PER_CIRCLE = 32;
const float ANGLE = 360 / (float)LINES_PER_CIRCLE;
Vec3 normal = new Vec3(0, 0, 1);
Vec3 dir = new Vec3(1,0,0);
Vec3 next_dir = null;
for (int i = 0; i < LINES_PER_CIRCLE; ++i)
{
next_dir = Vec3.Rotate(dir, ANGLE, normal);
RenderLine(g, pen, pos + dir * radius, pos + next_dir * radius);
dir = next_dir;
}
}
protected void RenderString(Graphics g, Brush brush, string text, float size, Vec3 pos)
{
g.DrawString(text, new Font("Arial", size), brush, ConvertToPoint(pos));
}
private void HookToWnd()
{
if (m_WndHandle == IntPtr.Zero)
m_WndHandle = FindWindow(null, m_WndName);
if (m_WndHandle != IntPtr.Zero)
{
RECT rect;
if (GetWindowRect(m_WndHandle, out rect))
{
if (rect.Left == -32000)
{
// the game is minimized
WindowState = FormWindowState.Minimized;
}
else
{
WindowState = FormWindowState.Normal;
Location = new System.Drawing.Point(rect.Left + m_TrimLeft, rect.Top + m_TrimTop);
int height = (rect.Bottom - rect.Top) - m_TrimBottom - m_TrimTop;
int width = (rect.Right - rect.Left) - m_TrimLeft - m_TrimRight;
Height = height; // do not overlap bottom
Width = width;
ClientSize = new Size(width, height);
SetWindowPos(Handle, HWND_TOPMOST, 0, 0, 0, 0, SWP_NOMOVE | SWP_NOSIZE);
}
}
else
{
// something wrong with window, may not exist
m_WndHandle = IntPtr.Zero;
WindowState = FormWindowState.Minimized;
}
}
}
protected IntPtr m_WndHandle = IntPtr.Zero;
protected string m_WndName;
protected int m_TrimLeft = 0;
protected int m_TrimRight = 0;
protected int m_TrimBottom = 0;
protected int m_TrimTop = 0;
private Timer update_timer = new Timer();
public static readonly Font SMALL_FONT = new Font("Arial", 8);
public static readonly Font SMALL_BOLD_FONT = new Font("Arial", 8, FontStyle.Bold);
public static readonly Font NORMAL_FONT = new Font("Arial", 9);
public static readonly Font NORMAL_BOLD_FONT = new Font("Arial", 9, FontStyle.Bold);
public static readonly Font BIG_FONT = new Font("Arial", 10);
public static readonly Font BIG_BOLD_FONT = new Font("Arial", 10, FontStyle.Bold);
public static readonly Pen BOLD_BLACK_PEN = new Pen(Color.Black, 3);
public static readonly Pen BOLD_WHITE_PEN = new Pen(Color.White, 3);
public static readonly Pen BOLD_RED_PEN = new Pen(Color.Red, 3);
public static readonly Pen BOLD_GREEN_PEN = new Pen(Color.Green, 3);
public static readonly Pen BOLD_BLUE_PEN = new Pen(Color.Blue, 3);
public static readonly Pen BOLD_LIGHT_BLUE_PEN = new Pen(Color.LightBlue, 3);
}
This is just a base class. It contains all basic functionality. All we need to do to create an overlay for our next "victim" game is derive from this class and override Render function. Overlays are super useful when it comes to rendering paths, positions and overall status/state data. Here is a sample from my Diablo 3 bot.
Usually it's useful to use color coding. For example when you are drawing path you can use different colors based on what is at the destination. By the way, that pixelized stuff you can see in upper left corner is another overlay...
Recording and analyzing
So we can display whole bunch of helpful stuff on screen, but sometimes things happen so fast that it's hard to tell what exactly happened. This is where video recording software comes with help. What we need to do in that case is to turn on all relevant debug information on an overlay and start recording. As soon as we were able to capture the issue we can replay the video frame by frame and see what actually happened. Here are some tools that I'm using:
- Bandicam - for recording well compressed yet high quality videos (paid, unregistered version can record for limited time)
- OBS Studio - very popular and powerful tool for recording and streaming (free)
- VLC - popular video player that has frame by frame playback (E key) unfortunately supports only forward playback
- VirtualDub - really old but super useful software to go frame by frame back and forth (however it can be hard to make it work with all video formats, however works well with Bandicam output videos)
I found that super handy when it comes to debugging fast moving stuff like skill checks or some weird behaviors (like bot continuously switching target or destination every frame). Some bugs are nearly impossible to capture when stepping in the debugger frame by frame because they are "real" time dependent. Aside from that sometimes you don't know when something will go wrong or you don't even know where to start. In that scenarios record and replay technique really shines.
Unsupervised
Now I assume that we have some kind of bot doing stuff when we sleep or are away from a computer. When things go bad we would like to know as much about it as possible so we can fix it and improve our bot for a future. My goal for Diablo 3 bot was to be able to run it unsupervised for at least a month while I went on a vacation. Before I was able to pull that off I had to track and fix all kinds of issues. The thing with bot development is pretty much the same as with any software development. As we continue to fix frequently happening (common) bugs we are left with those occurring either randomly or very rarely (legendary bugs). Sometimes you need to run a bot for multiple hours for some issues to happen. For that case we need to develop methods for detecting and gathering information about what went wrong, so we can analyze it later on. These methods will differ from project to project and I will go into more detail during Diablo 3 botting series at some point.
Screenshots and logs
Making screenshots is the simplest way to capture debug data. When we detect that something wrong happened (like our bot was reported stuck by navigation system) we can save a print screen including overlay in hope that whatever we see on the screen will help us figure out a reason behind this failure. That will not necessarily allow us to solve an issue right away, however we may be able to add some information to the overlay and/or logs so we can narrow down the number of possible causes. Detailed logs can definitely shed some light on events that might have led to the issue. For example when we are tracking some navigation related issues, along with screenshot and logs, we could also dump whole navigation state to separate file so later on we can load it to our tool and analyze it.
Recordings
Next step is recording single runs/rounds/matches (whatever that bot is playing). At the end of every single one of them we can decide to keep the video or remove it (to save disk space). At the start of every D3 bot run I was simply sending key which was defined as a shortcut responsible for starting recording in recording software. Then I was making sure that recording started, by checking if new file was created. At the end of a run I was stopping the recording the same way as I started it. Then I was checking if this run was a failure and if it was I was renaming video file to contain some sort of issues description.
I found it helpful to move Visual Studio output window near the game window so I was able to record game and output synchronously. That made it so much simpler to analyze logs.
Emails/Dropbox
This is not necessarily a debugging technique but it may be of use in overall development. In early stages of D3 bot project I was automatically sending myself an email after each run with screenshot attached. For screenshot I was hovering mouse over XP bar so current amout XP would be displayed and captured on screenshot. This was a really brute force way of gathering stats as I wasn't doing any memory readings. Here is the function for sending email in AutoIt which I was using.
#include <file.au3>
;AutoIt Error Object (Prevents script from crashing if an unexpected error occurs
Global $oMyError = ObjEvent("AutoIt.Error", "ErrorDebug")
Func _INetSmtpMailCom($s_SmtpServer, $s_FromName, $s_FromAddress, $s_ToAddress, $s_Subject = "", $as_Body = "", $s_AttachFiles = "", $s_CcAddress = "", $s_BccAddress = "", $s_Importance = "High", $s_Username = "", $s_Password = "", $IPPort = 25, $ssl = 0)
$objEmail = ObjCreate("CDO.Message")
$objEmail.From = '"' & $s_FromName & '" <' & $s_FromAddress & '>'
$objEmail.To = $s_ToAddress
Local $i_Error = 0
Local $i_Error_desciption = ""
If $s_CcAddress <> "" Then $objEmail.Cc = $s_CcAddress
If $s_BccAddress <> "" Then $objEmail.Bcc = $s_BccAddress
$objEmail.Subject = $s_Subject
If StringInStr($as_Body, "<") And StringInStr($as_Body, ">") Then
$objEmail.HTMLBody = $as_Body
Else
$objEmail.Textbody = $as_Body & @CRLF
EndIf
If $s_AttachFiles <> "" Then
Local $S_Files2Attach = StringSplit($s_AttachFiles, ";")
For $x = 1 To $S_Files2Attach[0]
$S_Files2Attach[$x] = _PathFull($S_Files2Attach[$x])
;ConsoleWrite('@@ Debug(62) : $S_Files2Attach = ' & $S_Files2Attach & @LF & '>Error code: ' & @error & @LF) ;### Debug Console
If FileExists($S_Files2Attach[$x]) Then
$objEmail.AddAttachment($S_Files2Attach[$x])
EndIf
Next
EndIf
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = $s_SmtpServer
If Number($IPPort) = 0 Then $IPPort = 25
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = $IPPort
;Authenticated SMTP
If $s_Username <> "" Then
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate") = 1
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusername") = $s_Username
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendpassword") = $s_Password
EndIf
If $ssl Then
$objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpusessl") = True
EndIf
;Update settings
$objEmail.Configuration.Fields.Update
; Set Email Importance
Switch $s_Importance
Case "High"
$objEmail.Fields.Item("urn:schemas:mailheader:Importance") = "High"
Case "Normal"
$objEmail.Fields.Item("urn:schemas:mailheader:Importance") = "Normal"
Case "Low"
$objEmail.Fields.Item("urn:schemas:mailheader:Importance") = "Low"
EndSwitch
$objEmail.Fields.Update
; Sent the Message
;ConsoleWrite("echo Sending" & @LF)
If Not IsObj($objEmail) Then
;ConsoleWrite("echo error sending (not an object)" & @CRLF)
$sendfailed = True
Else
$sendfailed = False
$objEmail.Send
EndIf
;ConsoleWrite("echo SENT" & @LF)
If @error Then
$sendfailed = True
SetError(2)
Return 0
EndIf
$objEmail = 0
EndFunc ;==>_INetSmtpMailCom
Func ErrorDebug()
Local $HexNumber
Local $strMsg
$HexNumber = Hex($oMyError.Number, 8)
$strMsg = "Error Number: " & $HexNumber & @CRLF
$strMsg &= "WinDescription: " & $oMyError.WinDescription & @CRLF
$strMsg &= "Script Line: " & $oMyError.ScriptLine & @CRLF
FileWrite(@ScriptDir & "\" & @ScriptName & "_ErrorDebug.txt", "------------------[Error]------------------" & @LF & $strMsg & @LF & @LF)
SetError(1)
EndFunc ;==>ErrorDebug
And here is sample mail that bot was sending (this particular mail was actually sent by memory reading version).
Of course soon my mailbox become spammed so I moved to Dropbox. Instead of sending email I was saving some stats and screenshots to Dropbox folder. I was immediately notified when any files was updated and could access them even from my smartphone. Here is some old sample of status file that was continuously updated by D3 bot.
####### BACKPACK ####### Ramaladni's Gift x84 Veiled Crystal x5000 Forgotten Soul x1163 Bottomless Potion of the Leech The Compass Rose* [644 Primary|644 Int|550 Vit|12 MS|8 RCR|1 Socket] Q: 94% GQ: 40,608% Marquise Emerald x3026 Halo of Arlyse* [634 Primary|634 Int|590 Vit|5 CC|1 Socket] Q: 109% GQ: 56,955% Firebird's Plume* [984 Primary|984 Int|972 Vit|201 Res|1 Socket] Q: 160% GQ: 77,16% Firebird's Down* [640 Primary|640 Int|550 Vit|675 Armor|201 Res|2 Socket] Q: 182% GQ: 105,304% Greater Rift Keystone x7 Marquise Amethyst x3021 Firebird's Tarsi* [650 Primary|650 Int|578 Vit|126 AllRes|6500 LifePerSec] Q: 213% GQ: 80,62% Veiled Crystal x3123 Strongarm Bracers [444 Primary|444 Int|441 Vit|5 CC] Q: 43% GQ: 56,233% ####### EQUIPPED ####### PlayerShoulders: Firebird's Pinions* [630 Primary|630 Int|598 Vit|12 LifePct|8 CDR] Q: 135% GQ: 71,342% PlayerLeftFinger: Halo of Arlyse [498 Primary|498 Int|6 CC|49 CHD|1 Socket] Q: 99% GQ: 75,536% PlayerLegs: Firebird's Down* [560 Primary|560 Int|646 Vit|129 AllRes|2 Socket] Q: 136% GQ: 73,24% PlayerRightHand: Chantodo's Force* [996 Primary|996 Int|9,5 CC|7 CDR|434 MinDmg|522 MaxDmg|478 AvgDmg] Q: 179% GQ: 114,117% PlayerWaist: Fazula's Improbable Chain* [627 Primary|627 Int|614 Vit|512 Armor|122 AllRes] Q: 143% GQ: 74,446% PlayerFeet: Firebird's Tarsi* [579 Primary|579 Int|641 Vit|503 Armor|112 AllRes] Q: 172% GQ: 101,17% PlayerHead: Firebird's Plume* [980 Primary|980 Int|897 Vit|6 CC|12 CDR|1 Socket] Q: 147% GQ: 100,053% PlayerHands: Firebird's Talons* [986 Primary|986 Int|10 CC|43 CHD|8 CDR] Q: 125% GQ: 95,016% PlayerBracers: Ancient Parthan Defenders* [632 Primary|632 Int|636 Vit|5,5 CC|18 ElemDmgPct] Q: 142% GQ: 96,137% PlayerTorso: Firebird's Breast* [619 Primary|619 Int|608 Vit|742 Armor|3 Socket] Q: 141% GQ: 74,08% PlayerLeftHand: Chantodo's Will* [837 Primary|837 Int|844 Vit|130 CHD|9 DmgPct|1654,62 MinWeaponDmg|2042,66 MaxWeaponDmg|1848,6 AvgWeaponDmg|1 Socket|3007,746 WeaponDps] Q: 133% GQ: 108,42% PlayerNeck: The Traveler's Pledge [9,5 CC|80 CHD|20 ElemDmgPct|1 Socket] Q: 94% GQ: 93,75% PlayerRightFinger: The Compass Rose [466 Primary|466 Int|494 Vit|6 CC|10 LifePct|1 Socket] Q: 85% GQ: 54,912% Play time: 14h49m(stats) 16h40m(total) Rest time: 1h54m Items salvaged: 2189 Legendaries salvaged: 506 XP: 55701M/h(old) 49521M/h XP gained: 825,831,777,484 Level: 70 Paragon: 1345 Deaths: 77 Crashes: 2 Runs stats: GreaterRift: 39/59 Rift: 35/42 Inventory space layout: XXXX-XXXE- XXX----XE- XXX----E-- XXX--X-E-- XXXX-X---E XXXEE----E
These are the most generic debugging techniques that I came up with. If you know other interesting ways to debug these epic bugs please leave a comment so we can all profit!