Nov 262012
 

While I was writing my follow-up to my previous article about competitive gaming design using case examples, I got sidetracked into a very interesting debate on David Sirlin’s Fantasy Strike forums. I’ve linked it here, in case you want to read it all, but I will attempt to summarise the major points of it here.

David Sirlin (and one other poster on the site) were insistent that all Xbox 360 fighting games have functioning true-skill based matchmaking. Here’s a quote from Sirlin:

All these fighting Xbox games have… skill-based matchmaking. They really and seriously do. HDR does. SC5 does. Even shitty SF HF did. As far as I know VF, SF4, and MvC3 do too. That is, they secretly look at the skill-rating (not necessarily the same as the displayed stats, those can be sugar coated) to give the best match they can amongst people currently available to play.

Also backing this is a very old article I managed to dig up about MS Trueskill on XBox Live (although I argued this kind of detail is not the ‘common knowledge’ Sirlin purported it was!).

Are you man enough to fight with me?

Guile only cares about gender-based matchmaking.

However the counter to this is my own experience, and the experience of every single person I’ve ever spoken to about it who plays fighting games, is that none of these games have effective skill-based matchmaking. They all give the appearance, and the results, of not even trying to match you against anyone even remotely close to your skill level.  Some, like HDR, also fail to match you against an opponent with a decent internet connection to you. To quote myself:

In the 1000s of (ranked) matches I played on HDR I never detected it was managing any form of skill based selection or ping based selection. I’ve always had a seemingly random opponents from all-over the ping range and all-over the skill range of players at all times.

To check I’d not imagined others’ reported experiences I sent out a quick survey question out to all the FG players I could easily access across various online channels to see if any of them thought that any fighting games they played on the XBox gave them decent results with regards to skill-based matchmaking. The results were completely in-line with my view – the only fighting game anyone thought had any kind of functioning skill-based matchmaking was SF4.

So this poses the question to me about how to marry up these two seemingly diametrically opposed facts. Am I, and everyone else I know of, all poor deluded fools who’ve actually been getting great skill-based matchmaking out of our XBox fighting games all along and somehow not realised it? Or is there some other reason why, despite the attempts to matchmake, it isn’t working very well?

Sirlin himself gave two reasons:

1. Small size of online community means there’s no-one of the same skill level available to play.

2. FGs not skill rating people per-character, meaning people are not correctly skill-rated with whichever character they are playing.

And another point was raised that

3. There are many different ways & styles to be at a certain ‘skill level’ in a fighting game that don’t interact well that can lead to lop-sided matches (varying knowledge of character matchups for example).

I completely agree that these points will have a bad effect on skill-based matchmaking, especially the first two. However where I disagree with Sirlin is that these reasons are the be-all and end-all of the problems, and I believe these games still don’t do nearly enough to getting a successful matchmaking system working, because, as I said there:

I’ve always had experiences like getting one of the top few hundred players (with an unplayable 400+ping because they’re in Japan or Mexico) in one match, to getting someone who can’t even do basic moves or block in the next match, even when there’s likely to be 100s of people online.

So I still posit these reasons don’t go far enough to explain the complete disparity of results I, and anyone else I ask to experiment, seem to get. The types of results I see would be the equivalent to a StarCraft II match pitting me against Masters players and Bronze players with the same frequency, or a Halo Reach Arena matching me up against a full team of Onyx players. These things just never happen – so why does the equivalent type of thing happen on fighting games, even allowing for the reasons above by Sirlin. I think that the method of “matchmaking” used in most fighting games actually massively exacerbates and contributes to the problems, rather than alleviating them. I shall attempt to explain why.

Halo Reach Matchmaking

Games like StarCraft II and Halo Reach use a system where all players waiting to play are placed in a “pool” and then they are match-made according to various criteria (including skill), and the actual host for the match is determined behind the scenes. This is what I call actual matchmaking. The vast majority of fighting games instead just allow the individual player to decide whether they want to host or search for a match. The hosters manually setup matches, and the searches then manually select a match from a filtered list of results – which is why I’d call this kind of ‘matchmaking’ a “list filter” system. Alternatively a searcher can click ‘Quick match’ to instantly join what would’ve been one of their filtered list results. This is where I did not realise there was even any ‘filtering’ for skill level of opponent in play, given the results I was getting, whereas Sirlin has said there is. One big problem with the list filter method is that it will always return some matches if it can. It won’t report “zero matches” if there is no-one close to you in skill (and/or connection) it will just give you the best few matches it can find. Quick match is even worse, just putting you into essentially any match at all, even though it’s the best one it can find.

The problem is that the result of this means it is incredibly unlikely for two equally skilled players to ever find each other in a match, even with a large pool of players. I don’t even believe it scales well at all with a large community size, because this will just increase the number of searchers, especially those using ‘Quick match’ who will get dumped into any old game as soon as possible. Even Microsoft’s Trueskill documentation covers this situation on their FAQ:

…in off-peak hours or in situations where there are not enough host sessions available, the match quality can suffer and it may happen that you are getting matched with people of much higher/lower TrueSkill.

My point is, because of the way the systems are set up within almost all fighting games with simple list-filter ‘matchmaking’ that there will NEVER be enough host sessions available to filter against unless there is some huge incentive to actually host matches rather than search. Whereas instead, most FGs make it easier and more incentivised to search for matches. Match searchers are quickly grabbing any and all available “better matches” for any particular player (& also players searching get the annoying “that session is no longer available” message repeatedly too).

To explain this with an example, imagine there are 100 players online playing 1v1 StarCraft II matches on your server at a given time. Some will be mid-game, some will be in a queue waiting to play at the same time you are. If there are 10 players pretty close to you in skill and connection out of the 100, as long as one of them is in the match queue at the same time you are (perhaps the queue takes 1 minute?) then you will get matched together. The queue can wait for as long as the developers has deemed necessary to get a good chance at a good match for you; essentially waiting for one of those 10 to be available to play a new game. It could even scale this wait time based around the amount of people online, or it could even wait for ‘as long as it took’ to get a good skill match. As ‘bad’ as this might sound, compare it to the fighting game example where, to get matched against one of those 10 players who is similar to you, they have to be hosting at pretty much the exact same time you are searching (or vice-versa), and they have to have not already been matched with any one of the other 90 players coming free and hosting/searching before you happen to get to them.

I believe this problem, more than any other is the crux of the problems when it comes to matchmaking not “working” in fighting games – or any other game with a list filter matchmaking. This “list filter” system will only work as long as there are plentiful hosts and they remain as open sessions for some time, so it can work for large “lobby” style matches, but not for 1v1 games. In the FG system, host matches are immediately snapped up and filled by a single searcher, and so very few are ever available at any given moment in time. It could work better, if instead of closing matches as soon as they were filled, hosted ‘sessions’ stayed open and allowed 4 skill-matched players into a ‘room’. It could also work better if searches would report no results if there was actually no-one close in skill, rather than responding with the ‘next best’ matches which reduces the chances of similarly skilled players from ever finding each other even further. But really what would be far better would be to use the StarCraft II system and actually let the players build up in a pool before they are matched. Even better would be to allow players more control over how their own matchmaking works, tailoring it to their preferences as to whether they care about skill level or ping connection of opponents for example.

As noted above, the one fighting game that seems to have gotten around this effect successfully, despite using a “list filter” system is Street Fighter 4. Even playing the latest AE2012 version on the PC – with a relatively small community – it still tends to give better results than any other fighting game. The normal host & quick match systems on it are  just as bad as other FGs, but the reason it works so much better is due to the “Fight Request” feature.This is essentially a special kind of hosting, with some major differences to the way hosting works in the above “list filter” examples. Firstly, you can continue to play computer opponents while you wait for a match. This is not only a immediate massive improvement for the player to allow them to play the game rather than just sit and wait, it also makes the whole system work better because there is now a good incentive to host, which creates more matches in the host pool. Secondly the fight requesting host player can decide if they want to allow close-skill opponents only, or any skill matches, or only higher skilled players, and they can also decide somewhat on what connections they want to allow by selecting  local region only or worldwide matchmaking. Again, more incentives to host due to more control – and importantly, suddenly, if you care about skill matchmaking, there is now the possibility of players NOT being matched who aren’t close in skill. No matter how they go about it, searchers will not get a result of a fight request host who’s asked for close-skill matches unless they are close in skill, and likewise, the fight request player can go on playing the CPU for any amount of time, with no-one ever being matched with them unless someone with close skill searches for them. This is a far better situation especially for a new player, or a player striving to improve, regardless of the size of the community and the other problems with matchmaking in FGs listed above. Of course, it still isn’t the ideal situation – the one I’ve been writing about for years now, where you’d combine true “pool” matchmaking with some kind of fight request features as well, and gaining the best of both systems!

Go home and be a family man!

Oh shut it Guile, still no gender-based matchmaking for you!

  5 Responses to “XBox 360 Fighting Games – Do they have effective skill matchmaking?”

  1. Hey Rik! I just read your article on matchmaking in fighting games, very good i must say. One thing i’d like to add though, although you touched on lag i think its a much bigger issue.

    In HDR/ST lag is obviously a big deal due to the moves being so quick, but simply due to how the footsies work in that game (for the most part) where is more about correctly spacing the right priority move i actually think its less of a big deal. On top of that you can double piano all reversals giving you a potential of 12 button presses (counting negative edge) and very rarely are normals punishable on block and the ones that are have huge amounts of lenience.

    But if you look at SF4 a lot of combos have multiple one frame links and a lot of moves are punishable by 3frame normals. Even though you can p-link some normals it still gives you only 1 frame extra for punish. Like you said in your article in an optimal world there would be varying skill levels with the same amount of lag but the one frame difference between 10 to 20 miles pretty much makes the game unplayable.

    I think simply random match making doesn’t work for fighting games, because if its random its unplayable for the most part i mean look at SFxT and UMvC3 no one in their right mind would play ranked over lobbies because it takes an hour to find someone with a good connection. The next step IMO for fighting games online is making sure it ALWAYS has ping and not the vague green/yellow/red bar shit they have at the moment!

    Skullgirls actually did something really interesting with this by giving you really lax windows for inputs. I’ll give some examples using SF4 (just to make it easier to understand); say you’re Balrog and you block Cody’s criminal upper which is -3, well the only way to punish is with your 3 frame stand jab. Now in Skullgirls you could input the 3frame jab a few frames before you actually come out of block stun and the same logic is applicable to combos where you can input the next move before the first one finishes. Its a really interesting idea and basically makes combos really easy, improves netplay and doesn’t mess up frame data OR inputs.

    Sorry i got a pretty side tracked from the actual point of the article, but that’s just my two cents!

    • Thanks very much for reading Jish. I appreciate the comments! I completely agree with you too re: lag. Its crazy bad in SF2 like you say. They really need to design a FG around lag a bit more if they want it to work better online – it sounds like Skullgirls is a step in the right direction there.

      You’re right this point was well worth expanding, and you can talk with a lot more authority about the technicalities of this than I can since you play so many more FGs 🙂 Very good points above! FGs need to be effectively matched on lag/connection as well as skill. Right now we really get neither, for the most part, as you say.

  2. as i think i said in that thread, i’m fairly certain that the skill option in sf4 fight request doesn’t actually kick in while you’re playing a computer and it’s holding a lobby open — I’m pretty sure it’s a regular lobby of the kind you can open yourself. but it seems to also attempt to join other lobbies in between computer opponents, and I’m guessing this is where it will use that option. I may be wrong, but I’m pretty sure it’s not using a special filtered hosted lobby, just automatically using a normal lobby and normal searching.

    • I’m not sure I completely follow you zem.

      If you put Fight Request on and put it on “Same Skill” “Same Region”, it is a regular lobby except now, the only way searchers can find your match and challenge you is if they meet those criteria. I am sure this is how it works. Playing at odd times of the day in the UK on the PC version, I can sometimes complete the game without seeing a single challenger if I’ve narrowed it down that much. Other times I will get the same challenger over and over as that’s the only searcher who meets my host criteria.

  3. I feel this sort of argument is too often used in software development, especially in game development: “We made it, it’s there, so stop complaining”.

    I couldn’t argue with Sirlin that there is match making built into the xbox platform. Having never used their API’s, I would presume that something like that would exist. However, does this make it effective and good? Definitely not!

    One of my favorite games was, and still could have been, Streetfighter II HDR on the XBox. Unfortunately, I stopped playing it quite a long time ago because the matchmaking was just so ineffective. Not only was I wasting my own time, but also that of anyone I was matched up with. So the argument is that the matchmaking is ineffective due to the player pool size, but I’m a classic example of someone who’s chosen to remove themselves from the pool because the matchmaking was such a large turd shaped object floating about in that pool. So chicken or egg?

    I don’t think that pool size should ever be an excuse for bad matchmaking as Remy points out in this article. There are lots of easy to implement ways around that.

    Just my 2c

Leave a Reply