One of the most basic ways to let people discuss and leave feedback is through a forum or message board with different topic categories. While it is a standard way to do this, it has many drawbacks. These flaws are the bumping factor along with the drama factor,
as well as the lack of quality ratings and also the indirect nature of reporting game events.
Drama threads are susceptible to drama bumps, where turmoil often make people voice their opinion more, which adds fuel to any flame. This in turn affects the entire mood of the forum. Since the nature of a badly written post is more likely to get both creative and destructive input, they flourish and influence the forum to a larget extent than good posts. The nature of a well written post will either become saturated with complements, making each following complement more likely to be reduced to a copy bump than a genuine contribution. This discourages contribution and puts an expiration date on such posts.
When it comes to things like reddit, the bumps and contributions are separate, meaning that it is easier to bury drama posts in “dislikes”, and to bump a good thread without saturating it. If a game developer wants to control and moderate the discussions of their game heavily on their own sites, people will still take the “superior” discussion elsewhere (on reddit). If a good discussion climate is preferred, game forums must step up and improve their functionality. One very good example of this is http://ideascale.com/
Next up is other external feedback systems, like surveys
What are surveys good for? They provide statistics and easier ways to compile opinions into structured graphs that can aid future development. But there are a few rules that game companies trip on when they are collecting feedback for their games. The most recent examples are the Final Fantasy XIV: A Realm Reborn, The Elder Scrolls Online and Battlefield 4 balance surveys. I will not go into detail on each and every survey, but instead describe some of the faults in how questions and answer options are presented.
1. Discarding whole submissions due to certain answers, very bad (FFXIV)
Apparently some submissions were discarded without warning, due to not matching certain criteria in the answers. It might have been (won’t buy the game yet), (haven’t played certain competitor) or whatever, but the people who tried to contribute to the success of their game were not informed when their effort as a whole, up to 30 min, would be discarded and unread. This is poor survey handling, that lost data was important data.
2. The answers must to a large degree cover most opinions about the question.
I’ve seen many yes/no voting polls and “pick only one” questions about the Battlefield series where multiple changes or features are described in the poll and the complexity of them are not reflected in the answers. I have heard that some of them are not really used as influence, but the more recent ones with balance changes are. Let’s take a look at this.
Here are some guidelines for such future polls, explaining how they improve data quality:
– If you ask “which of these features do you like”, the poll must allow picking serveral of them to get the best and most accurate results of their popularity. Asking “which do you like the most” is really a bad choice, because everyone’s second favorite might actually get considered as bad as their least favorite one. Putting in some effort will pay off here.
– If you ask “Do you want the changes described above”, with a yes/no answer, it is better to actually ask “Which of these changes do you support” with multiple checkbox choices.
– If you describe both nerfs and buffs to a feature, you can’t just ask about one of them.
In that case you must provide answers that say something like Buff it / Leave it / Nerf it.
3. Mixing positive and negative question types with same/different scales
If you ask a series of questions like The Elder Scrolls Online did, people will be very confused. Overall they had good scales and questions, but the answer options were poortly named. Here are some examples to consider when using the Likert scales:
– If you have a 5 position scale, many people will get right in the middle of it, or on 2 or 3. They are less likely to stand at the extreme edges of it and you’ll get a boring average spread. It might be better to have 6 or 7 scale positions rather than 4 or 5, because there is no safe middle and the mostly disagree/agree options will be forced to spread out more. When naming them, the best system is really the “disagree-agree” scale, because you can mix positive and negative questions without confusion. Imagine these questions:
“Do you think your character is unique and feels good”
“Do you think that combat lag is bad?”
If you had earlier answer scales with “disagree-agree” and you were instructed that going to the right on the scale meant agreeing more with that statement, you’ll have confusion when these mixed questions suddely have the answer options “Very bad-Very good”. If you feel that the combat lag is “bad” you would go to the left of the scale due to wording, but if you’d agree more with that statement, you’d go right… And if you use the “very bad-very good) scale in such a way… an answer scale of mostly bad (2) must be converted to match other negative questions where the answer option for (2) was “mostly disagree”.
Therefore, it is best to stick with the (strongly disagree – strongly agree) Likert scale, where “don’t know” means that you leave no answer on that particular question. The most important questions can be ranked with the “reply rate”, which highlights questions where users were convinced of their opinion. Giving it 6 scale positions means you cut out the safe answer, so that you are forced to strongly disagree, mostly disagree, slightly disagree, slightly agree, mostly agree or strongly agree. This works with both our positive and negative questions without confusion and you can categorize positive questions in one group and negative in another when compiling the answers. That way you can get the average positive ratings of your game in one batch, and the average negative ratings in another. From there you can identify which questions added most weight to the spread and you can categorize them depending on impact and reply percentages/importance.
Putting positive and negative questions together in the actual survey prevents the likelyness to put “overall 4s” across the entire page,which is a good thing. But the important lesson here is that you can analyze negative and positive questions separatly. The other lesson is to not let the answer options confuse people. To prevent this confusion, use the same scale, but keep the statements varied instead. This is better than asking the same type of questions, while using loads of different scales & conversions.
Which game has had the best feedback option then?
I’d say that an ingame feedback / report system is a very good way to handle feedback. The Elder Scrolls Online had the ability to leave ingame feedback by typing /feedback or /bug in the game chat. This sent the player to a screen where they could categorize the feedback or bug into different game areas that sent the report to appropriate devs to be handled by the right people. Imagine the following additions to such an ingame system:
– At the moment the player says /feedback or /bug, a screenshot is taken of the game state. They player is then allowed to crop and highlight certain areas of this picture.
– At the same moment, the computer has also gathered a DxDiag file, your computer setup and tempereatures, as well as a copy of the gamestate inside your RAM memory. The player then has the option to submit a selection of these with their feedback/bug.
– These feedback and bug reports can be directly received by appropriate devs, and they can see the cropped screenshots and watch and compare certain memory states while they are sitting with their game or dev client open, by using a feedback overlay toggle.
– Another option is to allow 15 second videos to be recorded and sent with this system. Keeping bugs and glitches away from places like Youtube and open forums may be a good idea if you do not want unneccessarily viral & unintended events seen by the public.
Combining this ingame system with survey analytics is possibly the best system!
Forums are best for discussing overall design choices and not for measuring specifics.
Next time I’ll probably spotlight a specific patch, game or a nice feature I’ve discovered.