[DevBlog] Trust Us, We're QA

[Mil]

http://devblog.runescape.com/view_post.ws?post_id=25

 

The forums have been bustling with questions recently regarding the process of QA with projects, and why some projects have bugs in them. Well, hopefully, this Developer Blog will explain the process of QA a little, so that you can better understand how content takes the step from a design proposal to being enjoyed by hundreds of thousands of players worldwide.

 

 

 

The step into QA

 

 

It's close to midnight...

 

 

 

When a piece of content comes into QA, it typically takes no more than few hours to find errors, glitches, bugs and concerns. Worry not, though, as this is expected and its our job to discover these things! From this moment to the content being released, we work through specific test plans on testing each element of the content, known trouble areas and general exploratory testing.

 

 

 

Any issues found are then sent to the designated developer for fixing, be it anyone from a graphics artist to a content developer to web content. An example of the scope of issues found in a project: an upcoming quest currently stands at 123 bugs, has been in QA for one week and is being tested thoroughly to get it released in time to be enjoyed by everyone.

 

 

 

More than breaking things

 

 

...and something evil's lurking in the dark

 

 

 

While a large portion of the task at hand is to break and destroy content sent our way as much as possible, which can be extremely enjoyable and equally frustrating at times, we are also very opinionated and believe we understand the players.

 

 

 

When a piece of content comes into QA, it is inevitable that opinions on how to improve design decisions or gameplay mechanics will fly about, trying to make the content more appealing to players. This can be anything from making a weapon look more deadly and suited to the style of RuneScape, to simplifying or increasing the difficulty of puzzles to keep the players engaged (but not bored or frustrated), and even completely redesigning core gameplay mechanics.

 

 

 

These feedback decisions are then relayed to the developers, and with the projects being their own, the decision rests on their shoulders. If the QA team feel particularly strongly about a decision, the concern can be raised to the leader of the particular department, who will weigh up the feedback given, the developers feelings, and the most important variable of all: time.

 

 

 

Priority

 

 

Squark, indeed

 

 

 

Time is extremely important when working on any MMO, and even more so when that MMO receives updates on an almost weekly basis (as opposed to the more typical three- month cycle). Its rather remarkable, actually.

 

 

 

Unfortunately, given the strict schedule for release the game has, priority has to be put in place for what can be done in the time allocated. Being given two weeks to fully QA a quest, for example, means that the QA team working on it have to determine where to allocate time, as in those two weeks, the quest has to be tested twice - once in the work in progress (WIP) test environment and once in the release candidate (RC) test environment.

 

 

 

Further on from this, to maintain such a quick turnaround on releases, the QA team typically has anything from four to ten projects on at once, split between the team (which also have to be balanced alongside any fixes for previous projects or improvements that could not have been implemented in time), as well as fixes for bugs found by players and relayed to us via the Bug Reporting Forums or Bug Report feature on the homepage (we read and investigate every report!).

 

 

 

Why is content broken?

 

 

 

There are typically three core reasons why a piece of content contains bugs at release:

 

 

 

Time

 

This applies more to game features than outright bugs. We know of a feature that might cause issues, but, unfortunately, the time it would require to try out and test alternatives may be longer than is available for the project. For issues such as this, we start to consider changes straight after the content is released and wait to see what feedback the players give, so we know the best direction to take. A good example of this is Mobilising Armies, which weve recently tweaked to iron out some issues players have had with it since launch (e.g. the number of briefing rooms).

 

 

 

Test Environments

 

As detailed above, and in even further detail by Mod Nexus in his Upload Manager Developer Blog, the content has to be tested in both WIP and RC. WIP is the core environment that all developers use to create content for RuneScape, be it for an update scheduled for the next week, or one for next year.

 

 

 

Even a small change in WIP can have a large impact on the game. It is not uncommon for one change to affect just a single item or feature from a selection of millions - something that is nigh on impossible to locate even if given a year to find. Not only this, but when the piece of content moves from WIP to RC, its like moving content from a copy of RuneScape that is six months in the future to current-day RuneScape. The differences can easily break content, requiring us to quickly discover what, where and how this has been caused - as this move into RC means the content will go up as the next update, the clock is ticking.

 

 

 

Bugs Missed

 

Given the scope and scale of the game, along with the constant progression of updates, bugs are occasionally missed. Only a small percentage of bugs that actually fall into this category?

 

 

 

When you consider that, for example, there are 713 head items, 211 cloaks, two graphical modes and 12 graphical settings, that createss 3,610,632 possible combinations. And thats just regarding helm and cloak graphical issues; the concerns are far greater when considering there are currently 12,988 objects, 11,037 NPCs, 47,451 item locations, 2,531 interface sprites...the list goes on.

 

 

 

Rest assured that the QA team work tirelessly on getting updates in the best shape possible for players, each and every week. Personally, being a large fan of MMO games (and typically being subscribed to many at a single time), you are getting every second worth of your time and membership investment into RuneScape.

 

 

 

Hopefully, this will answer some questions and raise more. Feel free to ask any questions you have on the forums. Happy adventuring!

 

http://devblog.runescape.com/view_post.ws?post_id=25

[Omali]

When you consider that, for example, there are 713 head items, 211 cloaks, two graphical modes and 12 graphical settings, that createss 3,610,632 possible combinations. And thats just regarding helm and cloak graphical issues; the concerns are far greater when considering there are currently 12,988 objects, 11,037 NPCs, 47,451 item locations, 2,531 interface sprites...the list goes on.

 

 

 

Thanks.

 

 

 

So, essentially, to those who cry that Jagex "doesn't do any QA on their updates", you get a resounding .

 

 

 

I'm sure Jagex appreciates being talked down to by the kind of children who can't figure out where the "any" key is located on their keyboards, or why Counter Strike crashed when they pressed F10 (The guy in chat said press it for good hax).

[3hitm4g3u]

When you consider that, for example, there are 713 head items, 211 cloaks, two graphical modes and 12 graphical settings, that createss 3,610,632 possible combinations. And thats just regarding helm and cloak graphical issues; the concerns are far greater when considering there are currently 12,988 objects, 11,037 NPCs, 47,451 item locations, 2,531 interface sprites...the list goes on.

 

 

 

Thanks.

 

 

 

So, essentially, to those who cry that Jagex "doesn't do any QA on their updates", you get a resounding .

 

 

 

I'm sure Jagex appreciates being talked down to by the kind of children who can't figure out where the "any" key is located on their keyboards, or why Counter Strike crashed when they pressed F10 (The guy in chat said press it for good hax).

 

 

 

[Grasle]

I doubt we'll see any less claims that QA does absolutely nothing. Whiners will be whiners.

[Omali]

 

 

 

 

 

 

I doubt we'll see any less claims that QA does absolutely nothing. Whiners will be whiners.

 

 

 

Indeed they will. Luckily, that's why mutes/bans exist.

[Golvellius]

Well, there you have it.

 

An extra helping of Jagex arrogance.

 

Must be a British thing.

 

 

 

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

I say this because every one since Pest Control has had one, TRUST ME.

[Maximusa]

In other words those in charge use as us free testers and have no problem admitting it. Ha.

 

 

 

Do they care that people lose items, money, time, fun due to lazyness/trying save cash.

 

 

 

See those who are smarter just avoid new content until everyone else has tested it as the company can not cope with game braking bugs before release.

 

 

 

This isnt about bashing the QA I am sure they do their best. Its just what options they have available to them isnt good enough imo. Or us.

[Runeman231]

I would've liked it much more if they had atleast attempted to justify some of their recent bugs, such as the unbelievably obvious "invalid teleport" bug. You can't even possibly blame that one on having thousands of players using the content live, all QA had to do was teleport to their player-owned house and they would've caught the bug..

[Omali]

Must be a British thing.

 

 

 

British? Arrogance? I don't see the connection.

 

 

 

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

 

 

In other words those in charge use as us free testers and have no problem admitting it. Ha.

 

 

 

That's all MMO developers, only some of them use Public Test Servers to test those updates before they're released. Doesn't get rid of all bugs though.

 

 

 

See those who are smarter just avoid new content until everyone else has tested it as the company can not cope with game braking bugs before release.

 

 

 

People who are smarter don't test anything until some time after release because that's the deal with ALL software. Anyone who has a lot of experience with MMOs/Software will tell you a few things: You don't get into an MMO until a few months after launch, you don't buy a new operating system until about six months after launch, you don't fly into new content on an MMO with all of your precious items on you, and there's no way for a company to accurately stress test how the server will react under live stress.

 

 

 

This isnt about bashing the QA I am sure they do their best. Its just what options they have available to them isnt good enough imo. Or us.

 

 

 

You have unrealistic expectations for an MMO. These bug tests are easier on any other genre because there's less load to work with, and a development team can easily simulate the load on, say, a 32 man server.

 

 

 

I would've liked it much more if they had atleast attempted to justify some of their recent bugs, such as the unbelievably obvious "invalid teleport" bug. You can't even possibly blame that one on having thousands of players using the content live, all QA had to do was teleport to their player-owned house and they would've caught the bug..

 

 

 

The only explanation is: Stuff happens. Take a look at several times when random updates caused the floor to warp in various areas of the game. As for server load affecting teleport: The servers were unable to pull the player's POH off of the external server where player owned houses are kept (POH's are stored on separate servers, that is fact). Due to the teleport location not being spawned, the server responded with an "invalid teleport"

 

 

 

I'm not saying that's what happened, but it is a reasonable explanation.

[Zepheras]

Well, there you have it.

 

An extra helping of Jagex arrogance.

 

Must be a British thing.

 

 

 

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

I say this because every one since Pest Control has had one, TRUST ME.

 

 

 

 

 

I read the Trust Me part of the title as a reference to this:

 

 

Rest assured that the QA team work tirelessly on getting updates in the best shape possible for players, each and every week.

 

 

 

 

 

They didn't say anything anything about trusting them in making every single update bug free did they?

[Omali]

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

I say this because every one since Pest Control has had one, TRUST ME.

 

 

 

I remember a post on the RSOF talking about how difficult it was to program waiting rooms like the ones you see in Pest Control (Where a lot of people are in and only a few will be taken into the game), as well as having a mini-game with more than 2 teams, and it makes me wonder why they keep making mini-games with those kinds of lobbies.

[Randox]

The game must be incredibly hard to test and trouble shoot. Everything is connected. For the teleport bug, it is quite possible the onle reason it existed was related to another activity elsewhere in the game. Some completly unrelated event, maybe a minigame, caused the problem. All this would have been inactive areas of the game during the test, but once active was causeing problems elsewhere.

 

 

 

Maybe the server load itself was the problem.

 

 

 

Would be nice if they said, but chances are it was not an obvious problem.

[Esper_Jones]

Two weeks of QA for a quest that took 3 months to develop? That actually explains a lot.

 

 

 

The two weeks includes any coding that developers need to adjust, gameplay adjustments, bug fixes, et cetera are all lumped into the same 2 weeks. No wonder there's so many bugs. QA should take at least twice the time of coding, if it's being done right.

[RU_Insane]

No one caught the "Thriller" reference?

 

Anyhow I'm glad they're coming out wiht more DevBlogs about these areas of concern. They really are doing more to reach out to the community, in a way.

[Suiku]

Some days ago I said they should make a development diary, I'm glad they listened me :thumbsup:

 

 

 

*totally oblivious at the fact he wasn't the only one saying it*

[Den]

Nice read. :

[jettrider]

We don't care about graphical glitches, we care about game-breaking glitches.

 

 

 

For example when Mobilising Armies was released only one of the four scenarios worked.

 

 

 

Either they forgot to try playing 75% of the game or their test environment isn't a good model of RuneScape. Either one is a big problem.

[Lunar_Drifter]

Interesting pictures. Has any mods transformed like that in actual servers? That'd be interesting to see.. a bunch of jmod seagulls flying around or something.

[Racheya]

Well, there you have it.

 

An extra helping of Jagex arrogance.

 

Must be a British thing.

 

 

 

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

I say this because every one since Pest Control has had one, TRUST ME.

 

Arrogance - sure :lol:

 

 

 

And I would trust them. Know why? Because they've been at their job for months/years, they've gone to university, gotten degrees and possibly worked at game companies all their lives. If you're saying that a whiny kid who just plays games knows more than a team of highly skilled QA people then there's something wrong. It's like telling a doctor you could do his job better because you watched it on TV.

 

We don't care about graphical glitches, we care about game-breaking glitches.

 

 

 

For example when Mobilising Armies was released only one of the four scenarios worked.

 

 

 

Either they forgot to try playing 75% of the game or their test environment isn't a good model of RuneScape. Either one is a big problem.

 

It's not a good environment, but that's because like they said it's the Runescape of the future, there may be updates in that, that make the code they use work - but since those updates aren't in our version yet it breaks. It explains how they miss seemingly obvious things, it's because the problem wasn't there for them. They can't simulate the stress either of hundreds of thousands players rushing to try the new content at the same time.

[Unknown_Warrior]

How about they trust me when I say, "I guarantee you that the next minigame you release will have at least one game breaking bug in the waiting room/entry system".

 

I say this because every one since Pest Control has had one, TRUST ME.

 

 

 

I remember a post on the RSOF talking about how difficult it was to program waiting rooms like the ones you see in Pest Control (Where a lot of people are in and only a few will be taken into the game), as well as having a mini-game with more than 2 teams, and it makes me wonder why they keep making mini-games with those kinds of lobbies.

 

 

 

Because team-related minigames is what the community likes best.

 

 

 

Two weeks of QA for a quest that took 3 months to develop? That actually explains a lot.

 

 

 

The two weeks includes any coding that developers need to adjust, gameplay adjustments, bug fixes, et cetera are all lumped into the same 2 weeks. No wonder there's so many bugs. QA should take at least twice the time of coding, if it's being done right.

 

 

 

The community simply won't stand for it. Do you remember the outcries when Jagex stated they'd have to postpone updates? Ye.

 

 

 

Inevitably, more time spent looking for bugs means less time to work on new updates. I'd prefer the 4 hours of bugs after an update if it means we get more updates that way.

 

 

 

I would've liked it much more if they had atleast attempted to justify some of their recent bugs, such as the unbelievably obvious "invalid teleport" bug. You can't even possibly blame that one on having thousands of players using the content live, all QA had to do was teleport to their player-owned house and they would've caught the bug..

 

 

 

And what are you basing yourself on? What do you know about it? Perhaps the bug was caused by the server not able to discern between a thousand houses, which promptly caused a loophole and crash as a result? If it was that kind of bug, the QA couldn't have found it because they simply aren't with a thousand people.

 

 

 

---------------------------------------------------------

 

 

 

In either case, I believe that if you decide to try out an update in its first 15 minutes - 1 hour of release, you're submitting yourself as a Beta tester. And if you decide to take your party hat set, 150M cash and full Bandos while doing it, you're responsible for anything that comes with that risk. The only time I believe Jagex is at fault is when it affects something that on a remote location. Say: logging off in an update and getting your Bank deleted. Or if it affects game-wide (like the non-attacking monsters bug). But all other times: whatever you're wearing, where your avatar is standing, whatever you have in your equipment or whatever you're making is entirely your responsibility during this time.

 

 

 

That said, it was a good read. Anyone notice the QA turning into crows? :-k

 

J-mod magic or a future item?

[Lunar_Drifter]

Yea, they can transform into anything that is an NPC. I mentioned that a few posts up. :D

[Awkward Talking Fish]

Pretty interesting read :D

[Gribbzldinho]

Two weeks of QA for a quest that took 3 months to develop? That actually explains a lot.

 

 

 

The two weeks includes any coding that developers need to adjust, gameplay adjustments, bug fixes, et cetera are all lumped into the same 2 weeks. No wonder there's so many bugs. QA should take at least twice the time of coding, if it's being done right.

 

 

 

For a brand new game or a huge, game-changing update? quite possibly.

 

 

 

For a small'ish update such as a quest? No.

[Lielac]

Interesting!

 

 

 

(I thought that was a thriller reference...)

 

 

 

These guys need some cookies and 'thank you for cutting down on the bugs's to outweigh the twits saying they either don't exist or are incompetent. Or both (how does that work...)

[Kietaro1]

What many people on Jagex's defense seem to be missing is not that those that are complaining are not complaining that bugs are getting through. More so they do not blame development, QA, or any number of things that cause any said update to be less than par.

 

 

 

Instead many of the complainers (myself included) find that too often this year we have received nothing for an update. Towards the start of the year, I felt as though these breaks were warrented as we were promised quality in updates when the Behind the Scenes was removed. It was done to remove time constraints and allow Jagex more time to really develop good updates. I agreed with that and have upheld my trust in Jagex throughout the majority of the year. Unfortunately as the year has progressed and more weeks have been skipped, that trust has begun to falter and iIfind myself uninterested in the updates that have been released.

 

 

 

Do not misunderstand me and think that I do not appreciate some of the updates that we have been given. The route finding update for instance truly made the game more enjoyable and I am sure that everyone enjoys the minstrels scattered around runescape easing the load of passer bys that stop to enjoy some of the best music in reunescape. Over all however, this year has been riddled with updates that have not drastically changed much of anything. The animations and new look of some of the weapons and scenery have been nice but are soon forgotten as Runescape fades back into the grind it was before those updates and that it has continued to be after them. In other words this years updates as a whole have left many players who have done the majority of what Runescape has to offer with nothing new to try, nothing new to explore, and nothing enjoyable to complete. There have been gems as shown above but my opinion stands as this year offering very little in the area of 'new' content.

 

 

 

In pyre need, for instance offered a new way to train firemaking but more importantly, a new way to firemake. It really changed a portion of the game drastically (enough for me to actually level firemaking). Smoking Kills revolutionized a major part of one of the most enjoyable (for most players) skills as well as offered new content for several other commonplace skills. Wile updates of that caliber have been released this year they have been few and far between which has result in many players playing very little. Sure we keep our membership up but I am sure that the developers want their content to be enjoyed and no avoided on the week of release in fear of losing hard earned portions of our character. By not playing, even with membership, we are not enjoying the content all the same.

 

 

 

I was lucky enough to be able to get into my house on the day the manageries were released, build my room, and go to work. On my way into work however, I could not get over the amount of bugs a experienced and how little that update will affect my day to day scaping. I built the room, collected the pets that are easily to get, put them in my house and was done. That is all that offered me. Sure the summoning familars will be more useful for combat but again it will not drastically change the way we play. Slightly more useful (the way it should have been to begin with) familars is a fix but even the familars we had could do the same. I will say it is useful but not something that makes this game more enjoyable on a day to day basis. That seems to be the majority of updates and the real reason we 'complainers' find ourself in a position to want to say something

 

 

 

/wall of text.