Should Jagex start beta testing with players?

[langer]

Every update, we see lots of bugs appearing and almost everytime, Jagex needs to fix them on the spot or the week after.

 

Many people ranted that Jagex in-house beta testers don't know how the common player plays and that relates to bug-filled updates and irrational content like the way too expensive rewards of dungeoneering upon release.

 

Should Jagex allow players to beta test future updates on the side? Like parallele servers that have no impact on the "real game".

 

Should Jagex choose to do it, do you think they should allow all players to beta-test or a select-few? Should they announce future updates to all players or only the select-few?

 

How would this affect the economy if everyone is aware of future updates or only the beta-testers?

[qeltar]

As a leading voice calling for a beta test program for years now, my position is well-known.

 

Jagex likes to get all hot and bothered about this issue, insisting that it is something they can't do. In that, they are basically implying that they are different from every other software company in existence, which is a bit hard to swallow. The reality is that they just dont wan't to, because of their control and secrecy issues. I mean, this is a company that can't even tell people whom they are charging $125 for an event, where the event will be held.

 

All they need to do is get themselves a small number of experienced, adult players and make them sign binding NDAs -- just like companies in all fields do every day. (Have you ever noticed that when Intel puts out a new chip, there are full reviews of it the same day? The reviewers don't wait in line at Best Buy and then pull all-nighters). There would be a few more details to work out, but they definitely could do just that.

[Ambler]

Living where I do- I know for a fact that a few people from my school have done some beta-testing, I think it was for the blood pact quest- as one of them was able to give me a very detailed description of what it was gonna be like- that or he looked in jagex cache >.>

[langer]

As a leading voice calling for a beta test program for years now, my position is well-known.

 

Jagex likes to get all hot and bothered about this issue, insisting that it is something they can't do. In that, they are basically implying that they are different from every other software company in existence, which is a bit hard to swallow. The reality is that they just dont wan't to, because of their control and secrecy issues. I mean, this is a company that can't even tell people whom they are charging $125 for an event, where the event will be held.

 

All they need to do is get themselves a small number of experienced, adult players and make them sign binding NDAs -- just like companies in all fields do every day. (Have you ever noticed that when Intel puts out a new chip, there are full reviews of it the same day? The reviewers don't wait in line at Best Buy and then pull all-nighters). There would be a few more details to work out, but they definitely could do just that.

 

 

I have to admit that I started this debate here after lurking a topic on TS/Runescoop forum 30 minutes ago. I'm glad you came here to reply because this subject is full of grey areas. I understand the need to make beta-testers adults-only and make them sign non-disclosure agreements, but how would you calculate the risk of leaks and their effect on the market? How would you reward these people (free membership? ).

 

Would the pros cover the risks? I know the gameplay would smoothen up a bit adn releases would probably be fullproof upon release most of the time, but the idea of select players knowing future updates and having the ability to invest or "sell the info" to other players bother me somewhat.

 

I consider myself to be an ethical person, but I cannot garantee that the select few would be...

[Rien]

I'd love for Jagex to implement beta-testing, even if I'm left out of it. All the secrecy is quite irritating to begin with, and if actual players test new content before it's released, then I'm sure far fewer bugs will slip through. Beta-testing would benefit both Jagex and the players, so I've no idea why they've refused to allow it for so long. :mellow:

 

 

I understand the need to make beta-testers adults-only and make them sign non-disclosure agreements, but how would you calculate the risk of leaks and their effect on the market? How would you reward these people (free membership? ).

There are literally thousands of players who'd be willing to beta-test for free, heh. Also, violating a NDA is a criminal offense; so long as Jagex explains the repercussions of leaking information, I doubt very many players would be willing to take the risk. I'm sure Jagex would be capable of keeping a tight lid on things.

[Guest jrhairychest]

No they shouldn't. Its way too big a risk to do so, and I can't blame them for it. No matter how pure the motives of players are I don't think there would ever be a day that Jagex would trust them with such information. With so much at stake a company would be insane to do it.

[langer]

No they shouldn't. Its way too big a risk to do so, and I can't blame them for it. No matter how pure the motives of players are I don't think there would ever be a day that Jagex would trust them with such information. With so much at stake a company would be insane to do it.

 

What if they'd release the information to everyone while it is being tested, so that the entire player base would know at the same time how the game will be affected via future updates?

[Sublexation]

I completely agree. Especially with a multitude of older players to choose from, they can avoid most of the bugs/balancing issues. And I bet that they could keep it secret someway or another. I have no experience in this field so I don't know logistics. I don't know how feasible this would be but it would definitely be beneficial for major updates such as skills, mini games, major graphical improvements, and the like. There is only so much in house testing you can do,

[Guest jrhairychest]

What if they'd release the information to everyone while it is being tested, so that the entire player base would know at the same time how the game will be affected via future updates?

 

Jagex like to keep updates to themselves to keep the game fresh. It'll spoil any updates/skills/quests for everyone.

 

 

I completely agree. Especially with a multitude of older players to choose from, they can avoid most of the bugs/balancing issues. And I bet that they could keep it secret someway or another. I have no experience in this field so I don't know logistics. I don't know how feasible this would be but it would definitely be beneficial for major updates such as skills, mini games, major graphical improvements, and the like. There is only so much in house testing you can do,

 

There's no way players would keep it under wraps. Be realistic. You'd never trust ANY of them as far as you can throw them. Most companies don't trust their own employees never mind their customers.

 

How would all this be a benefit? The best feedback is from the overall general population, not a few players who all of a sudden think they're bonafide games testers. They already have all the games testers they need via the population of the game. Sure, not everything goes right at times but thats why we have the forums for bug submission and feedback. A handful of players won't give you this or feedback from the general population as a whole. Some bugs or flaws don't become apparent for months.

 

Considering the size of the game, Jagex do a pretty good job at getting most things right.

[howbadisbad]

What if they'd release the information to everyone while it is being tested, so that the entire player base would know at the same time how the game will be affected via future updates?

 

Jagex like to keep updates to themselves to keep the game fresh. It'll spoil any updates/skills/quests for everyone.

 

Thats what NDAs are for...

I completely agree. Especially with a multitude of older players to choose from, they can avoid most of the bugs/balancing issues. And I bet that they could keep it secret someway or another. I have no experience in this field so I don't know logistics. I don't know how feasible this would be but it would definitely be beneficial for major updates such as skills, mini games, major graphical improvements, and the like. There is only so much in house testing you can do,

 

There's no way players would keep it under wraps. Be realistic. You'd never trust ANY of them as far as you can throw them. Most companies don't trust their own employees never mind their customers.

 

How would all this be a benefit? The best feedback is from the overall general population, not a few players who all of a sudden think they're bonafide games testers. They already have all the games testers they need via the population of the game. Sure, not everything goes right at times but thats why we have the forums for bug submission and feedback. A handful of players won't give you this or feedback from the general population as a whole. Some bugs or flaws don't become apparent for months. You are just flat out wrong here. As with combat changes most people are melee based so when combat changes that make melee overpowering come out (turmoil) with range and mage getting nothing, then that is not a good way to get feedback. The best way would be to handpick players who are experienced with such things (Like famous pkers should beta test pking updates) so you get the best information from all aspects.

 

Considering the size of the game, Jagex do a pretty good job at getting most things right. I would have to disagree here... especially about recent updates.

 

Jagex should have beta testers so we don't get more updates like "Dungeoneering"

[langer]

@ Howbadisbad

 

How would react a pker testing a future update that relies mainly with high prayer? Investing on D bones on the side? Do you trust every experienced players?

Beta testers would have only saved us a week of bad calibrated dungeoneering rewards. The skill in itself is probably the most played out upon release than any of the latest skills.

 

Summon got the "too expensive" treatment.

Hunters had to wait a couple months before discovering money-making through chins.

Construction got the "too expensive" treatment and hard to get supplies upon release.

Farming was voted the worst skill for a good 2 years before people realised how profitable it was.

Slayer... last time we got a skill that was played as much upon release.

[bedman]

Even if those players are honest, there can be misconceptions. I mean, let's say you are a beta-tester, and you are also thinking about buying 99 prayer, but you were waiting for dbones to drop. A few days later, you get to test a new update, that introduces new high lvl prayers. Again a few days later, dbones did drop like you expected, and you buy 40k of them, like you planned BEFORE you tested that prayer update. How is Jagex going to see the difference between you, and some beta-tester who is actually just thinking about reselling these bones once the update comes.

[schmuall]

Even if those players are honest, there can be misconceptions. I mean, let's say you are a beta-tester, and you are also thinking about buying 99 prayer, but you were waiting for dbones to drop. A few days later, you get to test a new update, that introduces new high lvl prayers. Again a few days later, dbones did drop like you expected, and you buy 40k of them, like you planned BEFORE you tested that prayer update. How is Jagex going to see the difference between you, and some beta-tester who is actually just thinking about reselling these bones once the update comes.

 

Maybe if you tell Jagex before you get to see the update?

[qeltar]

Nothing really novel has been raised so far. Every one of these objections/concerns comes up each time the matter of beta testing is discussed.

 

I'll have a more thorough post on this later on. But what it all really boils down to is this: not one of these issues is specific to Jagex. All software companies, and many companies of other types, have to deal with them. So the only way to justify Jagex not having beta testing is if you believe either that Jagex is so special that they simply can't follow the standard practices of the software industry, or Jagex is too incompetent to implement those practices.

 

I don't believe either is true.

 

Also: Jagex's problem is not QA in the sense of finding bugs. In fact, Jagex IMO does an *excellent* job of finding and eliminating obvious bugs. Given the complexity of the software, I consider RuneScape remarkably bug-free.

 

The real issue is feature design, gameplay balance, and having a proper understanding of how their new features will be used, as opposed to how they'd *like* them to be used. This is where Jagex falls on its face with nearly every update.

 

Take Dungeoneering for example. Okay, under some conditions you get a floor with a design layout problem, or an issue with a boss that doesn't appear. Annoying, but rare, and those problems will get fixed. I don't consider that a big issue. And no, a beta testing team would likely not have caught that stuff.

 

What a beta testing team *would* have told Jagex before they released Dungeoneering:

 

1. "Woah! Cool reward but are you serious, I'd have to play 200 floors to get enough XP to buy that!"

 

2. "Are you aware that you can just buy rune essence by the boatload and stand here and get a ton of RC XP"?

 

3. "When people find out you can't resume a solo game, they're going to flip out".

 

4. "Hey, I noticed I get way more XP per hour just playing on F2P. Was that on purpose?"

 

And so forth. Basically, all the stuff that *any* experienced player would have said within 24 hours of the release of the skill, but that for some reason Jagex never considered.

 

This is what is needed.

 

That users never use software the way software engineers engineer software is almost axiomatic in the industry. Jagex seems to be in utter denial about this, or simply not interested in exploring how to turn it to their advantage.

[Guest jrhairychest]

Thats what NDAs are for...

If you believe that you're seriously deluding yourself. Let's be honest about this, most players can't hold their own water if it gives them one-upmanship over other players, especially the 'knowledge is power' variety. All it takes is for rumors to spread and it ruins the suprise elements.

 

You are just flat out wrong here. As with combat changes most people are melee based so when combat changes that make melee overpowering come out (turmoil) with range and mage getting nothing, then that is not a good way to get feedback. The best way would be to handpick players who are experienced with such things (Like famous pkers should beta test pking updates) so you get the best information from all aspects. [/color]

 

If I'm so wrong then ideas like this would already be implemented so re-think your strategy on this issue. There's still the trust elements which stops most of this dead in the water. We already have forum feedback/bug reporting where they get an idea from the whole community, not just one or two who are 'famous'. There's no point on gaining a small section of feedback when you can gain a much broader aspect.

 

Considering the size of the game, Jagex do a pretty good job at getting most things right. I would have to disagree here... especially about recent updates.

 

 

Jagex should have beta testers so we don't get more updates like "Dungeoneering"

 

 

Again I disagree. They get a few things wrong now and again and fix most of the stuff they get wrong, and they'll fix dungeoneering. If it was so bad then you wouldn't be playing RS and neither would anyone else. I would agree with Queltars comments in his last post on the issue.

 

That users never use software the way software engineers engineer software is almost axiomatic in the industry. Jagex seems to be in utter denial about this, or simply not interested in exploring how to turn it to their advantage.

Games testers aren't necessarily software engineers as it makes them too expensive. Employed games testers tend to be paid less than developers and some of it is outsourced, though I expect in Jagex's case they wouldn't outsource due to being very protective about their own product.

 

I also point to the comments made above about the trust issue. I expect they already keep a tight leash on their own employees (my own organisation certainly does!) . While there may be benefits to having players test aspects of the product the risks outweigh the potential benefits.

[schmuall]

Just found this quote. Seems it wraps up this topic nicely

 

"Regarding desire for beta testing, this is something that the QA team supports fully and has voiced their interest in. Fingers crossed that this is something that both the QA team and players can have available in the future. "

 

Source:http://www.runehq.net/RHQInn/index.php?/topic/384563-exclusive-dungeoneering-interview/

[Bob_the_Mailman]

Yes, definitely; since Jagex can't even be bothered to check their coding let alone their games.

 

The real problem is finding suitable candidates, and how such a thing will work. It can't actually be linked character wise to the public Runescape, since if someone finds a glitch in Beta and keeps the items ingame it will be a bit pointless.

[Guest jrhairychest]

Just found this quote. Seems it wraps up this topic nicely

 

"Regarding desire for beta testing, this is something that the QA team supports fully and has voiced their interest in. Fingers crossed that this is something that both the QA team and players can have available in the future. "

 

Source:http://www.runehq.net/RHQInn/index.php?/topic/384563-exclusive-dungeoneering-interview/

 

Its a very nice wishlist quote but you seem to have missed the last comment off by mod Fetski, regarding the above quote from the same source:

The investment of time and resources needed to achieve this would, however, be huge.

 

Quite a large barrier me thinks.

 

We still haven't got past the issue of trust and security. This would represent quite a gamble from Jagex's part in none-disclosure, and theres no way you can legally enforce or even track it effectively enough. I could pick any group of players from any pool I want for beta testing, but there'd never be a day that I would trust them.

[Owned_Nex]

I think beta should be available to everyone.

 

Lets face it:

 

Every update has had some kind of bug associated with it.

There are always gameplay flaws in all updates.

Many updates simply indicate Jagex is losing touch with the playerbase.

 

Updates will happen, and everyone will be able to play; Jagex needs to STOP the whole "surprise!" theme and mature into the "here is the BEST we could do"

[Creaphis]

Trust is absolutely not the issue. If Jagex is avoiding the use of beta-testers because they fear that information about their updates would be leaked, then they chronically paranoid, because:

 

1) Software and engineering companies with FAR more valuable secrets are still able to protect them, because non-disclosure agreements DO work.

2) Trustworthy and insightful testers ARE available in large supply. For starters, Jagex could invite player mods, who have already been deemed trustworthy by Jagex staff, to evaluate the gameplay and design choices of new updates before they're released to the public.

 

If it really was so difficult to find good help, then the software and game industries would collapse in on themselves. Read qeltar's last post again, carefully, if you don't understand this. If you still believe that no trustworthy testers could possibly be found, then you may be suffering from paranoia yourself and should speak to a professional councellor about that.

 

Here's what is the issue, and what always is the issue when it comes to poor development of games and software: COST.

 

Right now, Jagex is releasing new updates weekly or bi-weekly, and is charging its subscribers $6 per month, which currently covers all costs of hosting, development and administration. I'm going to go out on a limb here and guess that not much of that $6 is profit. If Jagex added another stage of testing to its development model - even if the testers were volunteers - this would increase their operating costs. Jagex would have to hire new staff to administer the beta-testing program, Jagex would have to pay for more test servers, and, because the testers would certainly be finding many problems in the new updates, Jagex would have to hire extra development staff to help resolve these issues in time to release these updates on schedule.

 

To cover these extra costs, Jagex would have to raise the subscription price. The only way that the subscription price could remain the same as it is now with a good beta-testing system in place is if updates were developed more slowly and released less often. Now, ask yourself: would you be willing to accept either of these changes? Higher subscription fees, or slower updates?

 

No?

 

Well then enjoy your glitch-ridden, bizarrely-designed Runescape.

[Racheya]

To cover these extra costs, Jagex would have to raise the subscription price. The only way that the subscription price could remain the same as it is now with a good beta-testing system in place is if updates were developed more slowly and released less often. Now, ask yourself: would you be willing to accept either of these changes? Higher subscription fees, or slower updates?

 

No?

 

Well then enjoy your glitch-ridden, bizarrely-designed Runescape.

Thank you, I will enjoy it :)

 

I would hate to have to pay extra for some kiddies to play updates before I do. If you renamed 'beta testing' to 'early update access' people would be screaming from the roofs. I really don't see how being geographically located close to Jagex HQ deserves you the ability to test the game. Not only that, I really don't think NDA's would work. Someone posts it on a fansite website and the damage is done, there's *no* way they could stop it then.

[Mercifull]

People are selfish. How would those involved in beta testing be able to resist buying up all the resources for a future update? Something that means that a completely worthless item will suddenly be in huge demand. e.g. Nasturtium seeds are now ingredients for a potion or familiar that gives really good exp. We know some of the kind of people that play Runescape, would you trust them not to buy out all the seeds and sell for ridiculous amounds after the update?

 

However I do think that game engine updates need testing. There were a lot of issues with Runetek5 for example which would have been spotted by players.

 

Yes to beta testing for game engine updates, no to beta testing for new skills/quests/upgrades to existing skills etc.

[Mercifull]

Jagex has like 400 employees, thats a pretty large beta testing team there. Just get all the customer support ops and people not involved in development or content to have a go themselves.

 

I was reminded of this comic http://theoatmeal.com/comics/design_hell which basicly parodys how when trying to design a website you can end up doing bizarre things on request. I could see how something similar could happen to Runescape with Jagex creating a piece of content and it getting warped due to feedback from selfish players to the point where it has become a poor quality piece of work suited only to the people who gave the feedback in the first place, and not for the greater good at all.

[Beloved]

in short: yes they should

[Ezz]

I agree with Qeltar. There are definitely ways that this could be done. The main problem would of course be the economical implications, but if the update is leaked to general public it's basically like knowing every update a certain time before hand. This would not be so bad, it would not spoil the updates too much, as the updates are already "spoiled" by developer diaries and such. And updates, contrary to popular opinion, are not about surprises, one should not care if an update is spoiled or not. It's all about how good the update is and how it intergrates into the game, not about how fun and surprising it is during the first week of release. Now what do we choose? The element of surprise and satisfaction for a week, or content that intergrates well into the game and is a success from the very start of the release? I wouldn't mind knowing what an update is a bit beforehand and then having the update be a very good addition to the game.

 

Sorry if this post was a bit incoherent, I'm a bit tired right now :unsure: