Should Jagex start beta testing with players?

[@Dan3HitU]

Should Jagex start beta testing with players?

They actually already do this.

[Profins]

As long as they get bug abusers (original ones, not copy cats) on the job, I'll be happy. I mean, come on, who better to figure out new bugs than the people who've done it before? :P

 

More seriously though, NDA or not, there would be leaks. I have no clue how would they manage to control the leaks and such. It would affect economy greatly, in a way that some people would benefit from things way more than others. Even if beta-testers were to sign NDAs, they themselves would be able to gain advantage of their status by stocking up on supplies with no consequences. Jagex doesn't want that. They won't do this.

[Omali]

Thats what NDAs are for...

 

Players don't listen to Non Disclosure Agreements. To prove it, here is a photograph of the World of Warcraft: Cataclysm Alpha. Not my image.

 

[xxshady]

Indeed, the Cataclysm Friends and Family Alpha has already been leaked to the general public. If Jagex continues to think that updates should be secret surprises, then we are never going to get beta testing. If Jagex starts having Blizzard's mindset and doesnt care whether people know of upcoming content, then we will see betas. It's all about the company mindset.

[tortilliachp]

Should Jagex start beta testing with players?

They actually already do this.

 

this. But the beta-testering program isn't huge. that's why it's a well-kept secret. If it were widely known, updates would be leaked, speculations would flourish even more than currently, and the runescape economy would be based even less on real supply and demand, and even more on merchanting, speculation and hoarding.

 

 

You can't ever have enough testers to cover all bugs. Runescape is too complex, all serious, exploitative bugs (bar the dungeoneering exp bug of recently) have been results of marginal behavior beta-testers cannot be expected to find. It's not like they could get the rewards, or items etc. from beta-testing within a closed environment.

[Me_Hate_Libs]

Biggest problem is Jagex believes their own hype about their game, updates, work, etc. They think they have to protect themsleves at every turn and can't possibly be honest with their players about future updates or what they're working on or anything.

[qeltar]

Should Jagex start beta testing with players?

They actually already do this.

Prove it.

 

You always jump into threads with your self-congratulatory "I have insider info" claims -- back it up for a change. What beta testing program does Jagex have? Who is part of it? How is it used?

 

You can't ever have enough testers to cover all bugs.

As mentioned earlier, this is not why Jagex needs beta testers.

[Pirate_Felix]

No you will get the same crying as with flag of festivities

 

I WANNA BETA TEST *WHINE WHINE*

It gives you no ingame advantage

BUT WHY CAN THEY AND CANT I BETA TEST@!!@ *WHINE WHINE*

 

I WANT A FLAG

It gives you no ingame advantage

BUT WHY CAN THEY AND CANT I HAVE A FLAG!!@

[Me_Hate_Libs]

Pirate, Jagex has in the pass, supposedly, allowed their players to whine and threat and conjole and "riot" over decisions they have made that they say is in the best interest of the game. So they could still do beta testing regardless of the people complaining, if they decided to.

[Carl]

I see no reason what-so-ever for anyone other than Jagex empoyee's to "beta test" anything RuneScape related. There is nothing you will find that the QA team probably haven't already went over, it's when there's a hundred-thousand people playing the same content that bugs start to arise, not 50. Unless they decide to use known bug-abusers to beta test this isn't going to happen and doesn't really make of much discussion because all know it's not going to happen.

[Pirate_Felix]

bruno must apply for a job XD

[Den]

I WANNA BETA TEST *WHINE WHINE*

It gives you no ingame advantage

BUT WHY CAN THEY AND CANT I BETA TEST@!!@ *WHINE WHINE*

 

How does knowing future updates not give any ingame advantages?

[91soldin91]

The only thing that would be hard for Jagex to control would be in-game insider trading. The only way to 100% prevent this would be to not allow the beta testers to play the game while testing. Of course they could just threaten to remove anyone who insider trades from the beta program and even temp-ban their account. Seeing as it would be extremely easy to detect someone buying 20mil worth of dragon bones right before a prayer update, that would cut down on most problems.

[Barihawk]

Public beta testing is a failure and essentially boils down to people downloading it simply to play it as if it were a demo.

[langer]

Should Jagex start beta testing with players?

They actually already do this.

 

Please take the time to at least read the post. I asked about beta testing like they did before releasing RS2, when members and then everyone could try out the beta game. I'm asking if every update should be treated like this...

[qeltar]

As promised (reposted; original here)

 

Most of the concern about and opposition to the idea of beta testing for RuneScape seems to be based on issues related to Jagexs ability to implement such a program while simultaneously maintaing secrecy about upcoming changes. A lot of players seem to be incredibly worried that implementing a beta program will give those involved in the program an unfair advantage. There also seems to be a general consensus among the anti-beta crowd that Jagex somehow is dealing with a unique set of circumstances here.

 

A thorough discussion of all of these issues would take far more time than I have available at the moment. But I do want to throw out a few points for consideration. While youre not really supposed to number points in an essay, I will put convenience over custom and do so anyway.

 

1. There is nothing that unique or special about Jagex or RuneScape when it comes to secrecy or security. RuneScape may be a special game, and Jagex a special company, but the design, development and implementation of RS is no different than the process used by hundreds of other companies to make hundreds of other software products. Whatever the issues that are raised, all of these other companies have had to deal with them. And they all have, with very, very few concluding that they had to never implement a beta testing program because of those concerns.

 

2. Jagex already has ways of ensuring secrecy. This isnt Andrew and Paul Gower in a garage somewhere designing features and updates. Its a company with hundreds of people, and I doubt there is anything either planned or in progress within Jagex that isnt known by dozens of people, including many who are not directly involved. Despite this, real leaks about upcoming features and changes are very rare. Clearly, Jagex already knows how to ensure security in this regard.

 

The obvious response is to then say that its different when you are dealing with players versus employees. To that I say: why is that necessarily so? Sure, employees have something to lose if they violate a confidence, and perhaps more than a regular player ever could. But it is very possible to do the same thing with beta testers: give them a carrot to keep the secrets they have promised, while holding a stick in reserve for those cases where they do not.

 

There is nothing magical that transforms someone from untrustworthy git to saint by virtue of receiving a Jagex paycheck.

 

Most Jagex employees have regular accounts. How does Jagex prevent them from taking unfair advantage? Either they have a way, in which case it can be deployed here as well, or they dont, in which case it obviously doesnt really matter that much.

 

3. There are already trusted groups of RuneScape players. Would Jagex just pull random people off the street to hire as employees and share private information with? Of course not. And they shouldnt, and neednt, do that with beta testers either.

 

Nobody in favor of beta testing is suggesting they take applications from random players, or that people that Jagex knows nothing about should be allowed to enter such a program. Jagex already has groups of players that they entrust with power and information: player moderators and forum moderators. I believe there is also a forum for maxed out players, where some information is made available to which the average player is not privy.

 

These folks have already proven themselves to have at least a basic level of trustworthiness, or they wouldnt be in their positions. And as reference in point #2, they have something to lose if they dont do as they promise.

 

4. Risk of disclosure is not a justification for never allowing private information to leave a company. While some players may think that the idea of future updates being leaked would be some sort of unimaginable disaster, in the end it is just a few items changing in price in a game. There are tons of companies that use non-disclosure agreements to protect againt real potential financial loss if secrets are violated: were talking about millions of actual dollars being on the line, not just millions of geepees. If they can find ways to make things work, so can Jagex.

 

5. The impact of potential leaks is severely overblown. Suppose Jagex implements a beta program, and one bad apple gets into the program and violates confidence. I dont think this would really be the end of the world. Its not so long ago that all updates were telegraphed via a Behind the Scenes posting at the start of every month. This didnt exactly cause any major problems, and in fact, many players wish those announcements would return.

 

A more recent example of deliberate leaking of an upcoming event was the bonus XP weekend. And while that did cause distortions in the market and make a few items difficult to buy or sell for a while, it wasnt the end of the world.

 

6. Bad updates and quicky fixes can be just as disruptive as leaks. Dont underestimate how badly the market and the game as a whole can get messed up because of poorly-considered updates. The recent positive feedback loop with respect to the jewelry trader is just one of many examples. Avoiding bad design decisions means avoiding these disruptions.

 

7. Jagex can manage the unfair advantage issue. There are a number of ways that Jagex can minimize the issue of some players getting unfair advantage due to knowledge they have about upcoming items. For example, they could restrict player trading during a beta test period; or they could prevent those who agree to be testers from logging in during the run-up to a release. These players can be monitored and managed as necessary to ensure compliance with policies designed to ensure fairness.

 

Even better, they could just stop with all the secrecy and tell us whats in the pipe. If everyone knows the basic gist of the updates, nobody has an unfair advantage.

 

Picking players who already have achieved so much that they dont need an unfair advantage would also help. I highly doubt that someone with all 99s and a billion gp in the bank is going to risk his/her account and status by violating a pledge of secrecy as part of a beta program.

 

8. Im more concerned with what I am doing than what others are doing. Even if Jagex did absolutely nothing to prevent beta testers from getting the inside scoop and taking advantage of it, I really wouldnt care all that much. It wouldnt be ideal, but its a small price to pay for a quality product.

 

So there, I said it: I dont really give a fig if a beta tester makes a bit of money on the side. It would be a tiny drop in the bucket, certainly not even close to all of the money made by merchant clans, for example.

 

Insider trading happens in the stock market. Nobody suggests this as a reason to shut down stock trading. The idea that something that would improve the game for everyone should be shot down because of the chance of a few people getting a small advantage for a few days or weeks is short-sighted and silly.

 

9. The issue is design and balance, not bugs. Opponents of beta-testing keep focusing on bug-fixing, usually saying something like a small number of people wont find bugs any better than Jagexs in-house testers. Trueand irrelevant. Jagex doesnt need players testing releases because of bugs; it needs them because the company consistently puts out releases that need tweaking due to not having properly anticipated how real customers would use them.

 

And finally:

 

10. We are customers, not children. I find the notion that it is impossible for Jagex to find a small number of real players who are trustworthy enough for a beta testing program highly insulting. We are customers, not little children who cant be trusted with anything. Beta testers should be adults who have already proven that they are reliable and trustworthy, and they should be treated as such. Believe it or not, there really are people out there who place a higher value on their personal ethics and reputation than making a few extra gp in a game.

[pulli23]

Public beta testing is a failure and essentially boils down to people downloading it simply to play it as if it were a demo.

You apparently have no idea what real beta testing is..

 

Been beta testing `game maker´ very much (game maker is a "game development IDE + programming language"). And during the beta test (All test together took half a year) over 1000 bugs were found and fixed.. And only about 50-100 people were actively reporting issues..

 

 

However I do share jagex concerns with "impractically in runescape". Only thing they could do is: you can sign up for a beta, and during that beta your can't use the "old version" anymore (and your gains would be wiped at the end of the beta).

[Creaphis]

I agree with Qeltar. There are definitely ways that this could be done. The main problem would of course be the economical implications, but if the update is leaked to general public it's basically like knowing every update a certain time before hand. This would not be so bad, it would not spoil the updates too much, as the updates are already "spoiled" by developer diaries and such. And updates, contrary to popular opinion, are not about surprises, one should not care if an update is spoiled or not. It's all about how good the update is and how it intergrates into the game, not about how fun and surprising it is during the first week of release. Now what do we choose? The element of surprise and satisfaction for a week, or content that intergrates well into the game and is a success from the very start of the release? I wouldn't mind knowing what an update is a bit beforehand and then having the update be a very good addition to the game.

 

This is a very important point. Our arguments over whether players can be trusted are ultimately moot, because it really doesn't matter if we know what content is going to be in updates in the first place. Everywhere else in the game industry, the goal of advertising is largely to give potential players a good idea of what's going to be in a game to encourage them to go ahead and buy it. Jagex, on the other hand, has this peculiar belief that keeping future game content a secret will somehow make it more enticing. I doubt that this belief has any basis in fact. Personally, I'm more likely to look forward to an update if I know what it is that I'm looking forward to, and I'm also perfectly happy to play through game content that has been in this game for years - so, obviously, it's not secrecy about the content that keeps me interested in this game. Secrecy like this is unnecessary, and even counter-productive, because it prevents Jagex from getting useful player feedback before updates are released.

 

With this point in mind, here's what I think is the best possible beta-testing system:

- Some players are carefully (or perhaps randomly?) selected to test future content before release.

- These players are not asked to keep the content of the updates a secret - they are even expected to spill it.

- These players provide feedback about gameplay and design choices back to Jagex staff so that updates can be modified in these important ways before release.

 

Note that this is not an open beta, so most players will still have to wait for the update to get to try it, and they will still look forward to updates as much as they always have. Note also that without any element of secrecy in the beta testing process, we don't have to worry about player trustworthiness, NDAs or "insider trading." Note also that some bugs will always slip through the beta-testing process, but this isn't important. What is important is that major design decisions, which would be considered mistakes if released, are reevaluated before it's too late.

 

Unfortunately, this would still increase Runescape's operating costs, as I mentioned earlier, so we'd still be paying more per month or be getting fewer updates under this system.

[Glove]

Well, the main problem with releases generally isn't bugs, but balance issues. You know, like when a quest is released with an agility obstacle that is coded so you have to pass an agility check with each step (at increasing difficulty) instead of one or two agility checks, ultimately making it much harder for people with low or middling agility to pass the obstacle than intended. I don't know how they test this kind of thing, but something is going wrong for them somewhere in the process because this sort of thing is a recurring problem with their implementation of designs. Somewhere in the design and implementation process certain things are given more weight than others (like I suspect with dungeoneering that the cost of reward items was looked at heavily in relation to the impact on the rest of the game outside of dungeoneering and priced from there, and then that cost wasn't compared and compromised to the level and amount of experience it would take to get the tokens that was already set for the skill itself causing a huge disparity between the two) and then probably not reconciled with other elements (ultimately undervaluing those elements and causing the imbalance); I mean, I don't know, obviously I don't work for Jagex so I have no idea what the specifics of their process is, but the results end up undeniably skewed. Having a professional beta team that was independent from the design, development and implementation teams (as well as skilled with assessing balance) would probably greatly improve this.

 

That said, there are quite a few obstacles and disincentives for Jagex to actually do this that don't stem solely from Jagex being stubborn secretive bastards.

 

1. They would need to hire professionals.

 

-They would need to hire professionals instead of using the player population because it would give those player testers an advantage over other players. Not only would there be a murky grey area where the testers might use their knowledge to buy up resources before a big update and then profit immensely, but there's also the fact that knowing how a skill or update works from the get go (having already gone through the awkward 'zomg this is new, how does it work?!!' phase) means that you already know how to train efficiently or how to use it to your advantage right away while other players (the non-testers to a large degree) don't. Since RS has a large number of ultra competitive players, this would cause a huge ruckus; one that is more trouble than it's worth. So, they could use players, but only if those players never played the main game again. Not to mention that whatever you think about the fairness, levity and consistency of Jagex's decisions in regards to keeping players on even footing, Jagex thinks they're doing a good job of it and most likely wouldn't want to jeopardize that by implementing something like this on a large scale.

 

However, one of the major problems with using professionals is that they cost money. They would have to either get this money by cutting into their profit margin (what incentive do they, as a company, have to do this when plenty of people are willing to pay for membership with the current development and implementation model?) or by raising subscription prices (yeah, no, one of their main selling points is that they don't cost as much as online games from other, bigger companies). They could, maybe the improved implementation and integration of updates would improve their subscription rates and they would make more money, but it's a risk. They have a business model, that while flawed, is working to earn them lots of money. Players complain and wax poetic, but they still pay.

 

2. The community itself.

 

-Okay, stay with me here and don't take this personally. I'm not speaking about or for each and every individual player, instead I'm talking about generalities. Every update there is a segment of the population that complains loudly; sometimes with good reason, and sometimes just because there is a strain of people resistant to change. There is a tendency for hyperbole in that every change that a segment of the population likes is 'super awesome!!!! (insert epic win face here)' (whether or not it is broken or unbalanced) and every change that a segment of the population dislikes is broken or unbalanced (again whether or not it actually is). This a problem that every game developing company faces. But what's unique about RS is how long the game has existed. This causes a larger divide in the community because the game is constantly changing so each new 'generation' of player (if you will, there is no 'generation' that is inherently better or has a 'truer' vision of what RS is) has a different idea of what RS is or should be; this overlays and amplifies the divides between the usual differences in play style and stylistic likes/dislikes that usually occurs within any community around a game. This means that even if Jagex does change it's business model and design/development/implementation model, they are most likely still going to get a heck of a lot of negative feedback from players saying that things are broken just because they don't like them. While on its own this would be a minor disincentive, paired with the two other factors mentioned and that plenty of people already pay for the game with method of updating and fixing unbalanced material the way it is, this just becomes amplified.

 

Should they implement a new system of beta testing? From a design perspective, it would probably make new content integrate much more smoothly, so yes. From a business perspective, it's questionable whether the potential for profit outweighs the drawbacks and costs.

[howbadisbad]

But their ridiculously secretive with their updates. Blizzard on the other hand tells you whats going to be in the next patch months in advance sometimes.

 

 

Why? The world may never know.

 

The best I can come up with is suspense. But then you get disappointed with an update like dungeoneering and the rants forums go crazy.

[Guest jrhairychest]

Sorry chaps but no-one for the argument of 'pro player beta testing' is convincing me here, for the following reasons:

 

• I dont care how long a player has been playing or their conduct, you still can't trust them. They're not a paid employee so there's no way you can regulate them. So what exactly will they see or do? You can guarantee it won't be as much as a bona fide games tester, employed by Jagex, who has signed a NDA as part of their employment contract. I couldn't care less whether some posters ridiculously discount this but its the way Jagex (and many others) work.
  
• If we remove trust for a second, then in reality we're all beta testers anyway. That's part of what we do by providing feedback.
  
• Glove's point on hiring professionals is quite valid.
  
• Its already been quoted by a jagex Mod about the cost and resourcing implications of doing this type of thing, so what benefit at what cost?
  
• You have no guarantees that Jagex will act on issues raised by it (see rants etc. for things that have needed 'fixing' but never done)
  
• There's no guarantees that the players themselves will do a good job. I can honestly see many players taking up this option for the wrong reasons, mainly for their own vanity.
  
• The game works well enough as it is. If you really think it's that bad then maybe you should consider playing something else. Personally I love the game, warts n all.
  

[Arcane_Rift]

All they would need to do is create a dedicated beta-server that is a mirror image of the current world and allows players to log-in and test the new content. They can allow for members only to login just like they did for the beta-test switch to RS2. I don't see any downside to this.

 

I don't see how loading one additional server with content can be that costly. Heck, they can even bring down one members server and set it up for a beta server. I doubt it would be missed that much.

 

This information obviously would be made public and everyone would have access to it.

[tortilliachp]

All they would need to do is create a dedicated beta-server that is a mirror image of the current world and allows players to log-in and test the new content. They can allow for members only to login just like they did for the beta-test switch to RS2. I don't see any downside to this.

 

I don't see how loading one additional server with content can be that costly. Heck, they can even bring down one members server and set it up for a beta server. I doubt it would be missed that much.

 

This information obviously would be made public and everyone would have access to it.

and what about the insane amounts of GE speculation, hoarding and manipulation?

 

left to its own devices? no, you give no plausible way of instigating wider beta testing than currently.

[qeltar]

I dont care how long a player has been playing or their conduct, you still can't trust them. They're not a paid employee so there's no way you can regulate them.

False, as a matter of fact. Again, every software company in the known universe finds ways to make this work; there's no reason Jagex can't do the same.

 

And I'm still waiting for an explanation as to why "leaks" would be the end of the world. We used to get them every month, on purpose.

Its already been quoted by a jagex Mod about the cost and resourcing implications of doing this type of thing, so what benefit at what cost?

They haven't given us any real figures nor even any sort of convincing argument about why they can't do this. It's not like they are hurting for funds. They just go "oh my, that would be so tough!" and everyone falls over themselves in agreement, without any critical thinking being involved whatsoever.

 

The real reason they don't have beta testing is that they are control freaks. They are obssessed with their "surprise" updates, secrecy, little hints and keeping everyone in the dark. And they simply don't care about what all of that nonsense is costing everyone, themselves included.

[RSBDavid]

Its all about trial and error. They should allow some players to test one future update after the Q&A finish up. The players must fillout a NDA and provide feedback on the update. If it works then fine, if not then they know one way of how not to approach beta testing.