First off after listening to the pod-cast on mmorpg.com. This was a great pod-cast! And inspired me to write this thread!
They are other testing threads out there but it’s about specific areas, unless i missed the other ones then accept my apologies.
At one point we all know that Pantheon will start testing phases with pledges in alpha and beta etc etc. The goal is to develop a testing etiquette build by players such as myself where it would help VR in the end!
Pending on homework and direction received from VR upon entering the game one day, this is my perception on the etiquette that should be used while testing the game! And all this in a constructive and respectable way as much as possible!
I believe after each game session any testers should have something to say.
How to do this?
i. Explain your experience from the first minute
ii. Identify difficulties learning the game from the beginning
I believe that this very important since usually a game for so many players is based on the first 30 minute in the game
If there’s a tutorial (don’t expect one from day one public testing) the tutorial would need to be complete because once you are done with the tutorial, no one should hold your hand and this is why this game is calling at me so much!
iii. Solo or group experience should be scrutinize.
Solo content
Crafting
Skills
Solo combats
And so on
Group content.
If in a group i believe strongly that a voice chat should be available if not from VR there are 3rd parties such as TS. And honestly it would be good to have a controlled TS for example so that information is not leaked out. But the reason why i believe voice chat would be important.
*So much can be discuss in a playing session with people in your group and can be addressed
*Tasked can be given specific members of the group in reporting specific bugs or experiences.
Group questing
Triggering group events
Group crafting
Group combat
iv. Environment glitches
Visual bugs i.e.: trees in mid-air
Running on different settings
Testing the game profiles
AI – text/behavior
v. Testing AI text spelling errors and so forth
Behavior if aggressive how aggressive will it chase you to the end of the zone?
Sharing thoughts on AI behavior – if it feels or doesn’t feel right in your encounter with the AI.
I’m sure i have not covered all aspect, feel free to build this thread up!
Cheers
I'm just hoping, since the pre alpha and alpha testing stages are supposed to be used for actual testing and not as a head start (full character wipes for each stage, even after open beta I believe was stated in the stream), that the world is locked down and those who've pledged to get tester access use the time to test only what the devs need tested day by day. Basically to improve the game rather than plot out the best route for XP or other shenanigans. As stated in the stream you are being granted the task of bug hunting to break the game, not getting an advantage based off how much you pledged.
I'd prefer only x amount of zones be active at once, as needed to test (would help to stress test zone loads too).
Classes x, y and z active at a time to test their mechanics rather than all at once.
Pretty much a screen on log in saying what's up and live that 48 hr span or something.
Etc.
(And yes I pledged to get one of the tester tiers myself aswell)
I say let the water flow down freely. Highly specific / instructional testing should be an internal thing. When it comes to external testers being brought on board, wouldn't you want them to be free to do whatever / wherever / however to see what comes up that devs & staff never foresaw ?
I think the key to the testing phases would be to have powerful tools at the devs disposal. Tools that can keep track of xp earned per character per hour per day (or whatever). Tools that can track income of coin / items per character, and then functions within these tools to run reports to see who is doing what & where. This sort of thing lets your birdies fly free, able to turn over rocks and scour corners unconsidered by development, and then allows them to fix mobs / drops / areas / environments that are unbalanced in comparison.
Sure, asking testers to come raid a mob, or test zone load capability, etc. is all well and good, but the structure you're suggesting here is probably a tad over zealous imo.
I think the most important thing will be the set of tools which the tester can use to report and give feed back. I know that if I am running along, trying to keep up with my group and I see something on the side of the screen that might be a bug or glitch, I don't want to have to open up a complex set of tools, or record all of my observations on paper. Having a couple buttons that quickly bring up reporting devices and other feedback tools would be very helpful. The ability to hit one button, chose my zone from a pull down menu, type in several sentences about something I saw, or an impression that I had and then send it would be great. The ability to report an error or bug should be simple and quick. If this is the case, my only worry will be for the devs and how many reports they will get!
Examples:
Zone: Kings Reach
Issue: Black Bear just NE of the crossroads upside down and jumping backwards. Also, untargettable.
Zone: Kings Reach
Impression: City walls are very impressive. The architecture took my breath away. I immediately sent a pic to my friend. Prehaps they could be a little less yellow though, more gray.
It was flat out stated in the Dec stream (I believe it was by Brad) that it'll be structured testing and not a "go forth and play as you will". We bought in to test the game, not play early.
I know opinions will differ and such as it is. These are development forums where we are asked our input towards the game, not how the game will actually be. /Shrug.
Structured testing can mean a lot of things. It can mean they open up 1 zone and say test everything here. So that is basically go forth and play just in a certain area. That is usually how testing goes. When devs find a trouble area or want more specific details they may ask for smaller targets but rarely is that the case.
As long as it's actual testing, which I'm sure it will be. Many games these days have lost the root purpose of "beta" or at the most their testing comprises of gathering data on player behaviors that's collected in the background (which can definitely be useful, but it's not comprehensive). Give me all the tools I need to give extensive feedback and bug reports!
I was waiting for one of these threads to pop up :D
Convo and I will actually be putting together a kind of demo/tutorial on how to properly test and report bugs using our bug reporting system, so stay tuned for that closer to when we are ready to announce pre-alpha testing :)
Beefcake said:I am sure they will state their expectations and desires to us clearly. It's not up to us to create the format or procedures. They run the testing, not us. They will tell us what the want tested, communicated, and the format.
Correct, although I would have said it a bit nicer, he was only trying to help and I appreciate that ;)
Xilshale said:I'm just hoping, since the pre alpha and alpha testing stages are supposed to be used for actual testing and not as a head start (full character wipes for each stage, even after open beta I believe was stated in the stream), that the world is locked down and those who've pledged to get tester access use the time to test only what the devs need tested day by day. Basically to improve the game rather than plot out the best route for XP or other shenanigans. As stated in the stream you are being granted the task of bug hunting to break the game, not getting an advantage based off how much you pledged.
(editted because apparently a bullet outline doesn't work)
Jasper has it right. They key is details and the ability to reproduce. As a dev IRL nothing irks me more than getting a "bug" report that is little more than a screenshot of a vague error. I can't fix things I can't reproduce.
I am really looking forward to getting my hands on pre-alpha. Nice to be on the other side of the fence for a change.
It may not intentionally be a "head start", but it absolutely gives testers an advantage over non-testers, no matter how small.
And if I had a dollar for every time I've seen testers find bugs, not reveal them, then exploit them after launch, I'd have probably 50 bucks. :) The days of wide-eyed innocent & naive fans doing exactly what they're told as part of alpha or beta is long past. At least 10% of your testers are there to find exploits they can use for RMT activities post-launch, including dupe bugs, table enumeration, zone enumeration/mapping, teleportation, movement speed hacks, cheat client development, all of that.
New MMO's are a source of revenue to a portion of humanity, and they're rarely/never part of the development team or hosting company of that new MMO.
The serious issue that confronts VR on this topic is the ease with which Unity can be compromised, client side. It's very likely VR won't be making an entirely custom executable from scratch, with encryption/obfuscation options that are customizable with every build, and even less likely that will be the case during alpha/beta. During that time, those malicious testers will be debugging, memory scanning, and analyzing the hell out of your client with the goal of producing MySEQ, Pantheon edition.
With such a long public testing phase, the accusations of favoritism will also be something the VR PR machine will need to deal with. (Why does UberCo get to test Amberfaet when the rest of us are testing Halnir Caves!??!! Why is there a developer in the guild UberCo??? Why is there a developer on UberCo's Mumble/TS/Vent/Discord?!?) It's never ending.
Even how exploited bugs are dealt with will be a problem. Case in point, during the development of SOTA (Unity, MMO) a guild found a currency dupe exploit. After they exploited it to the point that they had enough to absolutely wreck the testing economy, they told the devs. The devs did not remove the currency. They also didn't wipe the currency between testing builds. So, currency was valueless, yet they used economic testing data from the server, and made design decisions based on it.
In short, due to their inaction (no rollback during testing?) they invalidated over a month of tester effort. All the economic data, server side, was polluted/contaminated. When confronted publicly about this exact issue, they said "In retrospect, we probably should have rolled back". Yet when similar things happened later, they still wouldn't roll back. At many points, they lamented a lack of tools and procedures to deal with such a catastrophe.
After that, it was pretty easy to see they had no intention of seriously testing out any Customer Service processes or procedures prior to launch, so testers did whatever they wanted, and there was no repercussions of any kind. As a result? Exploited terrible bug filled game. Learn from the mistakes of others, VR! :)
Gnog said: Your post makes me sad Vjek. Not saying you're wrong. Just a sobering reminder of the challenges any MMO faces to create a fair playing field unpolluted by exploiters and RMTers. I am hopeful that VR will do better than other games on that front.
Which is why I'm ultra leery of the "play til level something for free" method they are planning. Unless free access has so many limits it's practically a solo tutorial, which would defeat the purpose.
Pretty sure we'll have clear instructions on what the Devs want tested..and that we'll have /bug or /report buttons that we can use during testing and at the end if needed. It may be that our Devs have different requests as we go along. Ugh ..stop talking about this and let us in!! hehehe *dies waiting* (kidding)
@vjek Par for the course...and in former games if people were caught exploiting they were banned for it. We should all do our best to find exploits and report them right away before release. That is if we care about Pantheon's success.
Just for piece of mind - We will control the testing as we see fit, we will be gathering data and watching live on visible and invisible characters, we will have many VR team members in-game and watching, directing, helping, communicating plus controlling access and play sessions, so I wouldn't worry. We have all tested other games in some form or another, some of us are very experienced in this field and some of our devs create the games and know all the tricks and exploits to watch for, we have things in place for that exact reason, so what someone may think is a secret bug for them or their guild at launch will most likely already be detected and logged for patching before it goes live, if the odd one slips through, it will be patched once the data starts flowing on the live game.
Kilsin said:Just for piece of mind - We will control the testing as we see fit, we will be gathering data and watching live on visible and invisible characters, we will have many VR team members in-game and watching, directing, helping, communicating plus controlling access and play sessions, so I wouldn't worry. We have all tested other games in some form or another, some of us are very experienced in this field and some of our devs create the games and know all the tricks and exploits to watch for, we have things in place for that exact reason, so what someone may think is a secret bug for them or their guild at launch will most likely already be detected and logged for patching before it goes live, if the odd one slips through, it will be patched once the data starts flowing on the live game.
If a Dev witnesses a player/guild uncovering/exploiting a bug, and the player fails to report the bug in hopes of exploiting when the game goes live, will that player be punished? Testing access revoked? Warning email from the Dev team?
Homercles said:Kilsin said:Just for piece of mind - We will control the testing as we see fit, we will be gathering data and watching live on visible and invisible characters, we will have many VR team members in-game and watching, directing, helping, communicating plus controlling access and play sessions, so I wouldn't worry. We have all tested other games in some form or another, some of us are very experienced in this field and some of our devs create the games and know all the tricks and exploits to watch for, we have things in place for that exact reason, so what someone may think is a secret bug for them or their guild at launch will most likely already be detected and logged for patching before it goes live, if the odd one slips through, it will be patched once the data starts flowing on the live game.
If a Dev witnesses a player/guild uncovering/exploiting a bug, and the player fails to report the bug in hopes of exploiting when the game goes live, will that player be punished? Testing access revoked? Warning email from the Dev team?
That is something we will deal with internally, we won't be revealing all of our possible outcomes as it would most likely be case by case with a general guideline/rule in place to base a decision off, but that is for us to worry about, not you folks.
I suggest everyone watch the following video to help you learn how to give the devs better, more productive feedback: https://www.youtube.com/watch?v=0T2oQE-HSbI
Day9 is full of wisdom
Bazgrim said:I suggest everyone watch the following video to help you learn how to give the devs better, more productive feedback: https://www.youtube.com/watch?v=0T2oQE-HSbI
Day9 is full of wisdom
I like this guy, directives are what I ignore on a daily basis :)
