Dev Diary: Updates to our Testing Program

Hi everyone,

Recently, we opened applications for a new testing program to help us ensure that we are catching more bugs and generally reaching a higher level of quality with our updates and the new content that we are adding to Pantheon. We are now in the process of kicking off that program and welcoming our first groups of testers. In the interest of transparency, we are pleased to share more information about how this testing program works, as well as our overall plan for future testing.

There are some big changes that could affect you even if you didn’t apply to be a tester, so read on!

Two types of testing, two groups of testers

To make sure that a game as complex as Pantheon works well for all our players on our regular servers, there are two different types of testing that we need to be able to do. We call these focus testing, and organic testing.

Focus Testing could entail evaluating individual changes to combat mechanics or class abilities, mechanical testing of a new gameplay system to make sure it works properly, or limited testing of new content to help us do initial tuning prior to a broader release. During focus testing, our Focus Testers are asked to do specific things, often over and over, and VR staff are in attendance to gather data and reproduce bugs in real time so that we can make fixes and adjustments immediately. Focus Testers are chosen through an application process. To facilitate focus testing, we have updated and renamed our Advanced Tester Program as the Focus Tester Program.

Organic Testing is when we open up a set of changes to a larger group of testers who play the game normally. We will monitor the effect of those changes over a period of several weeks to confirm all is working well before we push the changes out to everyone. This type of testing includes balance changes to experience and level pacing, economic changes, or really anything that might benefit from passively watching how players interact with it across the breadth of the entire game. This will take place on a Public Preview Server. Anyone with a Pantheon account in good standing can join a Public Preview Server. We encourage feedback on your experiences on Public Preview Servers.

Public Preview Servers

What to know about the public preview servers:

• The server will not always be open. When it is open, it will typically only be available for weeks at a time.
• Each time we open the preview server it will be a fresh server, and you’ll have to create a new character to play on it.
• Depending on the changes we are testing, we may place NPCs that allow you to boost your level or obtain gear to get started quickly. However, this is not guaranteed.
• The preview server may receive additional updates during the testing period, if needed.

The goal of the preview server is to allow us to gather data on how a change is working out before we push it out to all of the normal servers. This helps keep our normal servers more stable and allows us to try to make sure that the impact of changes is more fully understood before we push them out everywhere.

Focus Testing Program (fka Advanced Tester Program)

As mentioned, we have updated our focus testing program:

We’re making these changes for a few reasons:

Preventing Tester Burnout and Maintaining Coverage: We experienced a high burnout rate in our old program and sometimes ran into situations where we didn’t have enough testers available to form full testing groups. In our new program, we are planning to operate a larger pool of testers overall. We hope that this will put less pressure on individual testers and will in turn reduce the burnout that they experience over time.

Better coordination: Our old format utilized general group playtests that sometimes did not have specific objectives. While organic play and feedback will remain an important part of testing, our game has also grown to the point where we need to start asking our testers to focus on specific aspects that really need to be thoroughly tested. This allows our team to gather focused bug reports and feedback quickly whether we are looking at a gameplay mechanic or system, or a new or updated content area.

Things that have not changed: Our test server uses a separate database from our production game servers. This means:

• Testers will not be able to log into Production Servers with Test Accounts or vice versa
• Testers are not able to transfer items between Production Server accounts and Test Accounts

How Focus Testers are selected

In our last round of applications, more than 1,000 players applied to join the Focus Testing Program. To help get testing up and running for our upcoming high-level content, the first group of testers were blindly selected from this pool based on data relevant to our initial testing needs.

Here’s how our selection process will work for additional groups of testers moving forward:

1. We remove any invalid applications. These can be applications that have missing information, along with duplicates from the same person applying more than once.
2. We remove any applications where the player’s account has had a warning or suspension against it recently.
3. We remove any applications where the player has not logged in to the game within the last 90 days.
4. Once all three of the above steps are complete, we sort the applications by time zone, and we select testers based on trying to maintain coverage so that there are always multiple testers available to support scheduled focus tests.

As we start to fill out our overall tester pool, we will select randomly from the applications that make it through the filtering process above.

For any number of reasons, Advanced Testers may be retired or cycled out of the program, at which point we will select new testers from the pool based on the above parameters.

The rules of Focus Testing

Our testers are interacting with extremely unfinished systems and content that may be changed multiple times before they are publicly released. What our testers experience during focus tests is usually the earliest possible version of what eventually makes it out to the wider game.

To be part of the program, our testers sign a non-disclosure agreement and pledge not to share any details of their testing activities publicly. There are two reasons for this: First, this is meant to help prevent confusion among our larger community about what is coming to the game. Secondly, this helps protect our testers from any potential online harassment that might occur if their status was publicly known. As part of this, we also require our testers not to use the same character names as their main play characters, and to not share that information (including their guild affiliation) with other testers.

It’s important to stress that this NDA applies to anyone – family, friends, guildmates. If a tester violates the NDA, they will be removed from the program, and further action may be taken if necessary.

• In addition to following the NDA, to continue participation in the Focus Testing Program, we request: Participate in at least two feedback discussions each month.
• Inactive testers will be removed from the program.

Finally, we ask our testers to adhere to a strict code of conduct when interacting with each other and with VR staff. It is extremely important that everyone is treated with respect, both in-game and in feedback discussions.

• Failure to meet any of these requirements, or violation of the NDA, will result in the immediate removal of testing privileges and potentially being banned from the game.

Thank you to all the Pantheon: Rise of the Fallen players who have stepped up to participate in Advanced Testing!

– Nephele, and the Testing Team