Q&A With Quality Assurance

Q&A with QA Lead, Kim Morrison

On a small development team, not enough can be said about the importance of our internal testers.They are tasked with repeating the most menial of tasks to weed out bugs, and they spend hours trying every crazy thing a player could attempt. They work hand in hand with nearly every department of development, and are a key factor in getting builds ready for our Pre Alpha testers to get into the world. It doesn’t stop there either. They then venture through bug reports, testing everything reported, as well as finding the cause of issues to report back to programers and designers. Our incredible tester team is led by Kim Morrison, a veteran of Visionary Realms, who we sit down with in this month’s developer focus. Let’s peek inside the world of QA, and Kim’s journey with Pantheon thus far.

It’s my pleasure to spend some time with you today, Kim. Thank you for your time. Can you please start by talking to when/how you joined the team and your role in development for any newer members to the community?

Hi Tony! It is great to sit down with you and talk about Pantheon and QA. I officially joined the Pantheon team in April of 2017. I started as a QA Tester and I am currently the QA Lead. Before being hired by Visionary Realms, I was a fan (and still am a fan) of the project. I was there for Brad’s first Tweet about Pantheon. I developed a rapport with Brad and offered to do some early testing for the project. The rest is really history!

Being a veteran to the project, what are some fond memories or moments you have working through development that you can share with the community?

I have several fond memories, many of which relate to time spent with team members. I can remember the first time I logged into Pantheon. Brad McQuaid was showing me around. I was a little star struck back then. Some other things that come to mind are interesting bugs that we have found over the years. I recall “sledding” on the hills of Avendyr’s Pass and fighting “hair” mobs in Black Rose Keep. And there was that time I GM Killed everyone at the end of a live stream.

We have some inside jokes that are fun and one in particular has become quite popular within the community relating to Roenick. One day I might give my side of the story behind Roenick’s first on-screen, in-game death. I believe it was at PAX. I hear one community member has mentioned publicly that it could be my fault? Pfft! Anyone who knows me, knows I’m a great healer! But, I’ve digressed, there are many many great memories so far and I know there will be many more made in the future.

One of your many key focus items is partnering to greenlight builds for our PreAlpha testers to get into. With the new HDRP build coming soon, what does it look like for you and your team both prior to getting into the build, and once you get your hands on it?

Before getting into the build I like to meet with as many members of the dev team as possible to gather information on their testing needs and priorities. Then Mark and I work together to write test plans for each zone and major system. We are currently in this phase for the HDRP build. By the time the build is in our hands, we will have a well mapped out plan for testing every corner of the world and every system.

Once we get our hands on the build, it is full steam ahead! We will spend the next several weeks to months working through our test plans, iterating with dev and giving feedback. This means we will cast and use every ability hundreds of times in different conditions, we will trade and gather and do the repetitive and more mechanical and purposeful testing procedures. We will submit and help prioritize bug reports. I’ll invite members of the dev team to test with us occasionally to demonstrate where some of our feedback is coming from, do team building and give them a chance to observe players running through the content they have created. But it is also important to understand that part of testing is also playing the game organically. I will definitely be bringing back our internal static group to do organic game play once or twice a week. During these play sessions we do not use developer commands and it is a great way to find bugs we weren’t particularly looking for and to gather intel for feedback. I will also start gathering data and thoughts to share at our PA planning meetings.

I imagine during submitting feedback on builds, there are a multitude of partners you take in the development and technology side, what’s it like to have so many moving components when giving feedback and important fixes? Who are some of your key partners on the team, outside of your QA team?

Well of course Mark and I work closely together on testing and planning. I’m generally the one to do outreach with other departments. On the Creative side, Adam is generally my go-to for most general things and for class feedback. I check in with Tyler a lot on combat and balance. If I have a programming question I go to Kyle since we are both night owls. But, I interact with pretty much everyone, even our awesome CEO, which is cool because CEOs aren’t always accessible in the workplace. Some members are more relevant to my daily work than others, but as I’ve alluded to earlier, my job has other considerations beyond testing which would have me interacting with lots of folks on the team. And who can forget IT. Once QA is ready to put their stamp on a build, we pass it off to the IT folks, who are the last stop before a PA session!

When Pre-Alpha sessions go live, can you talk about what makes a Pre Alpha tester’s bug report more helpful for your team? What advice can you give testers in submitting the best bug reports?

Generally speaking it is my goal to work with other members of the PA planning team to identify areas in which we want the testers to focus. The best advice I can give is to come to a session prepared, having read the information that we publish for the session. We like to have a list of known issues that shouldn’t be bugged and a list of focus items. We are really mostly interested in bug reports pertaining to that session’s focus, but we are ok with other bug reports being submitted as well. Just don’t be surprised if you report a bug that was outside of the scope of the test and you see that bug is present in the next PA session. We have sprints planned out in advance and you might have identified a bug that isn’t within scope to be fixed yet. Beyond that, the more information you can give the better. When you open the bug reporter, the system captures your coordinates, so keep that in mind when you open the bug reporter. Try to submit the bug in the very spot that the bug occurred, even facing the spot, if relevant. For instance, if you are reporting a seam or hole in the world that can only be seen when you face a certain direction and look down, document that. Try to provide us with detailed step by step instructions on how to reproduce the bug. These steps will save us a lot of time.

Once a Pre-Alpha session is finished, what’s the average load of tickets your team has to go through, and what’s the typical turnaround time?

We have seen as many as a couple thousand tickets, but once we narrow them down to a unique list, we are usually left with a range between 100 and 200 tickets. The turnaround time for my team’s work averages a couple of weeks, depending on how complex the bugs are and how many we have to find reproducible steps for.
Once we have all the valid bugs identified, we give each bug a priority based on our current development cycle and what is forthcoming in scope. For instance, it did not make sense to prioritize network related bugs last year knowing that new network code was in the pipeline. For this reason, it is possible and expected that some bugs might be persistent through multiple PA sessions as I noted in my last answer.

Being a veteran to the team, as we work towards the HDRP build, what can you tell the testers, and overall community, about what the upcoming future holds?

We have huge changes coming. We are changing the rendering pipeline and with that we are introducing a lot of new tech for Pantheon, such as terrain streaming. We are going to be very busy testing all of the changes including the tech, balance, terrain and the combat changes that were mentioned in this month’s developer round table and live stream. It is certainly an exciting time to be part of this team and community!

Thank you Kim for sharing some tips to bug reporting, and allowing us to look behind the curtain at the process you and your team go through. We are thankful for you and your team’s work, as I’m sure the community is when they get to get into the world thanks, in part, to your team’s efforts.

Producers’ Letter	Monthly Recap	Q&A with QA	Into the Spotlight