Mass testing


Contents

What is Mass-testing?

Mass-testing events are organized by the CCP QA department on the test server Singularity or Duality in order to test parts of EVE, which need a high number of characters participating at the same time. In most cases the main interest is on the performance of the server and the client in extreme situations like massive fleetfights.

Mass-Testing provides the following benefits:

  • allows to gather performance trend-data for EVE (both client and server)
  • allows to test high-priority changes by a large number of EVE players
  • allows to get critical feedback about new features/changes to EVE, before we release them to Tranquility (or "TQ," EVE's production server).

Why is this not being tested by CCP in-house?

Public mass tests are indeed not the only way to test performance related changes. Specially thin clients (see this Devblog from CCP Atropos) are a very important tool, especially recently. For some tests thin clients are much better, for example due to the possibility to repeat a test with exact the same environment. BUT: Some problems slip through, as automated tests never have the same behavior as real players in a fleet fight. For example: Many players get nervous and might click the jump-button multiple times, when the gate jump is very slow - but not all of them. Another example is also described in this DevBlog from CCP Veritas. Thin clients also won't find UI-related problems, for example if the overview is causing client-side performance problems.

The standard way to deploy a small server performance improvement is the following: First the change is tested locally by the developers and/or QA to make sure that it does not break anything (major). Then it is tested with thin clients - this is the best opportunity to get clear numbers on how much the server performance is improved. Next step is to run a mass-test on Singularity to make sure that it still works with real players. If everything is fine the improvement is deployed to Tranquility at the next opportunity. In most cases the improvement is first turned off on TQ and then turned on a few days later to be able to monitor it better.

How to join a mass test event?

Prerequisite: You need an active account on the test server (Singularity or Duality) , which should be the case if your account was active on Tranquility for the last few months

  • If your account is not active on the test server, please visit the Reactivation thread for more information about how to get it fixed.
  1. Install / patch a client to be able to connect to the test server, instructions can be found on the Singularity page or the Duality page.
  2. It is a good idea to use logserver, for the case, that any problems are occuring. Important: Logserver has to be started (and a workspace created) before the EVE client is started.
  3. As soon as you are logged in: join the channel MassTesting, where you will find more detailed instructions for the test in the MOTD (directly before the test).
  4. Follow the instructions, which are given in the MassTesting channel.
  5. CCP devs and ISD bug hunters are present about 30 minutes before the mass test and help with forming fleets and moving players to the start system.
  6. Do NOT engage any other player in combat during the mass test, except when told to do so. Breaking rules during the mass test makes the mass test less useful for CCP and might result in a ban from both test servers.

In most events the following instructions will be valid, but sometimes we have to change them for special tests:

  • Use the channel MassQuestions for questions, talking and immediate feedback. The channel MassTesting is normally moderated during mass test events, to make it easier to follow the instructions.
  • For joining a fleet we normally use the channels FleetW and FleetX (and sometimes the channel FleetY). Please do not use other channels to ask for a fleet invite (except if no fleet-invites are sent at all during an event, then ask in MassQuestions).
  • The channel MoveMe is normally used for moving players to a market-seeded system and the channel MassMoveMe is often used for moving directly to the start system of the test event.
  • Most tests include a fleet-fight at a gate - a PVP ship for medium range is normally useful.
  • Do NOT use smartbombs, bombs or warp-bubbles, except if we ask for it.

What to do after a mass test event?

  • Write feedback in the feedback thread, which will be posted in the testserver forum. The thread will be linked at the end of the test in the MassTesting channel.
  • If you noticed any bugs during the mass test: please write a bugreport
  • If you gathered logs with logserver and want to send them to us: Please create a bugreport and attach the compressed logs (.rar, .zip or .7z). The current limit for attaching files to a bugreport is 16MB. If your logs are larger you can either attach several files (which is easier when running logserver in server mode) or you can upload the compressed logs to a 3rd-party file-hoster and give the link in the bugreport.

What settings shall I use?

  • It is useful (but not required) to run the client in windowed mode, to make it easier to change to logserver.
  • The FPS can be displayed with the shortcut Ctrl+F, we normally ask in the feedback thread for the average FPS of your client during a fleet-fight, both zoomed in and zoomed out.
  • During the fleet-fight we suggest you disable brackets to improve the client performance, which can be done by clicking the small white triangle in the top left of the overview window and selecting "Hide all brackets" (or use the shortcut ALT+SHIFT+Z, if you have not changed the default shortcut). We might ask to turn on brackets for special tests, but normally we assume in the feedback thread that brackets are disabled.
  • It is possible to get details about the client performance (for example memory-usage) by opening the client monitor with CTRL+ALT+SHIFT+M - but keep in mind, that this monitor might slow down your client a bit.

Free skill points

As small incentive and reward for participating in mass test events CCP QA adds two million free skill points on our standard test server Singularity to the characters which were present in the mass test. These skill points are given through the skill reimbursement tool.

Rules and clarifications for free skill points:

  • The character has to participate in the mass test. If anybody joins late or leaves early he might get nothing.
  • Rule-breakers might get removed from the list.
  • 2 million skill-points per mass-test, they are normally applied one or two days after the mass test.
  • When a new mirror is applied the skill-points of the last five mass tests (max. 10M SP) are reapplied (within 2 days after the mirror).
  • Questions, complaints about missing skill-points: Only in this forum thread.

More info