7th June 2018

Extended Maintenance - Merge and Northdale Preparations, scheduled 6 months ago

Greetings adventurers, As you may know, we will be merging our realms during todays maintenance. In addition to this, we will also be performing some extended maintenance on our infrastructure in preparation of launching our fresh realm Northdale.

Todays maintenance is estimated to last between 4-12 hours. During this time period all of our game realms and login servers will be unavailable.

We will update this page frequently.

All timestamps are in Server Time (UTC+1)

Update 1 - 12:43:

We are making great progress on the maintenance, currently backing up nearly 400gb worth of data. Once that the backups have been verified and saved in multiple locations, we will proceed with the reinstallation of one of our servers. When this is completed, we will start the merge process, which is estimated to take 4-12 hours.

Update 2 - 14:07:

The database merge has begun. Just to be on the safe side we're going to run it on both the server and a development machine incase something happens.

Update 3 - 15:33:

The backup process took a bit longer than expected due to unknown slowdowns in OVH's network.

Our main server is now being reinstalled and once it is back up we will configure it and re-import the DBs, this process is estimated to take around 2 hours.

After that, our merge script will have to run and merge the databases, this is the part of the maintenance which will take the most time.

Update 4 - 17:15:

The merge process is taking a lot longer than expected, we have slightly underestimated the sheer amount of data that we are dealing with.

We currently have to merge 3069739 characters into a fresh database, however we are going to start another instance of the merge where we merge two databases into Lightbringer instead (technically the merge script does the same thing, but the reason for using a fresh server as a target means that we don't have to reimport any dumps if there's any corrupt data)

We'll give this approach a go on our second machine which means we only need to merge Anathema+Darrowshire characters into the Lightbringer database, which should save us a few hours.

Another temporary thing that we might do is to start up the servers again in a "read only" state which means any progress is not saved. We might also add the PTR gear vendors to the servers to allow for people to have some fun together before the merged server comes online.

Update 5 - 18:32

After 3 hours of running the merge script, the progress is only 389450/3069739 ( 12.69% ) which means that the merge is likely to take at least another 24 hours.

As we mentioned earlier, we are toying with the idea of launching a temporary PTR realm with gear vendors while you guys wait. We have setup a poll below.

Please submit your answer if this is of interest or not.

https://www.strawpoll.me/15844364

Update 6 - 21:34

The merge is currently 815581 / 3069739 (Characters merged - 26.57% ) - We estimate that Lightbringer (merged) will be back online in approximately 22 hours provided that the merge script successfully merged the realms.

The PTR server is online with gear vendors - so jump on and have some fun in the meantime

Update 7 - 00:22 - June 7th

Merge process is currently 1177203 / 3069739 characters - 38.35%

Update 8 - 09:01 - June 7th

Merge process : 2179011 / 3069739 - 70.98%

Update 9 - 18:18 - June 7th

After the merge process is complete, additional post merge analysis and work still remains. This includes dumping the merged database, uploading it from my local machine to the server, importing it and then verifying and fixing potential issues. This might take anything between 2-5 hours. However due to the project being run by adults with jobs and other responsibilities this process is guaranteed to take longer than 2-5 hours. A rough ETA on the server being back online is between 6-12 hours from now provided the merge was a success.

Update 10 - 23:43 - June 7th

There's nothing new, but people were asking for an update, so here's an update.

Post-merge analysis and cleanup is still ongoing.

Update #11

We have moved the partially, and nearly fully, merged database onto our server. This took around 30 minutes to upload from our development machine (which has much faster hard drives and therefore was ahead of the server-side process), and about 3 hours to import. The current database size is 50GB.

We are currently at 2933744 / 3069739 characters merged, or 95.56%, with approximately 135,000 remaining. The total number does NOT take into consideration any characters that may have been deleted due to accounts having more than 10 characters (which is actually quite a considerable amount). This is merely an indication of our current position, so in actuality we are closer than what it reflects.

Once all characters have been merged, we will be spending some time to validate the data and ensure consistency. Due to the time this entire process has already taken, we plan to take some shortcuts in our post-processing to bring the server back up in a timely manner. This includes some mail and auction data changes, which we will share with you once we arrive at that point.

I cannot really give you an ETA on the remaining time , but I do expect it to be operational in the next 12 hours or so, putting us at around a 48h window for the whole affair.

We thank you for your patience and understanding throughout this rather epic task we are undertaking, and apologize for the duration it has ended up extending to.

Update 12 - 15:05

Lightbringer is back online.