Friday, June 26, 2020

Quick Update

For folks with apps that still use the http://lwXXX.ultraseedbox.com:PORT URL format, please note that this needs to be manually changed to http://lwXXX.usbx.me:PORT in order to connect.

This will be updated in UCP as soon as we can.

Thursday, June 25, 2020 - Update #2

Website and UCP issues should now be resolved.

ruTorrent issues should be resolved now, as well. Please use the following steps to resolve:

  1. Reinstall web server in your UCP dashboard.
  2. Restart rTorrent torrent client.
  3. Test and close ticket if you have one regarding this (It will help a lot!)

Deluge installer is also fixed (please close your ticket if you have one for this issue too).

Thank you guys, much love!

TODO:

  • OpenVPN to be deployed.
  • AutoDL-Irssi and Transmission application installer errors to be fixed.
Thursday, June 25, 2020 - Update #1

We wanted to give you all an update regarding our current situation. Over the last few days, we’ve done our best to restore services to all you fine folks but someone has decided to kick us while we’re down and issue a DDoS attack on ultraseedbox.com. This has certainly made headaches worse. But please understand that we are doing the best we can to deal with this whole ordeal.

To serve you all better, we are going to make some changes:

  • We are going to communicate over Twitter so that everyone can receive updates. And expand our communication ways.
  • We have decided to increase the compensation from 7 days to 15 days.
  • The compensation will include a traffic reset for all clients.

What are we working on?

  • We are currently working on the website, Control Panel and WHMCS to bring them up and make them more robust. They should be available soon.
  • Once UCP is up, Communication system between UCP and Servers will be initiated.
  • We will fix the application with the UCP that is throwing errors.

We would like to say that please do not believe in any rumors or speculations. Rumors can’t be avoided but they can be shut down. We can assure you that no customer information has been compromised. In its shortest possible form, this is what happened: A vulnerability was discovered and it necessitated immediate action in the interest of protecting user data, which included taking all of our servers offline to reinstall and reconfigure the OS, implement a patch for the vulnerability and then restore services. Once we’re able to get everything back to normal operation, we will share the details.

Please know that our mail system is currently down so there are limited ways we can communicate with you. Discord, Twitter, Website status (when working), Reddit. We will keep you updated in the very best way we can.

PS: All the compensation changes will be made right after everything is back and customers are back to reliable service. We will need to script it to implement it.

Twitter: https://twitter.com/ultraseedbox
Reddit: https://www.reddit.com/r/ultraseedbox
Status: https://ultraseedbox.com/status
Discord: https://discord.gg/YXGYS9D

Tuesday, June 23, 2020 - Update #2

All servers are coming back online and are now accessible. We would advise to restart their apps and the webserver via the UCP. As we expect an influx of users, some of our core servers will be overwhelmed. Installation/upgrade of apps will resume normally once activity goes down.

SSH/SFTP/FTP password will also need to be changed in your UCP due to the Operating System refresh. You can change this in your access detail section of your UCP. You may be greeted by a known_hosts/ssh key error within your terminal. Please try the following command to resolve it:

ssh-keygen -f "/homeXX/user/.ssh/known_hosts" -R lwXX.usbx.me

For those clients who is still having problems with their slots that are not listed above, you may have to send a ticket so we can help.

More announcements will come from us in the next few days, including a post-mortem of the incident.

As announced earlier, we will add 7 days from your original due date and your allocated bandwidth will reset to all current users as part of the compensation of the downtime.

It cannot be said enough, thank you so much for your patience and understanding. Have a great day.

Tuesday, June 23, 2020 - Update #1

This is our last quick update of the downtime.

We just managed to get all servers online and are in the process of final checks and redeployment, but these will still be partially accessible.

As previously stated, during this time SSH, FTP, systemd (hence rclone mounts not being run), torrent/usenet clients and the webserver are currently inaccessible. Dockerized apps, however, are running. You may access the dockerized apps via the direct URL of your slot (http://lwxxx.usbx.me/:PORT) or if your slot happens to have Plex installed and you store your media locally, you may access your Plex instance via https://app.plex.tv/ in the meantime.

As we are in the final steps, we would humbly request an additional 12 hours for us to properly redeploy our servers. Afterwards, a post-mortem of the incident will be followed. All current users will be compensated for the downtime.

As always, we thank you for being supportive and we humbly apologize for the downtime.

Monday, June 22, 2020 - Update #2

A quick update on the downtime.

95% of the NovoServe servers are now online and we are working on getting all YISP servers. We are continuing the process of checking each server and covering all bases. After a couple of reboots and finals checks, all servers will be fully online.

During this time, the servers are partially accessible. SSH, FTP, systemd (hence rclone mounts not being run), torrent/usenet clients and the webserver are currently inaccessible. Dockerized apps, however, are running but will be inaccessible due to the webserver being offline.

If your slot happens to have Plex installed and you store your media locally, you may access your Plex instance via https://app.plex.tv/ in the meantime.

Monday, June 22, 2020 - Update #1

We are slowly implementing the newly patched image to our servers and getting it all up. For this to deploy properly, we will need up to 24 hours as we slowly bring the servers up. Certain apps will be inaccessible in the meantime, which will be dealt on later. Once that is done and we are fully online, expect another announcement from us.

We are pleased to report that user data is fully intact. WHMCS (Client Area Portal) is also safe but precautionary steps will be done to mitigate any risks.

We sincerely apologize for this sudden downtime and we thank you for your patience and understanding through all of this.

Sunday, June 21, 2020
Vulnerability has been patched. Scripts are done and being tested currently. We will start to bring the servers online gradually.
Saturday, June 20, 2020
All servers are currently offline for some urgent maintenance to patch some newly discovered and potentially serious vulnerabilities across all servers. Expected downtime will be 1-3 days and all current clients will be fairly compensated for the downtime. We apologize for the inconvenience.