Hacker vandalised our buildbot and Github organization

Libretro Team

Approximately 5 hours ago, we were the target of a premeditated cybercrime attack on our key infrastructure.

The hacker did the following damage:

  • He accessed our buildbot server and crippled the nightly/stable buildbot services, and the netplay lobby service. Right now, the Core Updater won’t work. The websites for these have also been rendered inaccessible for the moment
  • He gained access to our Libretro organization on Github impersonating a very trusted member of the team and force-pushed a blank initial commit to a fair percentage of our repositories, effectively wiping them. He managed to do damage to 3 out of 9 pages of repositories. RetroArch and everything preceding it on page 3 has been left intact before his access got curtailed.

We are still awaiting any sort of response or support from Github. We hope they will be able to help us restore some of these vandalised Github repos to their proper state, and also to help us narrow down the attacker’s identity.

We wanted to clear up some confusion that may have arisen in the wake of this news breaking:

  • No cores or RetroArch installations should be considered compromised. The attacker simply wiped our buildbot server clean, there is nothing being distributed that could be considered malicious to your system. Nothing has happened here and there is no need for any concern.
  • For the current time being, the Core Installer is non-functional until further notice. The same goes for ‘Update Assets’, ‘Update Overlays’, ‘Update Shaders’.

The IP he was using while doing this was ‘54.167.104.253’, which seems to lead back to AWS.

We’re still assessing the situation but moving forward, we think that it’s probably best not to go forward with the buildbot server that was compromised earlier today. We had some long-term migration plans for a move to a new server, but this was always pushed back because we felt that we weren’t ready migration-wise. It might indeed be the case this is the catalyst for just starting all from scratch with a new server instead of trying to migrate the old one over. This would mean that the more commonplace builds for Linux/Windows/Android would be immediately available, but all the specialized systems like consoles, old MSVC builds and whatnot would have to wait for later until we have adapted this properly to the new system.

Lack of automated backups

This brings us onto another key issue – the lack of backups. We last performed a backup of our buildbot server about a couple of months ago. The truth is that while we pay a hefty amount for the servers on a monthly basis already, there is simply not enough money to pile on automated backups as well. We could really use your support on Patreon to help lighten our financial burden here, especially since this now-pretty-much-mandatory server switch will likely cost us an insubstantial amount of money upfront while we keep the current server running for a month longer.

How will we restore things

So, how are we going to restore things? We hope that Github will be able to restore the affected repositories. If they are unable to do so, we could rely on the goodwill of users to source us with git repositories with the full history intact.

As for the buildbot? No idea to be quite frank. If we make the switch to the new server, you’ll get Android/Windows/Linux up and running early again but all other platforms will have to be added as we go along.

It’s a shame what is happening to the emulation and homebrew community. When it isn’t developers leaving for greener pastures deciding it’s no longer worth it, prestigious developers like byuu are being forced to early retirement because of unsavory online gang-stalkers. In our situation, we can’t rule out the possibility that some of these attacks come from some of the same usual suspects (it isn’t the first time we’ve seen them abuse AWS for some of these attacks, we encountered them a year ago earlier targeting our lobby services). Whatever their aim may be, while they will not deter our will to continue working on this project, they have definitely increased our maintenance and cost burden for the time being. And for this we ask for your understanding and support as we attempt to come up with a plan to address these problems moving forward. Supporting us through Patreon is a great way of helping out, especially if we can reach the $1300 goal which means we can spend a bit more each month to make sure our stuff is properly backed up.

As if the complications with Android’s new store policies that requires us to coordinate with new contributors to come up with a workable solution was not enough of a headache, this comes along. With your help and support, we will overcome this and come out stronger than before.

Regarding the Android / Core Installer situation

While we’re on this subject briefly, while it’s off-topic, we felt the need to address this real quick. We will likely be making a version of RetroArch Android that is neutered ONLY for Google Play. It will mean that the Core Installer will not be available for this, and cores will come packaged in additional APKs that can be installed. Apparently there is a 50-core extra APK limit on this until it starts requiring a version of Android over version 8.0. So while trying not to artificially bump the Android OS system requirements, we’re deciding on a 50 core-APK limit for now. Hopefully we can fit nearly most of the cores within such narrow constraints.

On our download site (and on F-Droid), we will have a RetroArch Android version that will work as before – with the Core Installer feature completely left intact. We feel this is a much superior version to what will be available on the Play Store, but unfortunately Google will force our hand here.

RetroArch 1.9.0 won’t be releasing on Google Play Store for now, next version severely downgraded to comply with new policies

Libretro Team

We regret to inform you that RetroArch 1.9.0 will not be releasing on the Google Play Store for now.

RetroArch has been available on Android through the Google Play Store since 2012. We encountered two snags this week while trying to update RetroArch on Google Play. First, it complained that our APK size was too big, and that an APK could not be any bigger than 100MB. So we had to go back and start removing some shaders in order to get things to fit.

After this, we tried uploading again. This time, we hit another snag that we have not encountered before:

Issue: Violation of Malicious Behavior policy

An app distributed via Google Play may not modify, replace, or update itself using any method other than Google Play’s update mechanism. Likewise, an app may not download executable code (e.g. dex, JAR, .so files) from a source other than Google Play.

The way we interpret it, the former is not an issue since RetroArch cannot update itself. The latter is probably the issue. We take it that apparently now the Core Updater service is a point of contention, and that it has to go from RetroArch Android. We were not aware of this, and throughout the app’s near-decade presence on the Google Play Store, there has never been a problem before on this front.

Unfortunately, this means that we have to go back to the drawing board now and fundamentally re-engineer RetroArch for Android. However it will be re-tooled, it will unfortunately be a huge setback from an Enduser Experience for the user, there’s no way around that.

Some possibilities that might exist:

  1. We keep RetroArch dynamically linked, but each core has to be installed separately through the Google Play interface as installable DLC. NOTE: We have no idea if this even works the way it can with Steam, so it would have to be explored first.
  2. We make RetroArch statically linked and therefore there needs to be a separate new app store entry for every single combination of RetroArch with every single core.

Whichever of the two we choose, it will mean no more Core Updater. Given the interpretation of the rules, updating assets is probably still permissible as it would be pretty silly to block that, so things like updating shaders and overlays would likely still remain included.

We cannot stress how much of a pain in the ass it will be to have to retool RetroArch like this. It’s almost at the point where it’s almost not worth it from our perspective to do it like this and we feel tempted to just tell people to download it from our site instead, as we certainly have never made a single buck on the Google Play Store to begin with, so there’s no direct profit incentive there. We also don’t know if we even have the manpower right now to be able to make these fundamental changes, but we will certainly attempt to try.

For now, we recommend to users that want the ‘proper’ version to just go to our Downloads page and download RetroArch 1.9.0 from there on Android. You will be installing the APK directly on your phone. You might have to enable ‘Allow outside APKs to be installed’ or some similar setting on your phone for it to be able to be installed, but theoretically any Android phone should be able to install APKs outside of the Play Store.

Where to download RetroArch for Android for now

If you want the latest 1.9.0 version, you can either get it for now on F-Droid, or on our Downloads page here.

Right now on the Play Store, version 1.8.9 should still be available for now.

We apologize for the inconvenience and we hope we can offer a solution soon that is agreeable to both Google and that doesn’t cause us a huge maintenance burden either (although it assuredly will be). We also right now don’t really have a plan ready that will allow us to quickly move on this front, so we’ll just have to see how things go.