Our new Libretro Infrastructure is now going live! (Plus Dosbox Pure out for Android/Mac/Windows)

Libretro Team

Our team has pushed an inhuman amount of effort over the holidays into making sure we can finally make the big switch in our infrastructure that we have been building up to ever since the server hack this past September.

And while it nearly looked like we might not have anything to show for it before the New Year, looks like we finally have something ready for you guys!

Thanks to the increase in Patreon earnings, we have finally been able to ditch the old dusty buildbot infrastructure and move over to a completely modern, built-from-scratch infrastructure. How it works is that we have our own Gitlab server now that mirrors all the repositories on our Github organization. This server is considerably beefier and more expensive than anything we have had before, and is able to cut through the cores and RetroArch workload like butter.

Where can I download stuff?

Same place as before – buildbot.libretro.com.

What’s ready so far?

The following platforms are ready to go (and already have up-to-date nightly RetroArch builds, tho a build could be missing here and there):

  • Android (*)
  • Emscripten (Web Player)
  • macOS (64bit – cores only) (**)
  • Linux (64bit)
  • OpenDingux (for RG350/GCW Zero/others)
  • Nintendo 3DS
  • Nintendo GameCube
  • Nintendo Wii
  • Nintendo WiiU (***)
  • Nintendo Switch
  • Sony PlayStation2
  • Sony PlayStation Portable
  • Sony PlayStation Vita
  • Windows (32bit/64bit) (****)

* – See the paragraph ‘Special notes on Android builds’.

** – Nightly builds for macOS still have to be added.

*** – For the first time, the WiiU version has everything out of the box that you need. In the past, assets would not be pre-installed and it would be very tedious and cumbersome to have to install them to the right place. This is all taken care of for you now. As a consequence though, the nightlies are a lot bigger to download now.

**** – The Windows nightly builds right now do not include Nvidia Cg support (old deprecated standard for shaders) and there is no installer yet. We hope to work something out there soon.

What’s next?

There is stuff still remaining to be done that we hope to sort out over the coming days/weeks. This includes:

  • ARM Linux cores
  • MacOS (64bit – binaries)
  • MSVC 2003/2005/2010 cores
  • RetroArch for older Windows versions (requires MSVC2003/2005/2010)

There are also still some cores missing here and there. There are a couple of reasons for this. One of them is that we still need to setup some .gitlab-ci.yml files for these repositories. All of that will come shortly though we hope.

We are of course also going to be releasing a new stable soon – 1.9.1.

What will all this mean for the user?

  • Faster build times
  • More platforms supported
  • Faster stable releases

m4xw will later be writing an article that goes into technical details on our new server, but here is a layman’s person breakdown on what this all boils down to.

Faster build times as a result of being able to run cores and RetroArch builds in parallel now. The old infrastructure was nearly completely sequential and it made things much slower than it used to be. But of course this is only half the story – the server specs have also been doubled, explaining why things are considerably faster.

Thanks to a massive increase in storage space and a reduction in server load, we can also now support more platforms. With the old server, we were running against the limit pretty hard on how many more platforms we could add before we would run out of disk space yet again. We thankfully no longer have these problems.

The faster stable releases alone is a big one. With the old buildbot infrastructure, it would take over an entire day to push out new binaries for RetroArch. It imposed a huge amount of stress on the entire team as every single time we ran these scripts, something would invariably go wrong for whatever reason. We leave all that behind today, after having had to deal with this for years on end. What this means is that you’ll likely no longer have to see announcements on our Twitter saying something is bound to be released and then it takes another day or two for it to finally arrive.

Also, for us as developers, the new infrastructure is a dream to work with, greatly reduces our workload, it’s no longer just a collage of scripts but uses modern technologies. What it means is that we have to do less fighting with our infrastructure and have more time to spend on our actual project instead.

What will all this mean for the contributor?

In the past, if we wanted to add a new core to our buildbot, somebody would send a PR to the libretro-super repository to add this core to one of the ‘recipes’.

This is all officially obsolete as of now. None of this is used anymore. Instead, the way it works now is:

  • You create a .gitlab-ci.yml file in a core repository (random example here). It needs to be in the root directory of your repository. You specify all the platforms here that the core should be building for.
  • We then add this repository as a CI mirror to our Gitlab server.
  • If all goes well, it spits out builds and these will then be available for download on buildbot.libretro.com

Libretro-super is now no longer used as a key part of our infrastructure like it was before. That doesn’t mean it’s completely useless. It can still be useful for the regular user to build stuff for personal use, and we will continue to update it. But it is no longer a fundamental part of our infrastructure like before, more a user convenience tool for building cores.

So, in other words, you don’t need to send anymore recipe update PRs to libretro-super in order to add new cores to our buildbot. Instead, you add a .gitlab-ci.yml file, then you talk to us and we add your repo to the Gitlab server.

Special notes on Android builds

Read our dedicated news article here on the Android builds

The nightly build that can be downloaded on our site right now is the non-Play Store version. The non-Play Store version is the version of RetroArch you have used up until this point, with a Core Downloader that connects directly to our build infrastructure. Because we don’t have to care about uploading this APK to the Play Store anymore, it can be a tad bigger than 100MB and therefore contain all the assets that we previously had to strip to save on space.

The Play Store version will be rolled out very soon and there are some big changes that were made in order to conform to the Google Play Store’s new guidelines. Some of it we think you will love while there are also some caveats.

The Play Store version’s Core Downloader contain a curated selection of cores hosted by Google’s servers (due to Play Store’s policies). More on that in the article. The user of course will always have the option of grabbing the non-Play Store build on our website which still has the Core Downloader that connects to our build infrastructure.

Special thanks

Special thanks to all the people that helped participate in this tremendous server migration: m4xw, jdgleaver, gblues, farmerbb, Steel01, fjtrujy, frangarcj, Xer Shadow Tail, and any others that participated. We would not have been able to make it without you!

And above all else, a huge shoutout to our Patrons who helped bring all this into reality! Without you guys, there is no way any of this could have been done, and we’d still be stuck with a slowly decaying infrastructure that was just impossible to salvage! A huge thank you, and with your monthly support the Libretro/RetroArch project will go into 2021 stronger than ever before!

Oh, and one last thing…

Dosbox Pure out now on the buildbot for Android/Mac/Windows!

We have just added Dosbox Pure for Windows, Mac (x64) and Android to our buildbot. Grab it from the Core Downloader as usual, and have fun! If you’d like to learn more about Dosbox Pure, read our dedicated article here.

Hacker vandalised our buildbot and Github organization

Libretro Team

Approximately 5 hours ago, we were the target of a premeditated cybercrime attack on our key infrastructure.

The hacker did the following damage:

  • He accessed our buildbot server and crippled the nightly/stable buildbot services, and the netplay lobby service. Right now, the Core Updater won’t work. The websites for these have also been rendered inaccessible for the moment
  • He gained access to our Libretro organization on Github impersonating a very trusted member of the team and force-pushed a blank initial commit to a fair percentage of our repositories, effectively wiping them. He managed to do damage to 3 out of 9 pages of repositories. RetroArch and everything preceding it on page 3 has been left intact before his access got curtailed.

We are still awaiting any sort of response or support from Github. We hope they will be able to help us restore some of these vandalised Github repos to their proper state, and also to help us narrow down the attacker’s identity.

We wanted to clear up some confusion that may have arisen in the wake of this news breaking:

  • No cores or RetroArch installations should be considered compromised. The attacker simply wiped our buildbot server clean, there is nothing being distributed that could be considered malicious to your system. Nothing has happened here and there is no need for any concern.
  • For the current time being, the Core Installer is non-functional until further notice. The same goes for ‘Update Assets’, ‘Update Overlays’, ‘Update Shaders’.

The IP he was using while doing this was ‘54.167.104.253’, which seems to lead back to AWS.

We’re still assessing the situation but moving forward, we think that it’s probably best not to go forward with the buildbot server that was compromised earlier today. We had some long-term migration plans for a move to a new server, but this was always pushed back because we felt that we weren’t ready migration-wise. It might indeed be the case this is the catalyst for just starting all from scratch with a new server instead of trying to migrate the old one over. This would mean that the more commonplace builds for Linux/Windows/Android would be immediately available, but all the specialized systems like consoles, old MSVC builds and whatnot would have to wait for later until we have adapted this properly to the new system.

Lack of automated backups

This brings us onto another key issue – the lack of backups. We last performed a backup of our buildbot server about a couple of months ago. The truth is that while we pay a hefty amount for the servers on a monthly basis already, there is simply not enough money to pile on automated backups as well. We could really use your support on Patreon to help lighten our financial burden here, especially since this now-pretty-much-mandatory server switch will likely cost us an insubstantial amount of money upfront while we keep the current server running for a month longer.

How will we restore things

So, how are we going to restore things? We hope that Github will be able to restore the affected repositories. If they are unable to do so, we could rely on the goodwill of users to source us with git repositories with the full history intact.

As for the buildbot? No idea to be quite frank. If we make the switch to the new server, you’ll get Android/Windows/Linux up and running early again but all other platforms will have to be added as we go along.

It’s a shame what is happening to the emulation and homebrew community. When it isn’t developers leaving for greener pastures deciding it’s no longer worth it, prestigious developers like byuu are being forced to early retirement because of unsavory online gang-stalkers. In our situation, we can’t rule out the possibility that some of these attacks come from some of the same usual suspects (it isn’t the first time we’ve seen them abuse AWS for some of these attacks, we encountered them a year ago earlier targeting our lobby services). Whatever their aim may be, while they will not deter our will to continue working on this project, they have definitely increased our maintenance and cost burden for the time being. And for this we ask for your understanding and support as we attempt to come up with a plan to address these problems moving forward. Supporting us through Patreon is a great way of helping out, especially if we can reach the $1300 goal which means we can spend a bit more each month to make sure our stuff is properly backed up.

As if the complications with Android’s new store policies that requires us to coordinate with new contributors to come up with a workable solution was not enough of a headache, this comes along. With your help and support, we will overcome this and come out stronger than before.

Regarding the Android / Core Installer situation

While we’re on this subject briefly, while it’s off-topic, we felt the need to address this real quick. We will likely be making a version of RetroArch Android that is neutered ONLY for Google Play. It will mean that the Core Installer will not be available for this, and cores will come packaged in additional APKs that can be installed. Apparently there is a 50-core extra APK limit on this until it starts requiring a version of Android over version 8.0. So while trying not to artificially bump the Android OS system requirements, we’re deciding on a 50 core-APK limit for now. Hopefully we can fit nearly most of the cores within such narrow constraints.

On our download site (and on F-Droid), we will have a RetroArch Android version that will work as before – with the Core Installer feature completely left intact. We feel this is a much superior version to what will be available on the Play Store, but unfortunately Google will force our hand here.