RetroArch 1.7.8 – AI Service – How To Set It Up

Libretro Team

Welcome to the future! Sometime ago, a #RetroArch bounty got posted proposing OCR (Optical Character Recognition) and Text To Speech services being added to RetroArch.

Some months later, and here we are – a bounty hunter valiantly took on the challenge and there is now a fully fledged AI Service up and running that works seamlessly with #RetroArch!

You use the AI Service like this – you enable the AI Service (should be enabled by default), you then setup the server URL (could be a local network address if you have the server up and running in your own network, or a public IP/URL in case you’re going through a service). After that, you only need to bind a button or key to the so-called “AI Service” action. You can bind this key by going to Settings – Input – Hotkeys.

In this video, you can see each of the two modes that the AI Service currently is capable of doing –

Speech Mode – Upon pressing the AI Service button, a quick scan is done of the text, and the recognized text is then translated to speech. You can press the AI Service button at any time and it will try to process the current snapshot of the screen it made. This mode is non-interruptable, meaning the game will continue running when you hit this button, and the output speech will take as long as it takes for the server to respond to your query and pipe the sound to RetroArch.

Image Mode – In image mode, it tries to replace the text onscreen with the output text. For instance, in the video you see above, the game is played in Japanese, so when we hit the AI Service button, it tries to replace the Japanese text with English translated text. This mode is interruptable – this means that when you hit the AI Service button, it pauses the game and shows you an image with the replacement text UNTIL you hit either the AI Service hotkey or the Pause hotkey again, then it will continue playing.

We encourage everybody that wants to submit feedback to us on this amazing revolutionary feature to go to our Discord channel and in specific the #retroarch-ai channel. We’d love to hear your feedback and we’d like to develop this feature further, so your input and feedback is not only appreciated but necessary!

VGTranslate using Google API keys

1.0 Get a Google API key

  • Create a billing account (taken from https://cloud.google.com/billing/docs/how-to/manage-billing-account )
    • Go to the Google Cloud Platform Console Manage billing accounts page and sign in or, if you don’t already have an account, sign up.
    • Click the Create account button.
    • Enter the name of the billing account, then click Continue. Note that if you see an Organization  drop-down, then you must also select an Organization before you can continue.
    • You might be prompted to select your country. The choice will affect the payment options you have in  the next step. The currency for your billing account is based on the country you select.
    • Choose the payments profile that will be associated with this billing account. You can choose an  existing payments profile, or create a new payments profile. If you choose to create, follow the  instructions on the screen to set up your payments profile.
    • Set your Account type as individual
  • Create an API Key: (taken from https://cloud.google.com/docs/authentication/api-keys )
    1. Navigate to the APIs & Services Credentials panel in GCP Console.
    2. Select Create credentials, then select API key from the dropdown menu.
    3. Click the Create button. The API key created dialog box displays your newly created key.
    4. This is your GOOGLE_API_KEY that we will use later.

2.0 Setup VGTranslate

  • Windows:
  • Linux:
    • Download the vgtranslate project from https://gitlab.com/spherebeaker/vgtranslate.git or use git clone.
    • Go to the extracted vgtranslate directory where setup.py is located.
    • Using python2.7, run: python setup.py install -If this step fails, you can try removing the problematic dependency.  Dependencies like gender-guess and kivy are not needed for most functionality. -Change the default_config.json file as in the windows case.
    • Go into the vgtranslate subdirectory and run: python serve.py

3.0 Setup RetroArch

AI Service Settings
  • Start RetroArch -Settings->Input->HotKey Binds->AI Service
  • Change to key/button of your choice
  • Settings->AI Service -Set AI Service Output to “Image Mode” (text translation) –
    OR set AI Service Output to “Speech mode” (text-to-speech)
  • Set AI Service URL to “http://localhost:4404”
  • Set AI Service Enabled to on.
  • Set Target Language to your language of choice
  • Source language will be used as a hint for the OCR, but can be left to “Don’t Care”
  • Run a game of your choice
    1. Image Mode (Text translation)
      1. Press your hotkey to pause the game, and wait for a translation to come back.
      2. Press your hotkey again to unpause and continue
    2. Speech Mode (Text To Speech)
      1. Press your hotkey to read out the text currently on the screen.
Bind the AI Service hotkey to a key or button
Bind the AI Service hotkey to a key or button

4.0 ZTranslate Service API Option

ZTranslate API Key method

  • Go to https://ztranslate.net/signup
  • Enter a display name, email, and password, and click register
  • Check your email for the verification link and click on it.  If you don’t see it, check your junk/spam folder.
  • In the top menu, click on “Settings”
  • Copy the API KEY near the bottom of the page.  This is your ZTRANSLATE_API_KEY.
  • Follow instructions in 3.0, but
    • For AI Service URL, put in: “http://ztranslate.net/service?api_key=<ZTRANSLATE_API_KEY HERE>”

5.0 – ZTranslate API via VGTranslate

Follow instructions 2.0, 3.0, and 4.0, except:

  • when modifying default_config.json, use the following: (link here)
  • Change AI Service URL to “http://localhost:4404”.
  • This will use the ztranslate API method, but will reduce latency compared to hitting the service API directly.

RetroArch 1.7.8 (v3) released + big core updates! (bsnes HD, Flycast, Android, etc)

Libretro Team

RetroArch 1.7.8 was a very ambitious release, and as a result, it is taking some time to iron out some of the kinks. Instaed of leaving you waiting for a month again to fix some crucial bugs, we’d rather release these point fix releases first instead so that we leave you with a rock-solid 1.7.8 in the end while we then shift our focus and attention to 1.7.9.

In light of that, we are releasing version 3 right now, which will be especially beneficial for Mac users. The future-proof Metal Mac version should now work flawlessly on Macs with an AMD graphics card (they previously produced heavy graphics glitches inside the menu). It has just been released! Grab it here.

If you’d like to show your support, consider donating to us. Check here in order to learn more.

For all other details surrounding version 1.7.8, we refer you to our original article here.

Changes

  • GLCORE: Ensure correct scaling of menu texture (with RGUI)
  • IPS: Soft-Patch any IPS size
  • METAL: Fix overlay issue – setup correct viewport before rendering overlay
  • METAL/STB: Fix font driver issue with AMD GPUs on MacOS.
  • MENU/RGUI: Correctly rescale menu when resizing window if aspect ratio lock is enabled.
  • OSX: Remove OSX suffix in window title
  • PSP: Fix audio conversion code
  • REMAPS: Fix analog remapping regression -analog remapping would break controls

bsnes HD – Released for Windows/Linux/Android, and soon iOS and Mac!

bsnes HD should now be available on Linux, Windows and Android for RetroArch users! It’s based on the latest version of bsnes, and it should be significantly faster than previous bsnes versions.

In this video, we show you some of the HD Mode 7 features that are unique to this version. They make for a fairly significant difference overall as we’re sure you’ll agree!

This core is not fully complete yet and might still have some omissions. Also, the ‘bsnes HD’ name is temporary, and we will be doing some house cleaning of the various bsnes cores we are maintaining soon. The plan is to have an improved Core Updater in later RetroArch versions that allows for better categorization and filtering in the future so that users can more easily manage their cores.

On Android for the first time!

This is the first time the latest version of bsnes will appear on Android, courtesy of Libretro/RetroArch! Our core version of this is called bsnes HD, and you can grab it from the Core Updater right now! Just make sure to update the core info files first (by going to Online Updater, then selecting ‘Update Core Info Files’). bsnes HD should be a fair bit faster than the other bsnes cores already available, plus it has enhanced overclocking features and the acclaimed HD Mode 7 features.

In this video you see it running on a Samsung Galaxy S10+ (Exynos model) with 3x HD Mode 7 scaling applied, and it runs at fullspeed all the way.

User and contributor harakari has reported that he can run HD Mode 7 at 4x scaling and still have games run at fullspeed on his iPhone XS Max, so if anything, expect even better performance on high-end iDevices!

Flycast – WinCE core now 30% faster on average and sound improvements!

The FlycastWinCE version should now be 30% faster on average for non-Windows CE games! We have figured out a way to have the Windows CE code additions no longer affect the main performance of the emulator. For that reason, after we have ironed out some of the final kinks, you can expect there to be only one Flycast core moving forward. On top of that, flyinghead has really gone to town with some much needed audio improvements on the AICA and DSP side. Witness this long standing audio sample bug that is now finally fixed with the game Resident Evil: Code Veronica. This and many other sound bugs (such as the audio samples continuing to be repeated during the battle loading scenes in Soul Calibur) have been fixed now.

World-first – Windows CE Dreamcast games running on Android!

The Flycast WinCE core is now available for Android users!

  • 30% speedup in non-Windows CE games thanks to extensive optimizations made to the dynarec by flyinghead. No more performance reduction of non-Windows CE games.
  • Windows CE support further improved.

It is now possible to play Dreamcast Windows CE games on Android! Please be aware that this is very CPU intensive and that you should probably expect 15 to 25fps on high-end Android phones right now.

Note that Windows CE games could still be unstable on Android, and that you need a real BIOS for Windows CE to work. It won’t work with the HLE BIOS.

NOTE: We anticipate that after the final kinks has been ironed out, that we will merge the Windows CE parts of this core back into Flycast, and that the separate Flycast WinCE core will disappear from there on. So this separate core is only a temporary thing for now. We will let you know when this will happen. Video was recorded on a Galaxy S10+.

Mupen64 Plus Next – No more 10 second startup times on Windows!

Thanks to an important bug fixed by mudlord, Mupen64 Plus Next should no longer take up to 10 seconds to start up any game on Windows.

Mupen64 Plus Next is an up to date version of Mupen 64 Plus with the latest GlideN64 renderer.

More progress reports on other cores soon

Cores are often updated on a daily basis, and a lot of the times, all the amazing enhancements and improvements they receive go underreported because we’re so busy with development. We feel it is time to shine more of a light on these changes, so we will be doing more periodic updates on core updates as they come along.