Ability to save files?

[timdream]

I tried to figure out a way to make the saved games persistent. After extensive documentation searching, I realized the root filesystem is always MEMFS mounted from the data. Would it be possible to merge/remount(?) the root with IDBFS? If is possible with emscripten it should be configured in this repo right?

The alternatives I could think of is to create custom shell.html for this project and point emcc to it, and try to talk to FS API from JavaScript. A nicer em-dosbox UI can be created as a bonus with this approach. I would love to contribute to that as well :)

[dreamlayers]

I've thought about this before but never got around to implementing anything. For now I'm focusing on making more programs usable, fixing problems and improving performance.

I was thinking I should implement something like UnionFS with a read-only MEMFS containing the game, overlaid with an IDBFS where the writes go. Writes to any file would cause that file to be copied to the IDBFS, and newly created files would go into IDBFS. Then syncing of the IDBFS needs to be managed. It's important to ensure that saved files are truly saved even if the emulator crashes later or the user closes the tab soon after saving.

The idea of doing stuff from JS outside of the compiled DOSBox C++ code is interesting. That might make the asynchronous nature of syncfs easier to deal with, and it might help ensure files get saved regardless of what happens to the emulator afterwards. Probably the best argument for this approach is that saving games is a general issue with emulation, and not something just related to DOSBox. It would be good to make that code reusable. It seems Archive.org people are working on a redistributable loader. It may make most sense to implement this feature as a part of that.

[timdream]

Thanks for the comment!

Having a UnionFS-like file system for Emscripten is indeed the proper upstream fix to our problem here, for the sake of reusability etc. I will look into it when I am available for that. Since I am not experienced in C/C++, it would probably be contributed to Emscripten as a JavaScript-implemented FS callable by the native program.

We would figure out when the UI should do syncfs in em-DOSBox afterwards -- maybe it's possible it does not have to.

Could you point me to the thing archive.org people is working on? It make sense to join in their effort if it's already started.

[db48x]

The archive.org loader is available at https://github.com/db48x/emdosbox-loader; you can see how it uses BrowserFS (https://github.com/jvilk/BrowserFS) to unpack the zip file. (BrowserFS has a ZipFS that only decompresses files when they're actually read, but it doesn't offer write access. A lot of programs assume that they can write to the disk and break when they can't so we copy this to an InMemoryFS and then run the game from that).

There are a couple of possibilities:

easiest: Persist the existing InMemoryFS to localstorage, the cloud, etc. Load the game from it instead of the zip on the next load. Of course, this means persisting not just your game save but the entire game as well.

easy: Make an in-memory filesystem and mount it as the a: drive, and instruct the user to save games to it. Persist this filesystem in localstorage or wherever and restore it on the next load. The downside is that not all games let you choose where your saves go.

moderately hard: write a UnionFS-type backend for BrowserFS, and let writes go to an InMemoryFS while reads come from the ZipFS or the InMemoryFS, as necessary. Persist the InMemoryFS to localstorage or wherever.

harder: Copy-on-write! Persist as desired.

You're welcome to help out with whatever strategy you're most interested in, and to join us in #jsmess on EFNet to chat.

[DerKoun]

I quickly hacked together a hybrid of the local storage and in-memory file systems. It should persist only changed and new files. It shouldn't persist deletions of files that existed in the in-memory at the beginning, but that should be OK for games. As the in-memory file system has to be filled with the files form the zip first, the local storage persistence has to be activated afterwards, via the "enableLs" method.

However, I've never written TypeScript before, so there is one thing I didn't know how to do syntactically, marked with a FIXME comment. Maybe someone who knows TypeScript or even has BrowserFS build-ready can finish and verify the code and test if it works. Thanks in advance.

import buffer = require('../core/buffer');
import browserfs = require('../core/browserfs');
import kvfs = require('../generic/key_value_filesystem');
import api_error = require('../core/api_error');
import global = require('../core/global');

var Buffer = buffer.Buffer,
  ApiError = api_error.ApiError,
  ErrorCode = api_error.ErrorCode;

// Some versions of FF and all versions of IE do not support the full range of
// 16-bit numbers encoded as characters, as they enforce UTF-16 restrictions.
// http://stackoverflow.com/questions/11170716/are-there-any-characters-that-are-not-allowed-in-localstorage/11173673#11173673
var supportsBinaryString: boolean = false,
  binaryEncoding: string;
try {
  global.localStorage.setItem("__test__", String.fromCharCode(0xD800));
  supportsBinaryString = global.localStorage.getItem("__test__") === String.fromCharCode(0xD800);
} catch (e) {
  // IE throws an exception.
  supportsBinaryString = false;
}
binaryEncoding = supportsBinaryString ? 'binary_string' : 'binary_string_ie';

var/private/? lsEnabled = false;  //FIXME: this should be a member that can be changed be the method one line below and cam be read from the inner class ImLsHybridStore
public enableLs(): void { lsEnabled = true; }
//TODO: in addition to enabling localstorage generally, a white/black-list or regex of file-paths could be used to e.g. avoid persisting temp files

/**
 * A synchronous key-value store backed by localStorage.
 */
export class ImLsHybridStore implements kvfs.SyncKeyValueStore, kvfs.SimpleSyncStore {
  constructor() { }

  private store: { [key: string]: NodeBuffer } = {};

  public name(): string {
    return 'ImLsHybrid';
  }

  public clear(): void {
    this.store = {};
    if (lsEnabled) {
      global.localStorage.clear();
    }
  }

  public beginTransaction(type: string): kvfs.SyncKeyValueRWTransaction {
    // No need to differentiate.
    return new kvfs.SimpleSyncRWTransaction(this);
  }

  public get(key: string): NodeBuffer {
    if (lsEnabled) {
      try {
        var data = global.localStorage.getItem(key);
        if (data !== null) {
          return new Buffer(data, binaryEncoding);
        }
      } catch (e) {
      }
    }
    // Key doesn't exist in localstorage, or a failure occurred; get from in-memory
    return this.store[key];
  }

  public put(key: string, data: NodeBuffer, overwrite: boolean): boolean {
    try {
      if (!overwrite && ((lsEnabled && global.localStorage.getItem(key) !== null)
                        || this.store.hasOwnProperty(key))) {
        // Don't want to overwrite the key!
        return false;
      }
      if (lsEnabled) {
        global.localStorage.setItem(key, data.toString(binaryEncoding));
      } else {
        this.store[key] = data;
      }
      return true;
    } catch (e) {
      throw new ApiError(ErrorCode.ENOSPC, "LocalStorage is full.");
    }
  }

  public delete(key: string): void {
    delete this.store[key];
    if (lsEnabled) {
      try {
        global.localStorage.removeItem(key);
      } catch (e) {
        throw new ApiError(ErrorCode.EIO, "Unable to delete key " + key + ": " + e);
      }
    }
  }
}

/**
 * A synchronous file system backed by an in-memory/localStorage hybrid. Connects our
 * ImLsHybridStore to our SyncKeyValueFileSystem.
 */
export class ImLsHybridFileSystem extends kvfs.SyncKeyValueFileSystem {
  constructor() { super({ store: new ImLsHybridStore() }); }
  public static isAvailable(): boolean {
    return typeof global.localStorage !== 'undefined';
  }
}

browserfs.registerFileSystem('ImLsHybrid', ImLsHybridFileSystem);

[timdream]

So here is my take:

https://github.com/timdream/emscripten/compare/kripken:incoming...timdream:save-to-idb?expand=1

This branch contains one commit with a few files changes to Emscripten that

• Start the root filesystem with IDBFS instead of MEMFS.
• Patch the file_packager.py so that it would do the initial sync before starting the game.
• Patch the IDBFS to introduce a "blacklist" -- list of preloaded files that shouldn't be saved into IndexedDB unless changed.
• Add a button simply hook to FS.syncfs(), when the user click on it files gets synced into IndexedDB.

It's not good enough for a pull request yet. I would like to find out if I could remove that button and simply sync the file system whenever there is a write file operation. And of course wrap the changes in build switches too.

Is this the better approach or this assumes too many things from the DOSBox game use case?

[timdream]

Just added a commit with switches. em-dosbox should use the following to turn on these options:

diff --git a/configure.ac b/configure.ac
index 74b04f6..3f2114e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -20,7 +20,7 @@ dnl AC_CANONICAL_HOST fails to detect Emscripten.
 dnl This contains kludges to make things work.
 AS_IF([test "x$enable_emscripten" = "xyes"],
       [host_alias=none
-       : ${CXXFLAGS="-O3"}
+       : ${CXXFLAGS="-O3 -s ROOT_IDBFS=1"}
        : ${CFLAGS="-O3"}
        ac_cv_exeext=.html])

diff --git a/src/packager.py b/src/packager.py
index 7393303..3d49990 100755
--- a/src/packager.py
+++ b/src/packager.py
@@ -75,6 +75,7 @@ def run_packager():
                                                     "file_packager.py"),
                                        datafile,
                                        "--no-heap-copy",
+                                       "--syncfs-before-start",
                                        "--preload",
                                        PACKAGE_ARG])
     except:

[dreamlayers]

It seems BrowserFS is the best place for this. I created a branch based on #4 (comment) by @DerKoun. I got it to compile by putting enableLs() in ImLsHybridStore and making it accessible via ImLsHybridFileSystem. Right now after calling enableLs() all I see is what's in local storage.

Whatever is used for saving files should not require whitelists or blacklists. Support for them is good to have so you can avoid saving temporary files with some programs, but having to create lists for every program in a large collection would be too much work. You can use lists during development, but the plan should be to stop requiring them eventually.

Consider that sites may host multiple programs. Their files should be separate.

[dreamlayers]

I don't think that #4 (comment) by @DerKoun can be used. Directories are the problem. It must be possible to see content from both file systems in the same directory. Code on that level doesn't know about directories, and BrowserFS uses random IDs so you can't have persistent directories pointing to InMemoryFS files. I see no reasonable solution on that level.

It seems best to work on the filesystem level. Here's what I've done so far. It's at a very early stage, but this does actually work with Em-DOSBox. I could create and change files, with them persisting in local storage. This function sets it up in preRun:

function setupBFS() {
  Module['addRunDependency']('filesystem');
  var request = new XMLHttpRequest();
  request.open('GET', 'Gwbasic.zip', true);
  request.responseType = 'arraybuffer';
  request.onreadystatechange = function() {
  console.log(request.status);
  if (request.status == 200 && request.response) {
  var Buffer = BrowserFS.BFSRequire('buffer').Buffer;
  var zipfs = new BrowserFS.FileSystem.ZipFS(new Buffer(request.response)),
      lsfs = new BrowserFS.FileSystem.LocalStorage();
  var ufs = new BrowserFS.FileSystem.UnionFS(zipfs, lsfs, 'prefixdir/');
  BrowserFS.initialize(ufs);
  // Mount the file system into Emscripten.
  var BFS = new BrowserFS.EmscriptenFS();
  FS.mkdir('/dosprogram');
  FS.mount(BFS, {root: '/'}, '/dosprogram');
  Module['removeRunDependency']('filesystem');
  }
  }
  request.send(null);
}

[free5lot]

As far as I can judge, the ability to save files (state of the game) should be considered as number 1 priority.
Because almost all games can't be finished with a single session.

[dreamlayers]

I understand the importance. I have added more commits to https://github.com/dreamlayers/BrowserFS/tree/unionfs, enabling copy on write, handling directories, adding prefixes which can keep pages separate in a shared file system. I have run Doom, saved and restored successfully. Not all operations are supported yet, but this is usable. It needs testing though. I strongly suggest not depending on this for valuable saved games and not encouraging anyone to depend on it yet.

I think a method for saving also needs to allow users to access their information for backing up and maybe even loading files. The information may be stored in their browsers, but it's not in a form that they can easily access.

Note that browsers may clear local storage using the policy for clearing cookies. You can lose all saved files that way.

[santeriv]

Has somebody tried already sqlite https://github.com/kripken/sql.js ?

[timdream]

Allow me to update what I have for now. So with great help from kripken, I managed to implement two features into Emscripten on top of IDBFS:

• Call syncfs() on IDBFS when the C fsync() is called emscripten-core/emscripten#2492 which will pause the runtime and do syncfs when fsync is called from C. (only possible with EMTERPRETIFY=1 and EMTERPRETIFY_ASYNC=1)
• Schedule a syncfs() call on fclose() and/or other disk operations emscripten-core/emscripten#3453 which will trigger deferred and batched syncfs on file changes (creation/renaming/deleting/closing). The feature is implemented behind a flag and to be merged.

So once these two features are in the Emscripten tree, em-dosbox should be able to save files just by using IDBFS. Of course, that might not be optimal if we want people to sync/share saved files, but that's not the feature I was looking at when I submit this issue.

The only tricky part here is how to deal with games that does not allow you to select directory to save files. The game I am playing with as a reference save the files as 1.RPG/2.RPG and so on the game dictionary directly. I have an WIP to implement yet another feature into Emscripten IDBFS that could skip preloaded game program files when doing syncfs, but I am not entire sure if it make sense to for Emscripten. I will talk to kripken if people agree these are the right solutions to this issue.

In sum, I would like fix this issue by introducing 3 features into Emscripten IDBFS and submit a patch to em-dosbox build config to turn them on (and also put the game running directory into a specific one backed by IDBFS instead of the root of the virtual filesystem).

[timdream]

@dreamlayers I did look at your work on BrowserFS and I think it's a great progress! However, I choose to continue to work with Emscripten IDBFS because re-implementing an Emscripten filesystem is a non-trivial work. This can be evidenced by your comment on the fact "not all operations are supported yet." and also by the fact new BrowserFS.EmscriptenFS(); only works with localStorage backend per README on the project.

IndexedDB is also finally more ubiquitous than before. There is less need to support different storage APIs in browsers, and primary use of em-dosbox is in the browser.

Again, I do agree with Emscripten IDBFS we will be missing the cloud/sharing use cases. I think we could still leverage BrowserFS for that by hooking it up with IDBFS. Or we could leverage the patches already going into Emscripten and implement another syncfs-based filesystem in BrowserFS. That should allow us to use the async backends.

(If you look at library_idbfs.js you would realize IDBFS is just a MEMFS with syncfs handling -- we could do that too!)

[timdream]

I would also need to point out what I am working on is almost only referring to playing a game on a page generated by packager.py here. To be comparable with a BrowserFS filesystem setup like what db48x/emdosbox-loader did,

• The BFS exposed to Emscripten should be something similar to IDBFS (leveraging MEMFS and it's call to syncfs, as implemented) so that we could use a async backend there.
• The new async backend should be a UnionFS backed by a Zip and IndexedDB

I can move on and contribute to BrowserFS once patches to Emscripten are done.

[iakshay]

Any update on this?

Also, is there an API to read/write Emscripten file system?

[timdream]

@iakshay Not much progress since the last comments. You could check the linked issues to find my WIPs.

[andres-asm]

I implemented browserfs with the dropbox backend in our project:
https://github.com/libretro/RetroArch/tree/master/pkg/emscripten

It may be of use to you. It basically mounts dropbox as an async filesystem, syncs the data and then starts emscripten.

[free5lot]

Not sure how, but games at archive.org save state including saves to files in dos games, if it's any help.

[db48x]

For the games at archive.org I used BrowserFS to save the filesystem state to Indexed DB: https://github.com/db48x/emularity/blob/master/loader.js#L824

[andres-asm]

Since you're using asyncfs, adding dropbox integration would be just a drop-in replacement for the persistent part.
The only problem is that dropbox asks you to hardcode the allowed referrer URLs.

https://github.com/libretro/RetroArch/blob/master/pkg/emscripten/libretro/libretro.js#L84
https://github.com/libretro/RetroArch/blob/master/pkg/emscripten/libretro/libretro.js#L140

@db48x Thanks for the reference I was meaning to replace localStorage with idbfs

[db48x]

Also the user has to authenticate to use dropbox. That's an extra step that I don't want to require in the Emularity. What I'd rather do is make it easy to add that sort of thing in where it's wanted, especially since the folks at the Internet Archive asked me if syncing saved games back to the Archive was doable. BrowserFS makes it pretty easy either way, as you said.

[andres-asm]

Yeah, I actually used a localstorage variable to check if the user wants to use dropbox so the user has the choice but I can understand why the extra steps are undesirable.

I implemented idbfs now it's working nicely for RetroArch :)

[DanStafford62]

I'm using DosBox to run Dbase IV - created a new database in there, and it shows up when I restart DosBox - is persistent. However, I have no idea how to locate the .dbf file for import to Excel after closing DosBox. Any ideas how I can pull this out of the virtual machine environment?