i think it will be very fast, given the flatbuffer...

# framework-elide

[Sam Gammon]

i think it will be very fast, given the flatbuffer backing and i'm anticipating using LZ4 for compression -- but i'm concerned that a custom binary-only format makes inspection difficult for devs

i like that it's a standard tarball but maybe, for example, it should also have a JSON file, so that it can easily be inspected?

the steps at runtime, i'm thinking, are

1. if enabled in app configuration:
a. load the tarball from JAR resources or file (it having been placed there by the Gradle plugin, or equivalent, modulo toolchain in use)
b. look for magic # header, grab crc32 and compression mode
c. seek to beginning of Tree flatbuffer, load it into memory while decompressing, according to compression
d. the Tree has the file tree and byte offsets for each file, which we have on hand now
e. verify the Tree by calculating the crc32 and comparing it
f. we load the data region as a RandomAccessFile, perhaps written to tmp
g. the Tree and data region are used to build an in-memory JimFS filesystem, decompressing while we go
i. while each file is decompressed, a checksum is calculated and compared
h. at runtime, guest requests file via I/O:
i. impl receives request, enforces security policies
ii. allowed requests go to JimFS
iii. JimFS serves it from memory in already-decompressed form
tradeoff decisions proposed:

• Init instead of runtime: load and decompress up-front so that file loading at runtime is instant, it's all already in memory
• Emphasis on fast decompression: LZ4 won't save major space, but it will minimize the impact on startup time to the greatest extent possible
• Non-goal of pre-compressed browser assets: The assets subsystem already pre-compresses assets at build time, so it's a moot issue
• Backed by well-tested code: After boot, JimFS kicks in which is well tested and performant (we don't have to deliver crazy runtime performance)
• Streaming and copy-free: As much as possible, we shouldn't have to copy while initializing the runtime, which Flatbuffers and streaming compression deliver
(for reference, this is the guest FS that will be used to load ESM, NPM, Python modules, etc).