Support binary files in ActiveCode (base64 delivery + per-language compile-also) - #1361
Support binary files in ActiveCode (base64 delivery + per-language compile-also)#1361skiadas wants to merge 5 commits into
Conversation
Binary files (e.g. a compiled .jar or .zip) are stored base64 in source_code. A new is_binary flag records that the contents are a binary payload, so the client knows to hand them to a server (Jobe) verbatim rather than treating them as source. Text files keep it False. The manifest processor reads the data-isbinary flag when populating the database, and get_source_code returns is_binary so a program on one page can recognize a binary file stored for another.
The migration round-trip adds and drops the source_code.is_binary column. The crud tests store and fetch a binary file's is_binary flag and verify text files keep it False. The endpoint tests confirm get_source_code surfaces is_binary in its response, so a program on another page can recognize a binary file.
A datafile carrying data-isbinary (or a DB row with is_binary set) holds base64 contents. The datafile component now preserves the data attributes when it replaces the source element, and livecode pushes the contents to Jobe verbatim instead of treating them as text. The file then lands in the program's working directory, so any language can read it.
compile-also names files that are part of the build. What that means depends on the language and on whether the file is text or binary: - text source files are compiled together with the main program; - a Java/Kotlin .jar or .zip goes on the compiler's classpath (-cp), so its classes are visible to the program (the -cp also rides in the interpreterargs, which must reproduce Jobe's Java defaults); - a C/C++ .o or .a is placed on the link line, after the source; - other languages just read the delivered file from the working directory.
The datafile tests check that a binary element keeps its data-isbinary attribute on the rendered container. The livecode tests cover binary delivery (base64 passed verbatim, never through parseJavaClasses) and the per-language compile-also wiring: jar files on the classpath, object files on the link line, text sources in compileargs, and nothing for interpreted languages.
|
@skiadas - Thanks for this, I did see something related pop up on the PreTeXt group and as a PR? But, I'm confused, maybe I've not spent enough time following pretext-dev. Does this replace datafile? I thought that was what I read. If so why? why not just extend what we have? If it replaces how are we going to provide backward compatibility for all of the datafiles already in the database and in many many books? @ascholerChemeketa would welcome your thoughts. |
Here's the relevant pretext PR: PreTeXtBook/pretext#3139 The new file tag replaces the datafile tag in a backwards compatible way. All the old datafiles get converted to files in the preprocessor/repair, and the file tag emits the exact same manifest/html in those cases. So on the runestone end even without this PR, but just with the pretext PR, the support for text or image datafiles or (text/image files in the new format) should be still working. So that is a first test to do, grab just the pretext pr and run runestone tests on it, nothing should have changed. On the runestone side everything is still called "datafile". But now it comes from pretext files. I just thought trying to shoehorn the new proposed syntax into the existing pretext-datafile element would be hard, confusing and misleading as these are no longer just "data" files. What this runestone PR does allow is the the handling of the file elements where the format is not image or text but binary, like a jar file, a wheel, a zip file, a C++ object .o file etc. The pretext PR enables those file[format=binary] elements to appear in pretext source, but their proper handling falls on Runestone. And yes I would very much like Andrew's and your eyes on both of these commits. |
|
The RS side changes seem pretty limited (binary datafile handling) and desirable regardless of what happens PreTeXt side. Rob hasn't had a chance to weigh in on the rebooted conversation on |
3 participants
Summary
Support for binary files (e.g. a compiled
.jaror.zip) supplied by a PreTeXt book and made available to server-side (Jobe) ActiveCode programs.PreTeXt's new
fileelement (see the companion pretext PR #3139) marks a file as binary withdata-isbinary="true"and stores its contents base64. These commits thread that flag through the stack so a binary file reaches the program's working directory intact, and so a program on another page can still recognize and use it.What each commit does
source_code.is_binaryboolean column, populated fromdata-isbinaryduring manifest processing, and returned byget_source_code. Binary files are stored base64; text files keepis_binary = False.get_source_codeendpoint coverage.datafilecomponent preservesdata-isbinaryon the rendered element;livecodehands a binary file's base64 contents to Jobe verbatim, so the file lands in the working directory and any language can read it.compile-alsomeans different things per language: text sources compile together, a Java/Kotlin.jar/.zipgoes on the classpath (-cp, with Jobe's Java interpreter defaults reproduced), a C/C++.o/.agoes on the link line, and interpreted languages just read the delivered file.Notes / limitations
fileelement living on another page, becauseget_source_codereturnsis_binaryso the client treats the fetched contents correctly..so) are not supported: the server cannot arrange for the runtime linker to find them. Object files (.o) and static archives (.a) work.DataFilesEditor/instructor.py) is out of scope here: it still treats uploaded files as text. Supporting binary uploads there (reading them as base64, storingis_binary) is a natural follow-up.fileis dropped from static (non-Runestone) output by PreTeXt.Related
source_code.is_binaryflag the author-facing editor will need; binary upload there (base64 +is_binary) is the natural follow-up to the out-of-scope note above..jarfiles via@datafile. The scenario binary-file support enables; the legacy text-jar/parseJavaClassespath is preserved.