[3.0] Exanite's Tracking Issue

[Exanite]

Exanite's Tracking Issue

This tracks a bunch of miscellaneous tasks that I accumulated while working on Silk 3.
I'm open for anyone to work on these tasks, but please discuss with me in the Silk Discord first.

In-progress

Tasks in progress will be moved to the corresponding pull request.

Uncategorized

This section contains informal reports, suggestions, or complaints about Silk 3.
I'm logging them here so they don't get forgotten about until I either open a formal issue or address them.

From Aqua:

• libSDL3.so is humongous (should not be nearly 14 MB)
  • Exanite: I checked my system version and that one is only 3 MB.
• Silk 3 does not prefer its own native binaries? I.e., it prefers the system ones instead of the ones shipped by Silk.
• Go through the thread Aqua has made: https://discord.com/channels/521092042781229087/1512914692853465088

From Ethereal:

• SilkTouch generator takes an immense amount of RAM: https://discord.com/channels/521092042781229087/607634593201520651/1513592461493145601
  • I need to also check with Curin to see how much RAM their device has. If their device doesn't have enough RAM, this might explain why the Microsoft job takes so long (aside from it already processing a lot of data).
  • Run the Microsoft job locally and see how much RAM it takes on my device.

From Nolram (https://discord.com/channels/521092042781229087/607634593201520651/1524213012402147348):

• Add Imgui bindings
  • Nolram recommended https://github.com/dearimgui/dear_bindings over cimgui (https://github.com/cimgui/cimgui)
• Investigate whether direct P/Invoke and static linking works in Silk 3 and how we want to support it.
  • See: I'm not familiar with what direct P/Invoke means though
  • Specifically, it involves some configuration, providing static builds, and maybe switching over to LibraryImport over DllImport.

High priority

Generator:

• ExtractEnumConstants does not preserve attributes placed on extracted constants (needs verification).

Bindings usage:

• Why don't we use the InlineArrayN<T> types provided by .NET? Should we use them?
  • Someone in the C# discord mentioned that these were added in .NET 10, which is what led to me add this task. This might be the reason we don't use them?
• OpenAL context should be disposable
  • This is noted by Perksey in the OpenAL.Tutorial001.HelloSound project. I need to figure out what this Dispose method should do exactly.
  • Check other API contexts as well for the same issue
    channels/521092042781229087/1514735149915439274/1519099432442794150
• The magic nullptr constant does not work. It fails to resolve to anything.
  • Apparently it works in the Silk 3 examples. Probably because it uses a ProjectReference instead of a PackageReference.
• The Vulkan bindings are double registering the LoaderInterface hook: https://discord.com/channels/521092042781229087/607634593201520651/1518220269519638599
  • I'm going to implement a temporary hack fix, but I don't believe it to be the correct fix.
    • We need a way to initialize an assembly consistently.
    • Currently there are multiple entrypoints and they aren't guaranteed to be called.
  • In Vulkan, we call RegisterHook in both Vk (defined in a manual file) and Vk.DllImport (defined in a generated file).
    • This causes a multiple registration error, which admittedly is my fault since I didn't realize OpenAL didn't do the same pattern.
    • Removing the duplicate call in Vk is the "fix" I mentioned above, but does not solve the underlying issue.
    • Static constructors are only invoked if the static constructor's type or members are accessed.
    • Notably, accessing Vk.DllImport does not trigger the static constructor for Vk, and vice versa.
    • Because we assume/require both static constructors to be called for proper behavior (Vk.DllImport registers the hook and is not extensible, so we have to register alternative shared library names in Vk), having only one of them be called is a problem.
• Add option to configure which vendor suffixes KhronosNonExclusiveVendor can identify. Eg: For OpenGL, this should be only ARB.
  • See conversation here: https://discord.com/channels/521092042781229087/587346162802229298/1531416150565453924
• Consider making function pointer structs work with nullptr. Seems like a nice QOL feature.
• OpenXR/CL/Vulkan: Add CurrentInstance/Device/Platform properties to interface partial.

Enums and other integral typedefs:

These have been proving to be consistently annoying to work with...
Vulkan and OpenXR have some special handling so this problem is less problematic, but still exists.

• I don't think the OpenGL bindings specify the method parameter types properly
  • Eg: void DrawElements(uint mode, uint count, uint type, void* indices);
  • This should use the enum type instead of uint.
• Vulkan: Some types are incorrect. uint is used instead of MaybeBool<uint>. https://discord.com/
• OpenCL: Same issue. Tracked in the smoke test project as todos.

Documentation:

• Add Vulkan smoke test project
• Differentiate between smoke tests and actual tutorials/examples. I would categorize the current SDL/OpenAL "tutorials" in Silk 3 to be smoke tests since they really shouldn't be referenced as sources of best practices.

Maintenance:

• Consider removing ModCSharpSyntaxRewriter
• Add SilkConstants class to generator
  • Ideally all magic numbers and constants are consolidated here and documented.

DI:

• Fix issues when running multiple jobs
  • Figure out why the headers are getting mixed up when running multiple jobs at the same time.
    • Seems to be related to TransformFunctions.
  • Figure out why the SDL handle structs are getting named as Silk.NET.SDL.ConditionHandle.gen.cs instead of ConditionHandle.gen.cs when running multiple jobs at the same time.
• Maybe rework how using statements are added. Not sure why, but using statements are randomly added/removed between Windows/Linux. This causes noisy diffs.
• Investigate how DI is set up in this project. I believe it is the core cause of the multi-job isolation issues. I need to verify this, but the jobs seem to share the same mod instances instead of being separated. This would explain why we need to access data by job key instead of it being simply injected into the mod instances.
• Rework registration/configuration system to be in pure C# instead of JSON
  • This also is related to reworking the DI setup.
  • This is something I volunteered to do.
  • See: https://discord.com/channels/521092042781229087/587346162802229298/1530907649561067693

Medium priority

Bindings:

• Use CLong where possible in the bindings.
  • For enums, use Constant<CLong, EnumHere>, and theoretically it should work.
  • See conversation here: [3.0] Add OpenCL and OpenXR bindings #2590 (comment)
• Consider implementing a IsProbablyABitmask function based on BuildTool's IsProbablyABitmask for identifying flags enums
  • Original comment: [3.0] Implement name-related improvements identified in previous Vulkan PR #2503 (comment)

Quality:

• Add some way of verifying our bindings are well polished.
  • These would mainly involve simple automated sanity checks that ensure our bindings our consistent. Could be done as a separate CLI project from SilkTouch.
  • Check for missing namespaces.
    • This was an issue discovered during [3.0] Split up ExtractNestedTyping and add tests for TransformHandles #2587. I've fixed one case of these, but haven't verified the rest of the generated files (since there's way too many to manually verify).
  • Check for spelling/casing issues? Identifiers like Sdl.PropGpuDeviceCreateDebugmodeBoolean are technically correct (native name is SDL_PROP_GPU_DEVICE_CREATE_DEBUGMODE_BOOLEAN, so Debugmode is "one" word), but we could have an audit tool that checks for these non-ideal cases so we can add manual overrides. Worth the effort? Probably not unless we care about perfecting the bindings.

Maintenance:

• ClangSharp switched from System.CommandLine to a custom command line argument parser. I noticed that we had verbatim copied code from ClangSharp for handling arguments in Silk 3, so we should update that.
  • This also lets us update System.CommandLine, which if I recall, we are still using a beta version of.

Low priority

CI:

• Prevent native builds from triggering unnecessarily when the PR description is edited
  • Native builds should only trigger once. If the last commit in the PR is from the native builds workflow, it should not run again.
• Figure out how to make our CI workflows work with forks.
  • I did some testing here and got a proof-of-concept working: https://github.com/Exanite-Dev/GithubActionTesting
  • Note that we need to be careful about security here. This involves escalation of privileges.

Maintenance:

• Cross reference changelog and past issues for Silk 2 to ensure that we don't have the same bugs in Silk 3

High level utilities:

• Reimplement the Vulkan struct chaining API
• Add image format utils for Vulkan/etc

IDE / Metadata:

• Ensure SupportedApiProfiles attributes are correct
  • Properly resolve API profiles for Flags/FlagBits types
  • Prefer name from NativeName for resolving API profiles
• Consider optimizing NativeName attributes.
  • This should involve renaming StripAttributes to CleanupAttributes and adding an option in its config to optimize NativeName attributes.
  • Also see: [3.0] Implement name-related improvements identified in previous Vulkan PR #2503 (comment)

Completed

Completed during #2588:

• Switch to SLNX solution format to improve readability/mergability
• From Ethereal: They had a CSharpier TypeLoadException when running the Win32 job on develop/3.0. Might be related to the error I saw half a year ago. If so, likely can be fixed by simply updating CSharpier. CSharpier has updated their Roslyn package since then.

Completed during #2591:

• Port back the --only option from Curin's branch so I don't need to specify --skip for all but one job

Completed or planned to be completed during #2590 (not yet merged):

• Begin work on the rest of the Khronos bindings
  • I'd like to start this early because this lets us spot issues early and we can use these as test cases for the rest of the changes.
• Generator usage: Explain SilkTouch CLI options
• Generator usage: Clarify that SilkTouch does not modify non .gen.cs .cs files during the Generated Bindings Output section
• Consider reworking data type trimming regexes.
  • This was discussed more here: [3.0] Implement name-related improvements identified in previous Vulkan PR #2503 (comment)
  • The summary is that the current set of 2 regexes doesn't elegantly handle the case where the ending of a word should not be trimmed, but the word itself can have data type suffixes.
  • For example: alSourceRewindv and alSourceRewind. The d in Rewind can be mistaken as a data type suffix, so we add it to the EndingsNotToTrim regex. However, the dv in Rewindv also has the same problem.
  • We likely need something more akin to WordsNotToTrimInto than EndingsNotToTrim.
• Investigate why we currently use DllImport in Silk 3.
  • Also added some light documentation to AddVTables in the mod MD docs.

Completed in dotnet/ClangSharp#699

• Consider opening a ClangSharp PR to fix issue with SDL_MAX_SINT64 generation on Linux
  • More context: https://discord.com/channels/521092042781229087/1376331581198827520/1443401810822828113
  • public const nint SDL_MAX_SINT64 = unchecked(0x7FFFFFFFFFFFFFFF); does not compile

Completed outside of a PR:

• For DrawElements/etc, check if the pointer to 0 issue present in Silk 2 still exists in Silk 3
• Investigate using the new BitCast optimizations: https://discord.com/channels/143867839282020352/312132327348240384/1517589517727436952
  • Eg: BitCasting a user struct to a Vector struct and immediately using an SIMD operation is now properly optimized. Free performance!
  • This mainly affects Otac0n's work, but I'm logging it here so I also keep it in mind as I work on the bindings.
  • Decision: Just going to keep this in mind. Don't think I need to actively review the bindings for this since there's not many places where this will be used.

Not doing

For transparency and for reference, I'll move any tasks I don't plan on doing here.

• Figure out how to handle CLA properly. Currently native builds will block CLA due to the use of a bot account.
  • Seems to be fine now.

[Exanite]

Reserved