Add sidecar for speed up ssd streaming#514
Open
GiorgioOppo wants to merge 6 commits into
Open
Conversation
…BUNDLE=1) Serve streaming expert-cache misses from a sidecar file where each routed expert's gate|up|down slabs are contiguous, so a miss becomes one sequential burst instead of three reads scattered across the GGUF. Same bytes, same numerics; built one-time next to the model (or in $DS4_BUNDLE_DIR), fingerprinted against the model and rebuilt when the GGUF changes. - ds4_ssd.c/.h: DSEB v1 sidecar format (build/validate/open), byte-compatible with the DwarfStar Swift port so one bundle serves both implementations - ds4_metal.m: remap exact expert-slab preads to the sidecar record at the lowest read level; every miss/seed path benefits, plain GGUF fallback - ds4.c: engine wiring (contiguous uniform routed-layer range only; mixed-precision boosted models are skipped) - ds4_cuda.cu, rocm: no-op stubs (Metal-only for now) - tests/expert_bundle_test.c: build/reuse/tamper/DS4_BUNDLE_DIR coverage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoyYYU8BwNXeJpt2WRsmXL
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoyYYU8BwNXeJpt2WRsmXL
- Metal: a failed sidecar read now retries the same slab from the model file (the sidecar can never break inference), logging the first fallback - Metal: F_RDADVISE readahead hints follow the remap to the bundle fd instead of prefetching GGUF ranges that are never read - API: ds4_gpu_set_streaming_expert_bundle takes the validated ds4_expert_bundle + layer table, returns acceptance so the engine only logs 'active' (and keeps the fd) when the backend really serves reads; the Metal remap reuses ds4_expert_bundle_record_offset for the layout math - build: per-process .tmp name (concurrent builders can no longer interleave writes), stale incompatible bundles are removed before the space check, statvfs failure is treated as unknown instead of full disk - open: overflow guards on the size arithmetic before trusting the file Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoyYYU8BwNXeJpt2WRsmXL
…mparison-0vj3ah Claude/swift main c sidecar comparison 0vj3ah
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoyYYU8BwNXeJpt2WRsmXL
…mparison-0vj3ah docs(bundle): measured +9% decode with the sidecar on M1 Pro 16GB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This sidecar could be speed up the ssd streaming by 20%
#491