v1.3.0 - #24
Draft
FKKimura wants to merge 146 commits into
Draft
Conversation
…apter separately, run inference with vLLM
- Add `get_quant_config()` returning GPTQ-compatible config - Add `_build_quantization_bits()` and `finalize_quant_config_for_save()` - Add `create_inference_layer()` to build GPTQLinear from JointQResult
…wledge, verified with vllm
…ion knowledge, and add smoke test
Develop/v1 1 0
GPTQLinear weight packing only supports wbits in (2, 3, 4, 8), so JointQ with bits=1 must build/save the inference layer with pack_weights=False. - JointQ.validate_params: warn when bits=1 to remind callers to pass pack_weights=False at inference layer construction time - GPTQLinear.from_saved_state: load qweight/qzeros as unpacked tensors when wbits=1, matching the unpacked save format
…/lora-merge-v1.1.0 Resolved conflicts in: - onecomp/quantizer/jointq/_jointq.py - onecomp/runner.py
Add QuantizationProgressTracker and wire it through calibration, chunked calibration, multi-GPU phase 2, QEP general and arch-aware paths. Runner gains quantization_progress flag (default on). Includes unit tests for ETA formatting and thread-safe stepping. Co-authored-by: Cursor <cursoragent@cursor.com>
…wledge, verified with vllm
Raise clear error for unsupported QEP quantizers See merge request onecomp/onecomp-lab!71
feat: quantization progress logs with ETA
* refactoring : QuantizationProgressTracker * update CHANGELOG.md --------- Co-authored-by: FKKimura <50981196+FKKimura@users.noreply.github.com>
- QuantizationProgressTracker: suppress duplicate logs after completion, use `is not None` for lock check, expand Google-style docstrings - Demote per-layer / per-chunk INFO logs to DEBUG so the [progress] line is the single canonical per-step INFO signal (onecomp/quantizer/_quantizer.py, onecomp/runner_methods/chunked_quantization.py, onecomp/qep/_quantize_with_qep_arch.py) - Chunked path: switch to per-group progress (was per-layer × per-quantizer); drop chunk_progress / layer_progress arguments - Shorten tracker labels: "Quantization", "Chunked quantization", "QEP" - Per-block MSE log in QEP arch path: drop redundant `[INFO]` prefix, switch to lazy `%` formatting, fix "Layer N" -> "Block N" - Expand QuantizationProgressTracker tests (10 cases, incl. thread-safety and overflow suppression) - Update CHANGELOG for the new [progress] line and INFO -> DEBUG demotion
See merge request onecomp/onecomp-lab!72
Fix three OneComp-side bugs surfaced by Takane VLM (Cohere2Vision) work: the `save_quantized_model` flow dropping VLM auxiliary configs, the `load_quantized_model` flow leaving tied `lm_head` and other non-quantized modules in the wrong dtype after `load_state_dict(..., assign=True)`. Issue 1 (save_quantized_model misses VLM aux configs): - Replace the hard-coded allow-list (processor_config.json / preprocessor_config.json) with an extension-based whitelist of `*.json` / `*.jinja` (excluding weight shards, weight index files, `config.json`, and `generation_config.json`) - Extract source-model-dir resolution into `Runner._resolve_source_model_dir()` with a `snapshot_download( local_files_only=True)` fallback that warns on failure instead of aborting the save (onecomp/runner.py) - Skip files already produced by `model.save_pretrained` / `tokenizer.save_pretrained` and INFO-log both copied and skipped entries for traceability Issue 2 (tied lm_head left at fp16 after load): - Call `model.tie_weights()` after `load_state_dict(..., assign=True)` to restore `lm_head.weight is embed_tokens.weight` identity that `assign=True` breaks (onecomp/quantized_model_loader.py) - Guard the call on `tie_word_embeddings=True` and `isinstance(model.lm_head, nn.Linear)` to avoid touching quantized layers such as `GPTQLinear` - Walk one level of nested sub-configs (`text_config` / `language_config`) via `_should_retie_word_embeddings` so HF VLM configs that move `tie_word_embeddings` under `text_config` are also covered Issue 3 (non-quantized VLM modules left at fp16): - Make `_build_empty_model_from_config` honor `torch_dtype` / `dtype` from `config.json` when the caller did not pass an explicit dtype, so empty modules start in the saved checkpoint's dtype (onecomp/quantized_model_loader.py) - Add `_cast_fp16_to_target_dtype` as a safety net that casts any fp16 parameter / buffer left on non-quantized modules to `model.config.torch_dtype` after `tie_weights()`, returning the list of fully-qualified names that were converted for logging and tests - Skip `GPTQLinear` / `DoubleBinaryLinear` and limit the cast to fp16 so mixed-precision fp32 LayerNorms are not disturbed; mutate `p.data` in place so tied-weight identity is preserved Tests (tests/onecomp/runner/, 30 cases, CPU-only, no network): - test_save_quantized_aux_files.py (7): whitelist copy, existing-file skip, `snapshot_download` fallback, subdirectory exclusion, INFO log for skipped files - test_load_tied_embeddings.py (7): tiny LlamaForCausalLM with `tie_word_embeddings` True / False, identity restoration, forward dtype, `_should_retie_word_embeddings` top-level / nested / all-False / unrelated-sub-attrs cases - test_load_excluded_module_dtype.py (16): config-driven empty-model dtype, fp16 safety-net cast, fp32 preservation, quantized-layer skip, `_resolve_dtype_from_config` parsing edge cases, returned module-name list Update CHANGELOG.md (v1.1.1 Bug fixes) and verify each fix in isolation by reverting it and confirming the targeted test fails.
See merge request onecomp/onecomp-lab!69
Fix-tests-examples See merge request onecomp/onecomp-lab!70
Develop/v1-1-1
Resolved conflicts in: - CHANGELOG.md - example/vllm_inference/example_gptq_vllm_inference.py - onecomp/runner.py
feature: Add legacy vLLM environment definitions (v0.12.0 / v0.15.1)
…vllm-infer_jointq feat: LoRA adapter sidecar save/load support with vLLM-compatible inference + JointQ smoke test
…/partial-quant-with-rotation-bug
…rotation-bug fix: Derive Hadamard online-hook target types from the model for rotation + partial-quantization
add standalone package for vllm 0.24.x with ROCm See merge request onecomp/onecomp-lab!94
fix seed for quantizer-test See merge request onecomp/onecomp-lab!98
Remove legacy LoRA examples and complete README examples list See merge request onecomp/onecomp-lab!99
* cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * add test and chalgelog, and apply linter * Reverted the code under the example --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…Qwen3.6 (#35) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * apply linter * revert * Revert "revert" This reverts commit 0e006d6. --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…model save (#36) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * revert over-erased comment * add example file for qwen36, revert existing example, and update CHANGELOG * add new example file for qwen36 * apply pre-commit * revert blank line * revert blank line * revert qep option in example_gptq_vllm_inference.py * revert model_id in example, and fix comment to take users to example_gptq_vllm_qwen36_inference.py * fix typo --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
…3.6 35B-A3B (#37) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * refactoring, linter, add test, and fix changelog * revert example modification * revert example modification * fix format * update docstring of Runner._exclude_moe_router_if_needed --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
* cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * bugfix: success save load * refactoring, linter, add test, and fix changelog * revert example modification * add test and fix changelog * revert example modification * fix format * bugfix: success save load * add test and fix changelog * add round trip test --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
Fix --extra-index-url for downloading vllm's wheel file (ROCm) See merge request onecomp/onecomp-lab!108
#41) * [update] support rotated gptq vllm inference (TP = 1) * [fix] restrict online hadamard targeting to dense mlp.down_proj - share the Hadamard target predicate between preprocessing and vLLM - limit online Hadamard application to dense `mlp.down_proj` only - exclude MoE expert `down_proj` paths from the current rotation flow - keep preprocessing and vLLM behavior consistent with current MoE support scope * [fix] support rotated GPTQ vLLM inference with tensor parallel handling * [fix] register rotated linear method for vllm weight_loader_v2 dispatch - register RotatedLinearMethod via vllm's weight_loader_v2 allowlist so wrapped GPTQ / GPTQ-Marlin layers keep the same loading path as unwrapped vllm methods - add regression coverage for rotated down_proj dispatch and base process_weights_after_loading delegation - add unit test for the online Hadamard target predicate - document the TP>1 all_gather cost in _apply_tp_hadamard - clarify that rotation hooks currently target dense mlp.down_proj layers only * [test]Add smoke test with TP=2 * [test] fix dependency * [test]Add rotation unit tests and runner save-path coverage * [test]TP1 rotated GPTQ e2e coverage * [test] Move rotation helper tests into utils package * [test]make RotationMetadata.from_quant_config tests explicit * [test]cover missing rotation utility edge cases * [fix] enable DBF + rotation support in vLLM plugin - Add RotationMetadata support to DbfConfig - Wrap quantized layers with RotatedLinearMethod when rotation is configured - Expose layer dimensions in DoubleBinaryLinear * [test] add DBF rotation plugin integration tests and Runner coverage * [test] TP1 rotated DBF e2e coverage * [test] strengthen DBF rotation plugin unit tests - Add test_get_quant_method_wraps_dbf_linear_method_for_quantized_down_proj: verifies DBFLinearMethod (not UnquantizedLinearMethod) is the base when quantization_bits has a real entry, and that _dbf_mod_cfg/_dbf_prefix are set - Add test_prehook_is_installed_on_down_proj_after_process_weights: verifies _onecomp_hadamard_prehook_installed is True after process_weights_after_loading through the DBF quantized path - Add test_create_weights_raises_for_tensor_parallel_size_greater_than_one: verifies DBFLinearMethod.create_weights raises ValueError when get_tensor_model_parallel_world_size returns > 1 (monkeypatched) * [test] trim redundant rotated-vs-plain e2e quantize tests The saved-config metadata contrast (rotated flag + quant_method routing) is already covered by the cheap fake-config unit tests in test_runner_rotated_vllm_save.py, so the e2e config-comparison tests only re-verified that logic on top of a full quantization run. DBF: drop test_plain_dbf_uses_different_saved_load_path and rename TestRotatedVsPlainDBFVllmInference -> TestPlainDBFVllmInference. The plain generate smoke is kept because plain DBF still loads through the in-house "dbf" plugin. GPTQ: remove TestRotatedVsPlainGPTQVllmInference and the now-unused plain_quantized_model_dir fixture entirely, eliminating a second GPTQ quantization run. Plain GPTQ ("gptq") is served by vLLM's built-in handler and exercises no first-party code, so the smoke added little value. * [fix] fall back to naive path on GemLite inference failure with process-wide disable * [test] add GemLite fallback regression tests and strengthen assertions * [docs] document GemLite automatic fallback behavior and TRITON_CACHE_AUTOTUNING * [test] strengthen GemLite fallback tests: fused 2D scaling0, real naive, bias Cover the production-critical and previously-untested paths in DBFLinearMethod.apply()'s GemLite -> naive fallback: - Add test_fused_fallback_matches_real_naive (part_count=3). Uses a 2D (part_count, in_features) scaling0 so it exercises the `scaling0.ndim == 2` per-part-index branch in _compute_parts() that the fused qkv_proj actually hits. With only _apply_gemlite forced to fail, it runs the real _compute_parts / _apply_naive and verifies torch.cat(dim=-1) concat, per-part scaling2/scaling4/bp offset slicing, and the naive numerics against an independent reference. - Add test_bias_is_added_to_output covering the final `out + bias` path. - Replace caplog with a warning_spy fixture that patches the module logger directly, so the warning assertions no longer depend on the vLLM logger keeping propagate=True. * [docs] clarify that OOM is not caught by the GemLite fallback * [test] modularize vllm_plugins test infrastructure: shared conftest, session fixtures, try/finally cleanup * [test] consolidate LLM import in conftest and guard DBF integration test * [test] remove dead code in build_vllm_llm and document DBF skip gate Follow-up cleanup to the vllm_plugins conftest consolidation: - conftest.py: drop the unreachable `return LLM(...)` after pytest.skip() (pytest.skip raises NoReturn, so the line never executes and no type checker is configured to require it), and remove the stray blank line left between the vllm import and its except clause. - test_rotated_dbf_e2e.py: note in the module docstring that the file is skipped by default and enabled via RUN_DBF_INTEGRATION_TESTS=1, matching the convention in develop/v1-3-0's test_global_ptq_integration_dbf.py. * [docs] add v1.3.0 rotation/vLLM changelog and fix outdated rotation notes Add a CHANGELOG entry for the rotation-save-load-vllm-infer branch covering rotated GPTQ/DBF vLLM inference, the DBF GemLite automatic fallback, and the new tests. Correct the now-stale "rotation is not vLLM-servable" notes in the docs: rotation-preprocessed checkpoints are servable through the mixed_gptq and dbf plugins (dbf is TP1-only), while built-in gptq paths (RTN/JointQ) are out of scope. * [style] Apply pre-commit fixes * [fix(vllm)] mirror base method's weight_loader_v2 support in rotation wrapper * [docs] clarify rotation wrapper weight-loader dispatch in CHANGELOG --------- Co-authored-by: koohr <kou.ohira@compmind.co.jp>
* [update] Bitpack GPTQ weights and zeros immediately after quantization * [fix] disable GPTQ bitpacking for JointQ initialization * [fix] existing GPTQ tests to use unpacked quantize results by default * [fix] existing AutoBit tests to use unpacked quantize results by default * [update] Propagate AutoBit bitpack mode before child quantizer validation * [update] Add bitpack-mode flag (bitpack_on_quantize) default to base Quantizer * [update] Validate bitpack_on_quantize support in GPTQ params * [update] Add GPTQ bitpack equivalence test and example * [fix] Limit GPTQ wbits validation to 15 bits * [update] add test_gptq_bitpack.py for cover bitpack metadata, dequant, inference paths * [fix] Fail AutoBit validation for unsupported GPTQ bitpack wbits * [update] Update CHANGELOG.md (bitpack_on_quantize) * [style] Apply black formatter * [update] Update CHANGELOG.md (v1.3.0-wip merge into bitpack-mode branch) * [update] Add DBF bitpack_on_quantize flag and validation Add a DBF-specific bitpack_on_quantize flag (default True) to the DBF quantizer dataclass and validate it as a bool in validate_params(). DBF packs arbitrary shapes via padding, so no bit-width/shape constraint is imposed; only the bool type is checked. * [update] Add DBFResult packed-state metadata and unpack helpers Add packed-state metadata (dbf_A_is_packed, dbf_B_is_packed, dbf_A_original_shape, dbf_B_original_shape) and a get_unpacked_binary_factors() helper to DBFResult so callers can obtain the unpacked +/-1 factors regardless of storage. The helper returns float16 for both packed and unpacked storage so the representation difference never leaks to callers. Add shape-aware unpack_binary_matrix() (and pack_binary_matrix alias) to dbf_layer so the pack/unpack shape-restore logic lives in one place. Old results without the metadata fields are treated as unpacked via getattr defaults. * [update] Bitpack DBF binary factors right after quantization In DBF.quantize_layer(), when bitpack_on_quantize is enabled and DBF produced both binary factors, pack dbf_A / dbf_B into uint8 CPU tensors immediately after run_dbf() and record the packed-state metadata on the DBFResult. The unpacked references are dropped early to reduce RAM held during quantization. On DBF failure / is_dbf_quantized=False the factors are left untouched, preserving the existing dequant error behavior. * [update] Make DBF dequant path packed/unpacked agnostic compute_dequantized_weight() now recovers the +/-1 factors via get_unpacked_binary_factors(), so it returns the same weight whether dbf_A / dbf_B are stored packed or unpacked (pack/unpack is value-preserving). The reconstruction formula is unchanged. Route the existing in-layer unpack call sites (BitLinearPacked.forward and DoubleBinaryLinear._unpack_bp) through the shared unpack_binary_matrix helper to avoid duplicating the shape-restore logic. * [update] Avoid re-packing already-packed factors in DoubleBinaryLinear DoubleBinaryLinear.__init__() / from_quantization_result() now accept already-packed dbf_A / dbf_B (with their original shapes) and register them straight into bp1 / bp3 without an unpack/re-pack round-trip; unpacked inputs are packed as before. GemLite initialization builds a short-lived unpacked copy for packed inputs only (not retained). The saved checkpoint format (bp1 / bp3) and from_saved_state() are unchanged. * [update] Make existing DBF tests packed/unpacked agnostic Update the existing DBF quantizer test assertions that assumed unpacked float16 dbf_A / dbf_B: check_quantize_layer branches on the packed flags (uint8/1D when packed, float16/2D otherwise) and reconstructs the weight via get_unpacked_binary_factors(); check_equal_results and the apply-to-module helper compare/store factors through the same helper so they hold for both representations. No changes were needed in the blockwise post-processing optimizers or the vLLM DBF plugin, which work on the unchanged bp1 / bp3 packed buffers. * [update] Propagate AutoBit bitpack mode to child candidates and DBF fallback Add a bitpack_on_quantize flag (default True) to AutoBitQuantizer and propagate it to candidate quantizers that support on-quantize bitpacking: validate_params() syncs the flag onto GPTQ/DBF child candidates (via _sync_child_bitpack_on_quantize) before their own validation, _assign_all_dbf() forwards it to the all-DBF quantizer, and the inject_dbf() fallback path forwards it to every DBF it creates. A non-bool flag raises a clear ValueError. * [update] Add DBF bitpack representation and propagation tests Add DBF quantizer tests covering both representations: default flag is True; packing stores uint8 1D factors with shape metadata; disabling keeps unpacked float16; packed and unpacked dequantized weights are bit-identical (both from quantize_layer and hand-built results); pack_binary -> unpack_binary_matrix is bit-exact across shapes; missing original shape raises; from_quantization_result registers bp1/bp3 with no re-pack and forward matches the dequantized linear; and a non-bool flag raises. Add AutoBit tests that the flag is synced onto DBF candidates and forwarded to inject_dbf fallbacks (1-bit GPTQ on a 64-wide layer to stay under the DBF threshold). * [update] Smoke-test DBF packed results through downstream consumers QEP, LPCD, chunked / multi-GPU quantization, and the cumulative-error analyzer all consume DBF results only via compute_dequantized_weight().to(device).to(dtype) and never touch dbf_A / dbf_B directly, so no functional change is needed for the on-quantize bitpack. Add a regression smoke test exercising that exact consumption path (analyzer._update_weights) with a packed DBFResult, confirming it succeeds and yields weights identical to the unpacked representation (no packed state leaks to callers). * [update] Add DBF save/load round-trip and vLLM e2e/config smoke tests - pre_process pipeline: add DBF quantized/dequantized save/load round-trip cases for TinyLlama and Qwen3 (5 cases: GPTQ q/dq, DBF q/dq, RTN dq) - tests/vllm_plugins/dbf/test_dbf_e2e.py: DBF quantize -> save -> config verification -> vLLM generation smoke (slow, CUDA/vLLM gated) - tests/vllm_plugins/dbf/test_dbf_config.py: DBF plugin config parse/dispatch smoke (DbfConfig) * [add] Add DBF bitpack runner smoke tests Add DBF packed-result smoke coverage for QEP, LPCD, and chunked calibration calc_quant_error paths. The tests verify that each Runner path produces packed dbf_A / dbf_B results and consumes them through compute_dequantized_weight() without leaking storage details to callers. * [fix] Expose in_features/out_features on DoubleBinaryLinear register_online_hadamard_hooks -> get_hadK introspects module.in_features, but DoubleBinaryLinear only stored _bp1_shape/_bp3_shape, raising AttributeError when re-registering Hadamard hooks on saved DBF-quantized rotated models. Expose in_features/out_features like nn.Linear (derived from the original unpacked shapes; also set in from_saved_state). * [update] Use DBF factor pack helpers * [update] Add DBF bitpack equivalence example Quantize a layer with bitpack_on_quantize=False vs True under identical seeds and assert the dequantized weights match bit-exactly (compute_dequantized_weight -> torch.equal). The corresponding DBF equivalence test is in tests/onecomp/quantizer/dbf/test_dbf_bitpack_equivalence.py. * [update] Add DBF vLLM inference example Parallels example_gptq_vllm_inference.py for the DBF path: quantize TinyLlama with DBF, save, then load and generate with vLLM. * [update] Document bitpack_on_quantize in DBF and AutoBit algorithm docs Add a bitpack_on_quantize row to the parameters table in docs/algorithms/dbf.md (DBF) and docs/algorithms/autobit.md (propagation to candidate / injected-DBF quantizers). * [update] Register DBF bitpack and vLLM examples in README and vLLM inference guide * [fix] Log bitpack_on_quantize overrides in JointQ and AutoBit * [doc] Clarify GPTQ bitpack-on-quantize behavior - Document the breaking change in CHANGELOG - Add a comment explaining qzeros reshape after pack/unpack * [refactor] Inline GPTQ_MAX_BITS usage in pack support validation * [fix] Centralize GPTQ bit-packing width checks * [refactor] Clarify GPTQLinear packed state handling * [update] Update CHANGELOG.md (DBF bitpack-on-quantize) * [style] Apply black formatter * [docs] update CHANGELOG.md for GPTQ pack_weights validation * [fix] Use is_packable_wbits to decide weight packing * [fix] Log pack_weights fallbacks for unsupported GPTQ wbits * [fix] Normalize float GPTQ wbits and fix config bits truncation * [docs] update CHANGELOG.md for GPTQ wbits normalization * [chore] Remove redundant bitpack equivalence examples Equivalence is already covered by tests, and these examples do not demonstrate user-facing usage. * [style] Apply pre-commit fixes * [docs] Clarify bitpack-on-quantize support boundaries - document GPTQ quantize-time and save-time packing constraints - clarify AutoBit propagation, overrides, and fused-group restrictions - document DBF and JointQ bitpacking behavior - record these doc updates in the DBF and GPTQ bitpack changelog sections * [docs] Sync example references after bitpack example cleanup - Remove references to the deleted DBF and GPTQ equivalence examples - Add the missing Qwen3.6 vLLM example to README and the inference guide --------- Co-authored-by: cm_ysmz <yohei.shimizu@compmind.co.jp> Co-authored-by: sikoji <seiichiro.kojio@compmind.co.jp>
… FusedMoE kernel support) (#39) * cherry-picking commit 7bf5621. * cherry-picking commit bd92a14. * Update model configuration and quantization settings; enhance blockwise attention type checks * temp * wip * succeeded in VLLM inference * succeeded in VLLM inference * git cherry-pick 4b38840 * revert over-erased change * Merge branch 'feature/qwen36_27b_save' of https://github.com/aki916/OneCompression into feature/qwen36_27b_save * Revert "succeeded in VLLM inference" This reverts commit 1b5bb84. * Revert "wip" This reverts commit 64fc908. * add test and chalgelog, and apply linter * Reverted the code under the example * fix loader, add test, and modify changelog * fix unfinished merge * byebye miss-merged implementation * update changelog * update docstring, docs, and README * Revert "revert" This reverts commit 0e006d6. * add test and fix changelog * Update model ID and save directory for Qwen 3.6 to version 35B-A3B; enhance exclusion keywords in quantizer settings * bugfix: success save load * succeeded in inferecing MoE quantized model with vLLM * refactoring, linter, add test, and fix changelog * revert example modification * add test and fix changelog * revert example modification * fix format * bugfix: success save load * add test and fix changelog * update CHANGELOG * succeeded in inferecing MoE quantized model with vLLM * update CHANGELOG * add test and fix changelog * update comment for review no.4, 6, 7, and 8 * generalize moe detection * comment2: fallback when group_size is None * comment3: add test * comment5: allow rtn fallback when every children quantizer GPTQ * missed adding __init__.py * lazy import is_moe_expert_g_idx_key in runner --------- Co-authored-by: aki916 <aki916.genyo@gmail.com>
Add CPU inference & GGUF export for OneComp quantized models (llama.cpp) See merge request onecomp/onecomp-lab!100
…ost-process audit metadata (#44) * [update] enable load → post-process → re-save for quantized models Allow a previously saved quantized checkpoint to be reloaded, refined with additional post-processes (e.g. BlockWisePTQ), and saved again, with an audit trail of which post-processes were applied. - quantized_model_loader: accept device_map=None/"" to leave the loaded model on CPU (post-processes assume CPU), and reattach quantization_config to model.config so a loaded model can be refined and re-saved without separately re-reading config.json. - runner: run_post_processes now accepts a pre-assigned runner.quantized_model (quantizer=None) in addition to building from a quantizer. Validate quantization_config and rotated-checkpoint consistency on the pre-loaded path, track applied post-processes, and append them to quantization_config["onecomp_post_processes"] on save so config.json records the full post-process history across save/load cycles. - post_process/_base: add build_metadata() to produce JSON-serializable {name, class, config} audit metadata for each post-process. * [fix] accumulate post-process metadata until save * [refactor] centralize quantization_config validation in utils Extract the quantization_config schema checks that were duplicated across the save and load paths into reusable helpers. - Add validate_quant_config(): dict-level check requiring "quant_method" and "modules_in_block_to_quantize", raising ValueError with a caller context label. - Add validate_quantized_model_config(): model-level wrapper that pulls quantization_config off model.config and delegates to the dict check. - Both paths now enforce identical required keys and raise the same exception type. * [refactor] make PostQuantizationProcess.run a metadata-recording template method Turn run() into a template method shared by every post-process so audit metadata is recorded consistently regardless of how the process is invoked. - Add _runtime.py with shared helpers: prepare_quantized_model_for_post_process() (validate + move to CPU), validate_rotated_checkpoint_consistency(), append_post_process_metadata(), and the POST_PROCESS_HISTORY_KEY constant. - run() now validates and moves the model to CPU, calls the subclass body, restores eval()/cpu() even if it raises, and on success appends an audit entry to quantization_config["onecomp_post_processes"]. - Add build_metadata() producing a JSON-serializable entry per process. - Make _run() the abstract method subclasses implement; rename BlockWisePTQ.run and PostProcessLoraSFT.run to _run accordingly. * [update] validate and reattach quant_config on quantized model load Make the load path symmetric with the save path and allow a loaded model to be refined and re-saved. - Validate config.json quantization_config via validate_quant_config() so loading enforces the same schema and exception type as saving. - Reattach quantization_config to model.config after loading, so callers can post-process and re-save without separately reading config.json. - Accept device_map=None/"" (typed Optional[str]) to leave the model on CPU. * [update] support post-processing a preloaded quantized model in Runner Enable the load -> post-process -> re-save flow and reuse the shared validation helpers. - run_post_processes(): use runner.quantized_model when already assigned, otherwise build one from quantizer.results; raise a clear error if neither is available. - save_quantized_model(): require model_config and validate the model's quantization_config via validate_quantized_model_config() before saving a preloaded model. - Remove the now-extracted validation helpers (moved to utils/quant_config and post_process/_runtime). - Document that quantizer may be None when driving run_post_processes() through runner.quantized_model. * [update] document pack_weights behavior in save_quantized_model docstring Clarify the two save paths in save_quantized_model(). When runner.quantized_model is already set (after run_post_processes(), or a load -> post-process -> re-save flow), the model is saved as-is and the pack_weights argument is ignored — the weights keep whatever packing layout they were built with. Note that pack_weights=True must be set on create_quantized_model() beforehand to produce a loadable checkpoint, and that pack_weights only applies when building from quantizer.results. Docstring only; no behavior change. * [update] add tests for quant config and post-process metadata * [refactor] rename GlobalPTQ/GlobalPTQDistributed run() to _run() Rename run() -> _run() so both processes run through the shared PostQuantizationProcess.run() template (base entry validation + audit-metadata recording) instead of overriding run() directly. Existing cleanup inside the methods is left unchanged. * [update] build packed quantized model by default in run_post_processes run_post_processes() built the auto-created quantized model with pack_weights=False, so post-process -> save_quantized_model flows that did not pre-assign runner.quantized_model produced an unpacked (vLLM-unloadable) checkpoint, diverging from create_quantized_model()'s pack_weights=True default. Build with pack_weights=True so Runner-managed post-process flows keep packed buffers by default. This matches the save path for post-processes that preserve the quantized layer structure, such as BlockWisePTQ and GlobalPTQ: GPTQLinear unpacks at forward time, and the GPTQ post-process adapters read and write weights through _weight_is_packed-aware helpers. Workflows that require an unpacked layout can still build and pre-assign a model with create_quantized_model(pack_weights=False, use_gemlite=False). Also expand the create_quantized_model / run_post_processes / save_quantized_model docstrings to document the packed-by-default behavior and the pack_weights=False escape hatch for unpacked layouts. * [update] revise post-process examples for packed save and reload flows Align the post-process examples with run_post_processes() now building packed buffers by default and with structure-preserving save/load. - example_blockwise_ptq.py: switch to the Runner-managed packed path (run() -> run_post_processes()) and save with save_quantized_model(). - example_global_ptq.py: save the optimised model in both safetensors and .pt, add a commented DBF(target_bits=1.5) quantizer option (with dbf_lr), and add a commented direct post_process.run() path. - example_global_ptq_dbf.py: import DBF from the top-level package and save in both safetensors and .pt, matching the other GlobalPTQ examples. - example_global_ptq_distributed.py: also save a safetensors copy. - example_lora_sft_knowledge.py: build with the packed default and use_gemlite=False; make device selection CUDA-optional. - Add example_blockwise_ptq_unpacked.py and example_global_ptq_unpacked.py for explicit pack_weights=False research/debug workflows. - Add example_reload_post_process_resave.py for the end-to-end load -> post-process -> re-save flow with accumulated metadata. * [update] document load -> post-process -> re-save and packed-by-default Document the new save/load behavior across docstrings, the API reference, and the user guide. - runner.py: clarify run_post_processes() (packed by default) and save_quantized_model_pt() (custom-module case; prefer save_quantized_model for structure-preserving post-processes). - quantized_model_loader.py: note that BlockWisePTQ / GlobalPTQ / GlobalPTQDistributed outputs load via the safetensors path, and add a Raises: section for the required quant_method / modules_in_block_to_quantize keys. - docs/api/post_process.md: drop the stale subclass "members: - run" filters now that run() lives on the base template method. - docs/api/runner.md: list run_post_processes in the API reference. - docs/api/quantized_model_loader.md: show device_map=None CPU loading. - docs/user-guide/post-process.md, examples.md, README.md: add the load -> post-process -> re-save flow, packed/unpacked guidance, the onecomp_post_processes audit trail, and links to the new examples. * [update] add v1.3.0(WIP)+feature/blockwise_save_load changelog for load->post-process->re-save, post-process metadata, and quant_config validation * [add] BlockWisePTQ+GlobalPTQ post-process chain examples * [update] document BlockWisePTQ+GlobalPTQ chain examples in changelog * [fix] Remove duplicate create_quantized_model call in run_post_processes * [docs] document PostQuantizationProcess.run() breaking change * [style] Apply black formatter * [docs] note that Runner.check() is not intended for the load -> run_postprocess flow * [update] lighten BlockWisePTQ config in blockwise+global PTQ examples * [refactor] drop unpacked post-process examples; document the path in docstrings - Remove example/post_process/example_blockwise_ptq_unpacked.py and example/post_process/example_global_ptq_unpacked.py - Drop the unpacked-example guidance comments from the packed examples - Document the unpacked pack_weights=False path (with required bit widths and a short example) in the BlockWisePTQ and GlobalPTQ docstrings - Update README.md and docs/user-guide/{post-process,examples}.md to point at the API reference instead of the removed scripts - Add tests/onecomp/post_process/test_post_process_unpacked.py: a CPU-only, no-download test that feeds BlockWisePTQ.run() and GlobalPTQ.run() unpacked GPTQLinear layers built via the real GPTQ/RTN/JointQ inference-layer builders for the mandatory-unpacked bit widths (GPTQ 5-bit, RTN 5-bit, JointQ 1-bit), and asserts each run() accepts them and leaves the layers unpacked, on CPU, in eval mode, with post-process metadata recorded * [update] drop provisional .pt save from GlobalPTQ examples - Remove save_quantized_model_pt() (with save_dir_pt and the ".pt saved" print) from example_global_ptq.py, example_global_ptq_dbf.py, and example_global_ptq_distributed.py, leaving save_quantized_model() safetensors as the single standard flow - Reword the save comments and module-docstring step lists to drop the "either way" / ".pt formats" wording - Update the README table entry for example_global_ptq.py to "safetensors save" * [docs] align LoRA save/load docs with the safetensors + adapter-sidecar flow - post-process.md: rewrite "Saving and Loading LoRA Models" to lead with save_quantized_model()/load_quantized_model(); demote .pt to a legacy note; fix the save-method comparison table; turn the vLLM "Limitations" warning (LoRA not supported) into an info note (LoRA IS servable via the sidecar) pointing at example_lora_gptq_vllm_inference.py. - examples.md: update the LoRA save/load snippet to the safetensors path, keeping .pt as a legacy note. - docs/api/quantized_model_loader.md: note that load_quantized_model() auto-applies the LoRA sidecar; mark the .pt loader as legacy. - README.md: list example_lora_gptq_vllm_inference.py. - runner.py: fix the save_quantized_model_pt() docstring (rendered into docs/api/runner.md) that steered LoRA users to the .pt format * [update] record executed/reason in post-process audit metadata Let post-process _run() return an optional result dict and mirror its executed / global_executed and reason fields into quantization_config["onecomp_post_processes"]. GlobalPTQ and GlobalPTQDistributed now return these on early returns (not_quantized, unsupported_method_<method>, no_params) so skipped runs are recorded with executed: false and a reason instead of being indistinguishable from successful runs. * [docs] update CHANGELOG.md for executed/reason early-return audit metadata * [docs] use actual API names in Runner.check() flow note Replace the nonexistent load() and run_postprocess() references with load_quantized_model() and Runner.run_post_processes(). * [style] Apply pre-commit fixes * [test] Align full-wrapper save fixtures with quant config validation - add realistic GPTQ quantization configs to save-path fixtures - verify full-wrapper config remapping and default-format pass-through - update the changelog * [docs] Clarify post-process save/load and fix LoRA docstrings * [docs] Correct CHANGELOG entries against the upstream/develop/v1-3-0 baseline --------- Co-authored-by: sikoji <seiichiro.kojio@compmind.co.jp>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.