PaCA changes which connections are trained

PaCA updates selected input columns of a pretrained linear weight instead of introducing a trainable low-rank branch. Its implementation retains only the corresponding input channels for backward computation. The research predates this release: the news is its integration into PEFT, rather than a newly invented training method.

The documentation supports linear layers, including quantized ones, and describes conversion to a LoRA adapter for compatible inference stacks. It also lists important limits: mixed-adapter batches are unsupported and training with sharded weights has not yet been tested. Hooks attached to the original base layer are not called through the PaCA forward path.

LoRA initialization becomes a task-specific choice

Astra uses a small calibration dataset from the intended task to estimate output-activation covariance. Its initialization uses the low-variance eigenspace of that covariance. The documented sequence is to set explicit target modules, run preprocess_astra, and then inject the adapter with get_peft_model.

NoRA provides another initialization option, while ordinary LoRA’s zero-initialized B matrix starts with no adapter contribution. Our assessment: compare initialization methods on the same held-out examples and training budget. A task-aware starting point is a hypothesis to evaluate, not evidence that a specialized assistant is already reliable.

A dummy adapter helps prepare multiple specializations

The hotswap documentation explains create_dummy_lora_config: it examines compatible adapter configurations and reserves target-layer coverage and ranks before weights are switched. This can help a service prepare for several task adapters without loading their weights during configuration.

The trade-off is potentially higher reserved adapter memory. The helper requires explicit target-module lists and the same base weights; regular-expression targets and all-linear are unsupported. It is not a universal converter between unrelated models or adaptation methods.

Distributed exports need participation from every rank

The FSDP guide specifies that distributed adapter export is a collective operation. Every participating rank must call save_pretrained or get_peft_model_state_dict, while only the writing rank receives is_main_process=True. Unselected base-model weights remain sharded rather than being gathered unnecessarily.

PEFT 0.22 requires Python 3.11 or newer. Its release notes recommend Transformers 5.17 or newer for the TP and FSDP2 improvements. Before upgrading a training environment, validate adapter export and checkpoint resume on a small representative run. The documentation linked here follows main and can evolve after the release.