Generative AI: Adapting LLMs with Parameter-Efficient Fine-Tuning

Adapting a large language model can mean changing its instructions, supplying external evidence, or training parameters for a target task. Parameter-efficient fine-tuning, or PEFT, makes the training option more practical by updating a restricted set of parameters. This lecture explains how instruction tuning and preference learning relate to PEFT, then derives and tests Low-Rank Adaptation, or LoRA.

Learning goals and prerequisites

You should finish able to choose an adaptation strategy, distinguish an objective from a parameterization, calculate LoRA parameter counts, explain its initialization and gradients, and plan a fine-tuning experiment with meaningful evaluation.

Read LLMs and Retrieval-Augmented Generation and Transformers and Self-Supervised Learning first. The worked matrices and local code below are synthetic. No foundation model, dataset, or measured LLM-training result is supplied here.

1. The adaptation ladder is a decision aid

Adaptation tailors a pretrained model or surrounding application to a task. A useful starting sequence is:

Option	What changes	Useful when
Zero-shot prompting	Instructions and output specification	A clear task may already be within the model’s capabilities
Few-shot prompting	Demonstrations in the context	Examples clarify format or decision boundaries
RAG	Relevant external evidence supplied at inference	Answers need accessible, current, attributable information
Fine-tuning	Learned model parameters or adapters	Repeated behavior remains inconsistent despite a well-designed application

This is not a rule that every task must climb all four rungs. Fine-tuning and RAG can be combined, and retrieval need not use a dedicated vector database. Choose the intervention from observed errors: missing current facts suggest retrieval; a stable formatting or task-behavior problem may justify training.

Ordinary inference-time prompting and RAG leave the generator’s weights fixed. Fine-tuning runs optimization. A hosted model’s name alone does not establish that its internal weights are accessible for local LoRA; select a compatible, licensed checkpoint and toolchain if you intend to train locally.

2. Base models and instruction-following models

A base autoregressive model learns to predict subsequent tokens. That objective can produce useful language representations, but does not directly specify “answer the user’s request helpfully.” A base model may continue the prompt as text rather than behave like a conversational assistant.

Supervised fine-tuning

Supervised fine-tuning, or SFT, trains on demonstrations of desired responses. An example may contain an instruction, optional input, and target output, or a multi-turn conversation with roles.

For prompt x and target response y₁,…,y_T, a response-token objective is:

L_SFT=−(1/T)Σₜ log Pθ(yₜ | x,y₁,…,yₜ₋₁).

Many assistant-training pipelines score only response tokens, masking prompt and padding positions from the loss. Other recipes score a larger portion of the sequence. State the policy explicitly, use the checkpoint’s chat template, and check whether the library shifts next-token targets internally.

High-quality demonstrations specify what good behavior means. More examples are not automatically better if they are duplicated, inconsistent, mislabeled, or drawn from a different task. Evaluate the exact behavior you are trying to change.

Reinforcement learning from human feedback

A classic RLHF pipeline collects human preferences between candidate responses, trains a reward model to predict those preferences, and optimizes the language model against that learned reward. A constraint or penalty can discourage excessive deviation from a reference model.

A simplified preference model is:

P(y_preferred ≻ y_other | x)=sigmoid(rφ(x,y_preferred)−rφ(x,y_other)).

If the reward difference is log 3, that probability is 3/4. Training a reward model learns a proxy for annotated preferences, not an infallible measure of truth or safety. Optimizing the proxy can expose weaknesses, making independent evaluation important.

The InstructGPT paper documents a historical SFT-plus-RLHF approach. This is not a mandatory two-stage recipe for every assistant. Direct preference optimization, or DPO, is an example of a different preference-learning objective that avoids a separately trained explicit reward model and the usual online reinforcement-learning loop. A thumbs-up click is also not automatically an immediate model-weight update; collecting feedback and training on it are separate processes.

3. Objective and parameter efficiency are separate choices

SFT, next-token pretraining, and preference optimization describe what the model learns from. PEFT describes which parameters or added structures are trained. LoRA can be used with different objectives; it is not itself an alternative definition of instruction following or alignment.

Full-parameter fine-tuning updates the base model’s weights. PEFT methods include low-rank updates, inserted adapter modules, and learned soft prompts or prefixes. Soft prompt tuning optimizes vectors; it differs from manually writing a few-shot text prompt.

Freezing most pretrained parameters can reduce storage for gradients and optimizer state. It does not remove the base model’s forward computation, and gradients may still need to propagate through frozen layers to reach trainable adapters.

4. Why full fine-tuning costs memory

Training memory includes base weights, trainable gradients, optimizer states, activations, and temporary buffers. Depending on precision and optimizer implementation, additional master-weight copies may be present.

For a hypothetical 7-billion-parameter model, two-byte base weights alone occupy about 14 GB in decimal units. Two float32 Adam moment arrays for all those parameters would add about 56 GB, before gradients, activations, or other storage. Actual requirements depend on sharding, offloading, precision, and implementation; this is accounting, not a hardware recommendation.

LoRA can avoid full-sized gradient and optimizer arrays for frozen base parameters. The saving in total GPU memory is therefore not equal to the reduction in trainable parameter count. Long sequences and large activation tensors can still dominate the run.

5. LoRA: constrain the update, not the base matrix

Let a linear layer use a frozen pretrained weight W₀∈ℝ^(d_out×d_in) and a column-vector input x. Ordinary adaptation could learn a full update ΔW. LoRA instead parameterizes:

W_eff=W₀+sBA,

where A∈ℝ^(r×d_in), B∈ℝ^(d_out×r), and typically s=α/r. The forward pass is:

h=W₀x+sB(Ax).

The update has rank at most r. The original W₀ need not be low rank, and a low-rank update need not have small numerical magnitude. LoRA does not guarantee that every useful fine-tuning change is low rank; it imposes a restricted, often effective hypothesis class.

The factors are learned directly during training. This does not require first fully fine-tuning W₀ and then factorizing its update. A higher r increases capacity and trainable storage but does not guarantee better validation performance.

                         frozen W₀
                    ┌───────────────┐
Input x ────────────┤      W₀x      ├──┐
   │                └───────────────┘  │
   └── A: d_in→r ── B: r→d_out ── ×s ── Add → h
        trainable      trainable

6. Count the trainable parameters correctly

A full matrix contains d_out d_in entries. The LoRA factors contain:

r d_in+d_out r=r(d_in+d_out).

For a square N×N matrix, the fraction is 2r/N. This excludes separately trainable biases or heads; count those too when reporting a complete model.

Correcting the 8,096-dimensional example

For N=8,096 and r=2:

• Original matrix: 8,096²=65,545,216 entries.
• A factor: 2×8,096=16,192 entries.
• B factor: 8,096×2=16,192 entries.
• Both factors: 32,384 trainable entries.
• Fraction: 32,384/65,545,216≈0.000494071, or 0.049407%.

The reduction is 2,024 times for this matrix. Counting 16,192 as the total accidentally omits one rank-2 factor; that total would instead match rank one with these dimensions. At r=1 the fraction is approximately 0.024704%; at r=8 it is approximately 0.197628%. A range such as 0.02–0.05% is consequently an example, not a universal LoRA property.

In a hypothetical 32-layer model adapting two 4,096×4,096 projections per layer at rank 8, the factors contain 32×2×8×(4,096+4,096)=4,194,304 trainable entries. Real architectures can use rectangular query/key/value projections, fused projections, or other target layers, so inspect shapes rather than extrapolating from an architecture name.

7. Initialization: zero update does not mean two zero factors

An ordinary LoRA initialization chooses A randomly and B as zeros. Then BA=0, so the model initially reproduces the base layer’s output, while gradients can start learning B. Implementations also offer other initialization schemes; this lesson uses the conventional one.

Let G=∂L/∂W_eff. Ignoring dropout and additional penalties:

∂L/∂B=sGAᵀ, and ∂L/∂A=sBᵀG.

With B=0 and A nonzero, A’s initial gradient is zero, but B’s can be nonzero. After B changes, A can receive a nonzero gradient too. If both A and B start at zero, both derivatives are zero under this ordinary objective. Standard gradient descent cannot start that adapter learning.

Thus “initialize ΔW to zero” is correct for the conventional start; “initialize both low-rank matrices to zero” is not. Keeping W₀ frozen also does not mean treating BA as a fixed correction: A and B remain trainable through the task loss.

8. A complete local LoRA update without an LLM

Use W₀=I₂, x=[1,2], target y=[0,0], rank r=1, scale s=1, A=[0.1,−0.2], and B=[0,0]ᵀ. Let L=(1/2)‖h−y‖² and use SGD learning rate 0.1.

Initially Ax=−0.3, h=[1,2], and L=2.5. The output gradient is [1,2], so ∂L/∂B=[−0.3,−0.6]ᵀ and ∂L/∂A=[0,0]. One simultaneous update gives B_new=[0.03,0.06]ᵀ and leaves A unchanged. The new prediction is [0.991,1.982], with loss 2.4552025.

This standard-library script checks the arithmetic and confirms that W₀ stays unchanged:

from math import isclose

W0 = [[1.0, 0.0], [0.0, 1.0]]
original_base = [row[:] for row in W0]
x, target = [1.0, 2.0], [0.0, 0.0]
A, B = [0.1, -0.2], [0.0, 0.0]  # Rank-one factors.
learning_rate = 0.1

def forward(a, b):
    z = sum(ai * xi for ai, xi in zip(a, x))
    base = [sum(wi * xi for wi, xi in zip(row, x)) for row in W0]
    return [base[i] + b[i] * z for i in range(2)]

def loss(prediction):
    return 0.5 * sum((p - y)**2 for p, y in zip(prediction, target))

before = forward(A, B)
g = [p - y for p, y in zip(before, target)]
z = sum(ai * xi for ai, xi in zip(A, x))
grad_B = [gi * z for gi in g]
grad_A = [sum(bi * gi for bi, gi in zip(B, g)) * xi for xi in x]
new_A = [a - learning_rate * da for a, da in zip(A, grad_A)]
new_B = [b - learning_rate * db for b, db in zip(B, grad_B)]
after = forward(new_A, new_B)

assert all(isclose(a, b) for a, b in zip(after, [0.991, 1.982]))
assert isclose(loss(after), 2.4552025)
assert loss(after) < loss(before)
assert W0 == original_base
assert grad_A == [0.0, 0.0] and any(value != 0 for value in grad_B)
print("Loss before/after:", loss(before), loss(after))

This is one small regression update, not language-model training. Its simple SGD step makes the calculation visible. Real LoRA training can use AdamW or other suitable optimizers; LoRA does not require vanilla SGD.

9. Apply LoRA to a compatible model

Attention projections are common targets, but LoRA can also adapt output projections, feed-forward matrices, and other suitable layers. Query/key/value matrices are not required targets in every experiment. Target names depend on the checkpoint architecture.

The following function illustrates Hugging Face PEFT configuration in an environment with compatible peft, transformers, and PyTorch packages. Pass an already loaded, compatible causal-language-model checkpoint; the function does not download or train one.

from peft import LoraConfig, TaskType, get_peft_model

def attach_lora(base_model):
    module_names = {name.rsplit(".", 1)[-1] for name, _ in base_model.named_modules()}
    targets = ["q_proj", "v_proj"]
    if not set(targets).issubset(module_names):
        raise ValueError("Inspect the architecture and choose its actual target modules")
    config = LoraConfig(
        task_type=TaskType.CAUSAL_LM,
        r=8, lora_alpha=16, lora_dropout=0.05,
        target_modules=targets,
        bias="none", init_lora_weights=True,
    )
    adapted = get_peft_model(base_model, config)
    adapted.print_trainable_parameters()
    trainable = [name for name, p in adapted.named_parameters() if p.requires_grad]
    if not trainable or not all("lora_" in name for name in trainable):
        raise ValueError("Audit unexpected trainable parameters before training")
    return adapted

The example intentionally requests adapter-only training. Some tasks also train a new head, biases, or selected token embeddings; configure, count, and save those deliberately rather than assuming that every non-LoRA parameter must always be frozen. Matching a module name is only a first check: inspect the actual modules, shapes, trainable counts, and a small forward/backward batch.

For SFT, prepare correctly templated instruction–response examples, define response and padding loss masks, and split related examples before training. Optimize only intended trainable parameters. Tune rank, target modules, scale, dropout, learning rate, and duration using validation evidence; keep the final test set independent.

10. QLoRA, merging, and deployment

QLoRA combines a quantized frozen base with trainable low-rank adapters. Low rank reduces trainable update storage; quantization reduces base-weight storage. They solve related but different memory problems. Computation and adapter storage do not simply become four-bit everywhere, and hardware/library compatibility still matters.

For a conventional compatible linear layer at inference, the update can be merged as W_merged=W₀+sBA. This avoids evaluating the separate adapter branch, although merging into quantized models requires care and supported tooling. Unmerged adapters can add operations; LoRA does not promise identical latency in every serving arrangement.

An adapter checkpoint generally depends on its exact base checkpoint. Save the base revision, tokenizer, chat template, adapter configuration, and any separately trained modules. Adapter-only weights are not a standalone replacement for the whole foundation model. Keep the original base if you need to switch adapters or return to its behavior.

Freezing the base does not guarantee preservation of its original behavior: the effective network includes the learned update. Measure regressions, instruction following, factual reliability, and task-specific errors. PEFT can overfit or learn undesirable patterns just as other training methods can.

11. A practical experiment plan

Start with a fixed prompt baseline and a held-out task evaluation. Check whether failures come from missing evidence, unclear label policy, or inconsistent model behavior before investing in training.

For a LoRA experiment, document the base revision and license, dataset and split, target layers and rank, true trainable count, precision, sequence length, loss masking, and optimizer. Compare task quality and general-behavior regressions with the baseline, alongside peak memory, wall-clock time, and inference cost.

Use a small batch to verify nonzero adapter gradients, unchanged frozen weights, valid token shifting, and padding exclusions. Then run the planned training and evaluate on held-out cases. A smaller checkpoint or trainable fraction is an efficiency result, not proof of better task performance.

12. Exercises with answers

1. Strategy: Should a frequently changing refund policy primarily live in an adapter? Usually start with retrieval of the authoritative policy; training does not provide a reliable update mechanism for every changing fact.
2. Objective versus method: Is LoRA the same thing as supervised instruction tuning? No. LoRA parameterizes trainable updates; SFT defines a learning objective and data format.
3. Count: For a 1,024×1,024 matrix at rank 4, how many factor entries are trained? 4×(1,024+1,024)=8,192, about 0.78125% of the original matrix.
4. Rectangular layer: For d_out=2,048, d_in=1,024, and r=8, what is the count? 8×(2,048+1,024)=24,576.
5. Scale: With α=16 and r=8, what is s under conventional α/r scaling? 2.
6. Rank: Does rank-two adaptation require the frozen base matrix to have rank two? No; only the added update is constrained to rank at most two.
7. Initialization: Why not initialize both factors to zero? Their ordinary task-loss gradients both vanish at that point, preventing learning from starting.
8. First step: With random A and zero B, which factor can initially receive a nonzero task-loss gradient? B; A can receive gradients after B changes.
9. Optimizer: Does LoRA require vanilla SGD? No; suitable optimizers such as AdamW can train the factors.
10. Deployment: Can an adapter checkpoint normally run without its base model? No; it depends on the compatible base weights and configuration.
11. Memory: Does a 2,024-fold parameter-count reduction guarantee a 2,024-fold GPU-memory reduction? No; base weights, activations, and other allocations remain.

Further reading and sources

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models: frozen pretrained weights and learned low-rank updates.
• Hugging Face PEFT LoRA documentation: target modules, configuration, initialization, and merging behavior.
• Ouyang et al., Training Language Models to Follow Instructions with Human Feedback: a historical SFT and RLHF pipeline.
• Rafailov et al., Direct Preference Optimization: a different preference-optimization formulation.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs: quantized frozen bases with low-rank adaptation.

The parameter counts and local update are constructed calculations. The PEFT configuration is an integration template requiring a compatible model and environment; no full-model fine-tuning result or hardware benchmark is claimed.

Feynman study lab

Explain it simply

LoRA learns a restricted change to a model’s weights using fewer trainable numbers.

Concrete example. A 100 × 100 matrix has 10,000 entries; a rank-4 update represented by 100 × 4 and 4 × 100 factors has 800 trainable entries.

Teach it back without looking

Close the lesson and explain the idea aloud to a beginner in two minutes. Define each technical term you use, walk through the example, and state one assumption or limitation. Then answer:

Does that mean the entire model needs only 800 stored numbers?

Check your explanation after you try

No. Base weights remain necessary. Savings depend on what is trained and stored; optimizer state, activations and inference setup also affect memory.

Find the gap and try again

If you hesitated, relied on a formula you could not explain, or missed the self-check, write down the exact gap. Revisit the relevant section of this lesson and work through its example. Close the notes, explain it again in simpler language, then change one number or assumption and predict what changes. Recalculate or check the lesson to test your prediction. Try once more tomorrow from memory.