Indian Institute of Technology Madras
BS Degree Programme · Examination & Study Paper
Deep Learning Practice
Quiz 1 · 26 Oct 2025 (Afternoon)
Examination Instructions & Guidelines
- Solve on notebook/paper first before checking solutions.
- MCQ: Exactly 1 option is correct. Negative marking applies.
- MSQ: One or more options correct. Select all to score.
- NAT: Numerical response. No negative marking.
- Scientific calculator allowed for all calculations.
- Official answer key is sealed at the end of this sheet.
- Q1MCQ2 Marks·Choose one option
Which of the following statements best describes the primary advantage of the SentencePiece tokenizer compared to a standard BPE implementation?
- AIt is significantly faster to train because it does not need to count pairs.
- BIt results in a smaller vocabulary size by always merging the shortest tokens first.
- CIt is inherently language-agnostic, treating text as a raw stream of Unicode characters, which is ideal for languages without clear word delimiters like Japanese or Thai.
- DIt is deterministic and always produces the same tokenization for a given string,unlike BPE which can be probabilistic.
- Q2MCQ2 Marks·Choose one option
When performing full fine-tuning of a large language model (e.g., 10B parameters) using Adam, which component consumes the most GPU memory?
- AThe model’s weights (parameters).
- BThe gradients calculated for each parameter.
- CThe optimizer states (e.g., momentum and variance).
- DThe vocabulary and embedding matrix.
- Q3MCQ2 Marks·Choose one option
A company wants to align its chatbot with values of being helpful, harmless, and honest. Human labelers provide ideal responses and rank AI-generated outputs. Which adaptation technique is designed for this?
- AInstruction Tuning
- BZero-shot prompting
- CReinforcement Learning from Human Feedback (RLHF)
- DContinued pre-training
- Q4MCQ3 Marks·Choose one option
- A
- B
- C
- D
- A
- Q5MCQ3 Marks·Choose one option
- A
- B
- C
- D
- A
- Q6MCQ3 Marks·Choose one option
What is a key implication of using a Parameter-Efficient Fine-Tuning (PEFT) method like LoRA when adapting a large language model for a new task?
- AInference speed is 10x faster.
- BOnly a small number of new parameters are trained while freezing original weights.
- CIt eliminates the need for labeled data.
- DModel must be retrained from scratch.
- Q7MCQ4 Marks·Choose one option
The WordPiece tokenization algorithm, unlike BPE, does not merge the pair with the highest frequency. Instead, it merges the pair that maximizes a likelihood score. Which of the following statements accurately describes this score and its implication?
- A
- B
- C
- D
- A
- Q8MSQ4 Marks·Choose all correct options
Which of the following statements accurately describes the Causal Language Modeling (CLM) objective used to pre-train models like GPT? (Select ALL that apply)
- APredicting randomly masked tokens.
- BAuto-regressive next-token prediction.
- CRequires causal attention mask.
- DSuited for encoder-only models.
- EMaximizes joint probability of sequence.
- Q9MSQ4 Marks·Choose all correct options
A research lab has access to a powerful 175B parameter language model. They need to adapt it for a highly specialized legal text analysis task, but they only have about 500 labeled examples and limited access to high-end GPUs for fine-tuning. Which of the following are viable and computationally efficient adaptation strategies? (Select ALL that apply)
- AFull fine-tuning
- BZero-shot prompting
- CFew-shot in-context learning
- DLoRA (PEFT)
- EPre-training from scratch
- Q10MSQ4 Marks·Choose all correct options
A team is fine-tuning a 7B parameter model on a single GPU with 24GB of memory. They are using the Adam optimizer (which stores 2 states per parameter) and 32-bit precision (4 bytes per parameter/ state/gradient). They find that they run out of memory even with a batch size of 1. Which of the following strategies could help them complete the fine-tuning process on this GPU? (Select ALL that apply)
- AIncrease learning rate
- BUse LoRA
- CUse quantization (8/4-bit)
- DSwitch to SGD
- EGradient accumulation
- Q11MSQ4 Marks·Choose all correct options
The evolution of NLP models shows a distinct shift from task-specific architectures to a ”pre-train, finetune” paradigm, and now towards large-scale, general-purpose models. Which of the following accurately represents this evolution and the capabilities at each stage? (Select ALL that apply)
- AThe earliest models (e.g., n-grams) were statistical, required task-specific design, and had limited generalization capacity.
- BThe ”pre-train, fine-tune” era (e.g., BERT, GPT) introduced transfer learning, where a model was first trained on a general language task and then fully adapted to a specific downstream task.
- CModern Large Language Models (LLMs like GPT-4) exhibit ”emerging abilities,” allowing them to perform new tasks with zero or few examples (in-context learning) without any weight updates.
- DWord2vec was a complete language model capable of generating text, similar to GPT.
- EThe primary innovation of transformers over RNNs was the use of recurrent connections, which made them more efficient to train on parallel hardware.
- Q12MSQ4 Marks·Choose all correct options
The three main families of Transformer-based models are Encoder-only (e.g., BERT), Decoder-only (e.g.,GPT), and Encoder-Decoder (e.g., T5, BART). Match the architecture to its most suitable pre- training objective and typical use case. (Select ALL that apply)
- AEncoder-only models are best for natural language understanding tasks (like sentiment classification) and are often pre-trained with a Masked Language Modeling (MLM) objective.
- BDecoder-only models are ideal for text generation tasks and are pre-trained with a Causal Language Modeling (CLM) objective.
- CEncoder-Decoder models are most suitable for sequence-to-sequence tasks like translation or summarization.
- DAll three architectures are pre-trained using the same Causal Language Modeling objective.
- Q13NAT3 Marks·Write the number
Your answer: - Q14NAT4 Marks·Write the number
A single transformer block in a GPT-style model has the following configuration: embedding dimension (d model) = 1024, num attention heads = 16. Each attention head has a dimension of d model /num attention heads. Calculate the total number of parameters (weights and biases) for the self attention mechanism (specifically the Q, K, V, and Output projection layers) within this single block. Report the answer in millions, rounded to one decimal place.(in M)
Your answer: - Q15NAT4 Marks·Write the number
You are given a GPT-style model with a vocabulary size of 50,257, a context length of 1024, and an embedding dimension of 768. The model learns its positional embeddings rather than using fixed sinusoidal ones. Calculate the total number of parameters in the model’s embedding layer, which includes both token embeddings and positional embeddings. Report your answer in millions, rounded to one decimal place.(in M)
Your answer: