Indian Institute of Technology Madras
BS Degree Programme · Examination & Study Paper
Deep Learning Practice
Tokenizers & LLM pre-training
Examination Instructions & Guidelines
- Solve on notebook/paper first before checking solutions.
- MCQ: Exactly 1 option is correct. Negative marking applies.
- MSQ: One or more options correct. Select all to score.
- NAT: Numerical response. No negative marking.
- Scientific calculator allowed for all calculations.
- Official answer key is sealed at the end of this sheet.
- Q1MCQ2 Marks·Choose one option
Which of the following statements best describes the primary advantage of the SentencePiece tokenizer compared to a standard BPE implementation?
- AIt is significantly faster to train because it does not need to count pairs.
- BIt results in a smaller vocabulary size by always merging the shortest tokens first.
- CIt is inherently language-agnostic, treating text as a raw stream of Unicode characters, which is ideal for languages without clear word delimiters like Japanese or Thai.
- DIt is deterministic and always produces the same tokenization for a given string,unlike BPE which can be probabilistic.
- Q2MCQ4 Marks·Choose one option
The WordPiece tokenization algorithm, unlike BPE, does not merge the pair with the highest frequency. Instead, it merges the pair that maximizes a likelihood score. Which of the following statements accurately describes this score and its implication?
- A
- B
- C
- D
- A
- Q3MSQ4 Marks·Choose all correct options
Which of the following statements accurately describes the Causal Language Modeling (CLM) objective used to pre-train models like GPT? (Select ALL that apply)
- APredicting randomly masked tokens.
- BAuto-regressive next-token prediction.
- CRequires causal attention mask.
- DSuited for encoder-only models.
- EMaximizes joint probability of sequence.
- Q4MSQ4 Marks·Choose all correct options
The evolution of NLP models shows a distinct shift from task-specific architectures to a ”pre-train, finetune” paradigm, and now towards large-scale, general-purpose models. Which of the following accurately represents this evolution and the capabilities at each stage? (Select ALL that apply)
- AThe earliest models (e.g., n-grams) were statistical, required task-specific design, and had limited generalization capacity.
- BThe ”pre-train, fine-tune” era (e.g., BERT, GPT) introduced transfer learning, where a model was first trained on a general language task and then fully adapted to a specific downstream task.
- CModern Large Language Models (LLMs like GPT-4) exhibit ”emerging abilities,” allowing them to perform new tasks with zero or few examples (in-context learning) without any weight updates.
- DWord2vec was a complete language model capable of generating text, similar to GPT.
- EThe primary innovation of transformers over RNNs was the use of recurrent connections, which made them more efficient to train on parallel hardware.
- Q5MSQ4 Marks·Choose all correct options
The three main families of Transformer-based models are Encoder-only (e.g., BERT), Decoder-only (e.g.,GPT), and Encoder-Decoder (e.g., T5, BART). Match the architecture to its most suitable pre- training objective and typical use case. (Select ALL that apply)
- AEncoder-only models are best for natural language understanding tasks (like sentiment classification) and are often pre-trained with a Masked Language Modeling (MLM) objective.
- BDecoder-only models are ideal for text generation tasks and are pre-trained with a Causal Language Modeling (CLM) objective.
- CEncoder-Decoder models are most suitable for sequence-to-sequence tasks like translation or summarization.
- DAll three architectures are pre-trained using the same Causal Language Modeling objective.
- Q6NAT4 Marks·Write the number
A single transformer block in a GPT-style model has the following configuration: embedding dimension (d model) = 1024, num attention heads = 16. Each attention head has a dimension of d model /num attention heads. Calculate the total number of parameters (weights and biases) for the self attention mechanism (specifically the Q, K, V, and Output projection layers) within this single block. Report the answer in millions, rounded to one decimal place.(in M)
Your answer: - Q7NAT4 Marks·Write the number
You are given a GPT-style model with a vocabulary size of 50,257, a context length of 1024, and an embedding dimension of 768. The model learns its positional embeddings rather than using fixed sinusoidal ones. Calculate the total number of parameters in the model’s embedding layer, which includes both token embeddings and positional embeddings. Report your answer in millions, rounded to one decimal place.(in M)
Your answer: