A team is training a very deep neural network that does not fit in the memory of a single GPU. They have access to multiple GPUs. What is the primary reason to use model parallelism?

To speed up training by using more devices

To increase the effective batch size

To reduce the communication overhead

To fit a model that exceeds the memory of a single device

A team is training a large transformer model with 10 billion parameters. They have a cluster of 16 TPU v3-8s. The model fits in the memory of a single TPU v3-8. The training dataset is extremely large. They want to minimize training time. What is the best parallelism strategy?

Pure model parallelism across all 16 TPUs

Hybrid: model parallelism for the embedding layer and data parallelism for the rest

Pure data parallelism across all 16 TPUs

In data parallelism, how is the training data distributed across multiple devices?

Each device trains on a different subset of features from the dataset

The dataset is replicated on each device, and devices train on the same mini-batch simultaneously

Each device receives a copy of the entire dataset and trains on all of it

The dataset is split into mini-batches, and each device processes a different mini-batch

A team is training a model on a cluster of 4 GPUs using data parallelism. They want to ensure that all GPUs have the same model parameters at the end of each training step. What mechanism ensures this?

Each GPU uses a different learning rate

The model parameters are copied from the first GPU to the others

Each GPU independently updates its model parameters

Gradient synchronization and averaging across all GPUs

Google CloudProfessional Machine Learning Engineer

Domain 3Objective 3

Choosing Appropriate Hardware for Training PROFESSIONAL-MACHINE-LEARNING-ENGINEER Practice Questions (Page 1)

Part of the Scaling prototypes into ML models domain, which accounts for ~21% of the PROFESSIONAL-MACHINE-LEARNING-ENGINEER exam.

20questions here

4free pages

5concepts

~21%of the exam

Questions 1–5

1application · easy
A team is training a very deep neural network that does not fit in the memory of a single GPU. They have access to multiple GPUs. What is the primary reason to use model parallelism?
Select an answer first
2expert · hard
A team is training a large transformer model with 10 billion parameters. They have a cluster of 16 TPU v3-8s. The model fits in the memory of a single TPU v3-8. The training dataset is extremely large. They want to minimize training time. What is the best parallelism strategy?
Select an answer first
3foundation · easy
In data parallelism, how is the training data distributed across multiple devices?
Select an answer first
4application · easy
A team is training a model on a cluster of 4 GPUs using data parallelism. They want to ensure that all GPUs have the same model parameters at the end of each training step. What mechanism ensures this?
Select an answer first
5foundation · easy
A machine learning engineer is training a model that fits entirely on a single GPU but wants to speed up training by using multiple GPUs. Which parallelism strategy is most appropriate?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to PROFESSIONAL-MACHINE-LEARNING-ENGINEER

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by Google Cloud. “PROFESSIONAL-MACHINE-LEARNING-ENGINEER” is a trademark of its owner, used for identification only.