Approaches to Data Labelling CT-AI Practice Questions (Page 4)

Questions 16–20

1. 16foundation · medium

   What does inter-annotator agreement measure?

   • AThe consistency of labels assigned by different annotators to the same data
   • BThe accuracy of the ML model on the test set
   • CThe number of labels each annotator produces per hour
   • DThe time taken by annotators to label a batch of data
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A company is labelling a large dataset of customer emails for intent classification (complaint, inquiry, etc.). They have a small team of customer service agents who can label, but the volume is high. They are considering a semi-automated approach where a model pre-labels emails and agents review only the low-confidence ones. What is a potential risk of this approach that they must mitigate?

   • AThe agents will label the low-confidence predictions incorrectly, introducing errors.
   • BThe agents may become too reliant on the model and stop critically evaluating the labels.
   • CThe model may have a bias that is not present in the low-confidence predictions, so high-confidence errors go undetected.
   • DThe model will become more accurate over time, reducing the need for human review.
   Reveal Solution Discussion Select an answer first
4. 18expert · hard

   A startup is using a crowdsourcing platform to label a dataset of user-generated content for toxicity detection. They are concerned about the quality of labels and the potential for bias. They have a limited budget. What is the most effective way to balance cost and quality?

   • AUse a small number of highly qualified annotators and pay them a higher rate, rather than a large number of low-paid workers.
   • BUse a large number of low-paid workers and rely on majority vote to ensure accuracy.
   • CUse a single annotator per item to minimize cost, and accept the risk of lower quality.
   • DUse a pre-trained model to label all data and have crowd workers only review the low-confidence predictions.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A medical imaging startup is building a model to detect rare retinal diseases from fundus photographs. They have a small team of three ophthalmologists who can label images, but the volume is large and the conditions are rare. To maximize label quality while keeping the project feasible, what should they do?

   • AHave the ophthalmologists label all images independently, then use majority voting to resolve disagreements.
   • BUse a crowdsourcing platform to label all images, relying on the platform's scale to cover the rare conditions.
   • CHave one ophthalmologist label all images to ensure consistency, avoiding inter-annotator disagreements.
   • DUse a pre-trained model to auto-label all images, then have the ophthalmologists review only the low-confidence predictions.
   Reveal Solution Discussion Select an answer first
6. 20foundation · medium

   Which of the following is an example of rule-based automated labelling?

   • AUsing a pre-trained neural network to predict labels for new images
   • BAssigning the label 'spam' to any email containing the word 'lottery'
   • CA human annotator manually tagging each email as spam or not spam
   • DAsking multiple annotators to vote on the label for each item
   Reveal Solution Discussion Select an answer first