CODEWITHSUNDEEP
pythonamazon-web-servicesboto3mechanicalturk
Kyle
Kyle

Reputation: 351

When creating an MTurk HIT with AssignmentReviewPolicy, HITs are rejected incorrectly

TL;DR: My AssignmentReviewPolicy is leading to all HITs being rejected. How do I change my policy to accept when "defendant" radio button is selected?

Details:

I include a golden question in my HITs to allow auto approval or rejection via the AssignmentReviewPolicy. I'm using only qualified workers for whom this question would be trivial.

My expected output is auto-approving the HIT when someone selects the radio button with value="defendant" for the question "checker". I am following this tutorial to set the AssignmentReviewPolicy.

My Python, shortened for clarity:

hit = client.create_hit(
    Reward='999',LifetimeInSeconds=36000, AssignmentDurationInSeconds=3600, MaxAssignments=2,
    Title='strings',
    Description='desc',
    Keywords='keywords',
    AutoApprovalDelayInSeconds=259200, #259200 sec = 3 days
    Question=question,
    AssignmentReviewPolicy={
        'PolicyName':'ScoreMyKnownAnswers/2011-09-01',
        'Parameters':[
            {'Key':'AnswerKey', 'MapEntries':[
                {'Key': 'checker', 
                 'Values':['defendant']
                }]},
            {'Key': 'ApproveIfKnownAnswerScoreIsAtLeast', 'Values':['1']},
            {'Key': 'RejectIfKnownAnswerScoreIsLessThan', 'Values':['1']},
            {'Key': 'RejectReason', 
             'Values':['Sorry, we could not approve your submission.']},
            {'Key': 'ExtendIfKnownAnswerScoreIsLessThan','Values':['1']}
        ]
    }
)

My HTML:

<!-- Multiple Radios -->
<div class="form-group">
    <label class="col-md-4 control-label" for="checker"> What is the role of {company_name}?</label>
    <div class="col-md-4">
                <input type="radio" name="checker" value="plaintiff"> Plaintiff<br>
                <input type="radio" name="checker" value="defendant"> Defendant<br>
                <input type="radio" name="checker" value="judge"> Judge<br>
                <input type="radio" name="checker" value="attorney"> Attorney<br>
    </div>
</div>

The unexpected behavior is that no matter which option is selected, including "defendant", the answer is rejected.

I tried using client.list_review_policy_results_for_hit(HITId='') and I'm not sure what to take away from the output:

{'HITId': '',
 'AssignmentReviewPolicy': {'PolicyName': 'ScoreMyKnownAnswers/2011-09-01'},
 'AssignmentReviewReport': {'ReviewResults': [{'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'SubjectId': '',
    'SubjectType': 'Assignment',
    'QuestionId': '',
    'Key': 'KnownAnswerScore',
    'Value': '0'},
   {'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'SubjectId': '',
    'SubjectType': 'Assignment',
    'QuestionId': 'checker',
    'Key': 'AgreedWithKnownAnswer',
    'Value': 'false'}],
  'ReviewActions': [{'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'ActionName': 'review',
    'TargetId': '',
    'TargetType': 'Assignment',
    'Status': 'Succeeded',
    'Result': 'Assignment 3VBEN272OTK0OGFSGT KnownAnswerScore is 0%. Assignment rejected as a result. HIT extended as a result.'},
   {'ActionId': '38MQYK43N4YBSE3EAWAYNKWE10LP2D',
    'ActionName': 'reject',
    'TargetId': '3VBEN272OTK0OGFSGT',
    'TargetType': 'Assignment',
    'Status': 'Succeeded'},
   {'ActionId': '3SBIQ36PU450DWZXEWO36U167ELP3O',
    'ActionName': 'extend',
    'TargetId': '3QX22DUVPRJEBK749RF6N1SOE16VME',
    'TargetType': 'HIT',
    'Status': 'Succeeded',
    'Result': 'Extended by 1 assignment'}]},
 'NextToken': 'p1:HpeunOuRoWcvUyy1vi8zwEVMVptX3tv5BkWSag==',
 'ResponseMetadata': {'RequestId': '2f89af041eece',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amzn-requestid': '2f89af0b-667dc841eece',
   'content-type': 'application/x-amz-json-1.1',
   'content-length': '1260',
   'date': 'Tue, 24 Sep 2019 20:23:24 GMT'},
  'RetryAttempts': 2}}

Upvotes: 2

Views: 167

Answers (1)

Kyle
Kyle

Reputation: 351

The problem was with the format that radiobuttons return in answers. The tutorial represented this incorrectly.

The actual answer came back formatted as:

OrderedDict([('QuestionIdentifier',
              'checker.defendant'),
             ('FreeText', 'true')]),

Meaning the correct AnswerKey should be:

{'Key': 'AnswerKey', 'MapEntries': [
    {'Key': 'checker.defendant',
     'Values': ['true']
     }]},

Upvotes: 1

Related Questions