chore: Make release validation more robust to transient failures (#344)

We see transient failures in our releases because sometimes the validation
cannot find the new version that we just released due to eventual consistency.
Retry a couple times before failing to more gracefully handle this scenario.

Author: farleyb-amazon

codebuild/release/validate.yml

@@ -13,4 +13,19 @@ phases:
       - sed -i "s/aws_encryption_sdk/aws_encryption_sdk==$VERSION/" requirements-dev.txt
   build:
     commands:
-      - tox -e test
+      - NUM_RETRIES=3
+      - |
+        while [ $NUM_RETRIES -gt 0 ]
+        do
+          tox -re test
+          if [ $? -eq 0 ]; then
+            break
+          fi
+          NUM_RETRIES=$((NUM_RETRIES-1))
+          if [ $NUM_RETRIES -eq 0 ]; then
+            echo "All validation attempts failed, stopping"
+            exit 1;
+          else
+            echo "Validation failed, retrying in 60 seconds; will retry $NUM_RETRIES more times" && sleep 60
+          fi
+        done