Migration Failures from DOMjudge 7.x to 8.x: Hash Inconsistencies and Missing Judgetasks

[hqy2000]

Description of the problem

The migration scripts from 7.x to 8.x fail to migrate certain data:

1. They do not properly generate hash values for executables (see webapp/migrations/Version20210407120356.php), causing judgehosts to crash due to hash inconsistencies.
2. They fail to generate judgetasks for existing judgeruns. Therefore, when modifying global/contest configurations, maybeCreateJudgeTasks crashes due to the creation of duplicate values.

Your environment

DOMjudge 8.2.2

Steps to reproduce

1. Install DOMjudge 7.x.
2. Create some submissions and judgeruns.
3. Follow the Upgrading guide to upgrade to DOMjudge 8.x.
4. Create additional submissions and modify configuration values.

Expected behaviour

Submissions should be correctly judged, and configuration values should be updated without issues.

Actual behaviour

Judgehosts crash. The DOMserver returns a 500 error due to foreign key violations.

Any other information that you want to share?

We have attempted some fixes in our fork: 30b2606 and bf66294. However, these fixes are specific to our cases.

[tom93]

The first issue might be fixed by #2631.

[vmcj]

@hqy2000 can you provide your database dump so we can test with that?

[tom93]

For reference, here are the Docker commands I've been using to reproduce the first issue:

# Create the network:
docker network create dj

# Start MariaDB:
docker run -d --name dj-mariadb --net dj -e MYSQL_ROOT_PASSWORD=rootpw -e MYSQL_USER=domjudge -e MYSQL_PASSWORD=djpw -e MYSQL_DATABASE=domjudge -p 13306:3306 mariadb --max-connections=1000

# Run the DOMjudge 7.3.4 installer:
docker run -it --name domserver --net dj -e MYSQL_HOST=dj-mariadb -e MYSQL_USER=domjudge -e MYSQL_DATABASE=domjudge -e MYSQL_PASSWORD=djpw -e MYSQL_ROOT_PASSWORD=rootpw -p 12345:80 domjudge/domserver:7.3.4
# Write down the admin and judgehost passwords then kill the container.
# Dump the database:
docker exec dj-mariadb sh -c 'mariadb-dump domjudge -p"$MYSQL_ROOT_PASSWORD"' > /tmp/dump-7.3.4

# Upgrade to DOMjudge 8.2.3:
docker rm -f domserver
docker run -it --name domserver --net dj -e MYSQL_HOST=dj-mariadb -e MYSQL_USER=domjudge -e MYSQL_DATABASE=domjudge -e MYSQL_PASSWORD=djpw -e MYSQL_ROOT_PASSWORD=rootpw -p 12345:80 domjudge/domserver:8.2.3

# Start the judgehost:
JUDGEDAEMON_PASSWORD=... docker run -it --privileged -v /sys/fs/cgroup:/sys/fs/cgroup --name judgehost-0 --net dj --hostname judgedaemon-0 -e DAEMON_ID=0 -e JUDGEDAEMON_PASSWORD domjudge/judgehost:8.2.3
# (replace "..." with the judgehost password)

# Log in as user "dummy" and submit.
# Result: "Unexpected hash" error in the judgedaemon log.

I haven't managed to reproduce the second issue yet; my plan was to sign in while still running 7.3.4 and create a submission, then upgrade to 8.2.3.

[tom93]

I have a reproducer for the second issue:

1. Run domserver 7.3.4, sign in as user "dummy", submit.
2. Run judgehost 7.3.4 so it will judge the submission.
3. Run domserver 8.2.3, sign in as "admin", edit the demo contest, change the start time to be 1 minute later, save.

Result:

500 Internal Server Error
Sorry, an error has occurred: Internal Server Error.
Please contact a staff member for assistance.
An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-1' for key 'testcaseid'

Relevant output of symfony-log (line breaks added for readability):

request.CRITICAL:
Uncaught PHP Exception Doctrine\DBAL\Exception\UniqueConstraintViolationException: "An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-1' for key 'testcaseid'"
at /opt/domjudge/domserver/lib/vendor/doctrine/dbal/src/Driver/API/MySQL/ExceptionConverter.php line 62
{"exception":
"[object] (Doctrine\DBAL\Exception\UniqueConstraintViolationException(code: 1062): An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-1' for key 'testcaseid'
at /opt/domjudge/domserver/lib/vendor/doctrine/dbal/src/Driver/API/MySQL/ExceptionConverter.php:62)\n
[previous exception] [object] (Doctrine\DBAL\Driver\PDO\Exception(code: 1062): SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-1' for key 'testcaseid'
at /opt/domjudge/domserver/lib/vendor/doctrine/dbal/src/Driver/PDO/Exception.php:28)\n
[previous exception] [object] (PDOException(code: 23000): SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-1' for key 'testcaseid'
at /opt/domjudge/domserver/lib/vendor/doctrine/dbal/src/Driver/PDO/Statement.php:134)"
} []

Interestingly, if I try to edit the contest again, it succeeds.

[hqy2000]

@hqy2000 can you provide your database dump so we can test with that?

Sorry, we might not be able to provide our database dump as it contains lots of student information. However, tom93's reproducer should cover our case.

[vmcj]

@tom93 are you willing to fix this further? We already have #2577 to prevent this issue in the future.

[tom93]

@vmcj I had a look but it was too complicated for me; I'd need some hints. I think the relevant commit is d04f69c (First part of the new judgehost API., 2020-12-19), from #888.

[vmcj]

@vmcj I had a look but it was too complicated for me; I'd need some hints. I think the relevant commit is d04f69c (First part of the new judgehost API., 2020-12-19), from #888.

No problem, we'll discuss this after this WF on how to fix this.