Skip to content
This repository was archived by the owner on Mar 4, 2025. It is now read-only.

Commit 33108b1

Browse files
skyhitskyhit
committed
Add Guide for Submission System (#1)
1 parent b299fd9 commit 33108b1

File tree

9 files changed

+454
-0
lines changed

9 files changed

+454
-0
lines changed

‎submission-system/README.md

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Purpose of this Document
2+
3+
The purpose of this document is to describe how to setup your development environment and run the submission system locally.
4+
5+
# Overview
6+
7+
The submission system uses multiple AWS services and Topcoder APIs to handle member submissions, we have a solution to execute all those services using mocks and fake versions of the service locally.
8+
9+
We will be using Docker and docker-compose to run or mock up the following services:
10+
11+
1. File microservice and S3: Used to upload and download submission files;
12+
2. DynamoDB: Main submission datastore;
13+
3. Informix: Legacy datastore, data is update in Informix to allow legacy applications to work with the submissions;
14+
4. NFS: legacy file storage;
15+
5. Challenge microservice: api to interact with challenge information;
16+
6. Member microservice: api to interact with member information;
17+
7. SQS: queueing service used to send and receive submission process requests;
18+
8. Kafka: queueing service used to send process failure notifications;
19+
9. Submissions UI: web page used to upload submissions;
20+
21+
We will execute the following services directly, from an IDE or from your command line, since these are the services we will be modifying:
22+
23+
1. Submission microservice: API used by the UI to initiate and finalize submissions;
24+
2. Submission processor: processes the submissions, which includes validating files, generating watermarked images, resizing images, generating consolidated zip files for download, updating Informix, and more.
25+
26+
# Pre-requisites
27+
28+
To prepare your environment to run the submission system, please start by installing these pre-requisites:
29+
30+
* Docker: https://docs.docker.com/engine/installation/
31+
* Docker-compose: https://docs.docker.com/compose/install/
32+
* Windows/Mac OS X:
33+
* Docker Toolbox: https://www.docker.com/products/docker-toolbox
34+
* Java 1.8;
35+
* Your favorite IDE (our team uses Eclipse and Intellij);
36+
* The submission-system.zip file, unzipped to a local folder. Henceforth in this document, we will call this folder submission-system.
37+
* (Optional but recommended) aws cli: http://docs.aws.amazon.com/cli/latest/userguide/installing.html
38+
* This will allow you to view data in the mongodb database, messages in sqs, etc.
39+
40+
# Local Configuration
41+
42+
## Hosts
43+
44+
Update your localhost entry in your hosts file (/etc/hosts) to map local.topcoder-dev.com:
45+
46+
127.0.0.1 localhost local.topcoder-dev.com
47+
48+
Windows:
49+
C:\Windows\System32\drivers\etc\hosts
50+
51+
192.168.99.100 localhost local.topcoder-dev.com
52+
53+
Mac OS X:
54+
/etc/hosts
55+
56+
192.168.99.100 localhost local.topcoder-dev.com
57+
58+
## AWS credentials
59+
60+
We will use fake AWS credentials to access the services, they will be loaded from your local AWS credentials folder, (~/.aws/). These credentials need to be consistent.
61+
62+
If you already have a ~/.aws folder, please make a backup of the contents so you can restore your system to the previous configuration once you are done working on the submissions system.
63+
64+
Create a ~/.aws folder, and in this folder create the following files:
65+
66+
~/.aws/config with the following content:
67+
68+
[default]
69+
output = json
70+
region = us-east-1
71+
72+
~/.aws/credentials with the following content
73+
74+
[default]
75+
aws_access_key_id = key
76+
aws_secret_access_key = secret
77+
78+
## Environment Configs
79+
80+
Inside the submission-system folder, you will find a env.sh file. This file contains local environment configurations, open it and edit the configurations in there according to your environment:
81+
82+
# The IP address of the computer running the submission system
83+
export IP=192.168.1.65
84+
85+
Windows/Mac OS X:
86+
87+
# The IP address of the computer running the submission system
88+
export IP=192.168.99.100
89+
90+
# Starting the Local Services
91+
92+
Export the env file configurations by issuing the following command inside the submission-system folder:
93+
94+
source env.sh
95+
96+
Inside the submission-system folder, issue the following command:
97+
98+
docker-compose up submissions
99+
100+
The first time you execute this command will cause all the required docker images to be downloaded to your local environment, this is a large download and it could take some time. Subsequent executions will be much faster.
101+
102+
Once you see the following message, all services are up:
103+
104+
submissions_1 | webpack: bundle is now VALID.
105+
106+
At this point you should be able to connect to:
107+
108+
* DynamoDB on localhost:7777;
109+
* Informix on localhost:2021;
110+
* User: informix
111+
* Password: 1nf0rm1x
112+
* Server: informixoltp_tcp
113+
* Jdbc url: jdbc:informix-sqli://localhost:2021/tcs_catalog:INFORMIXSERVER=informixoltp_tcp;
114+
* SQS on localhost:4568;
115+
* Kafka on localhost:9092
116+
* Zookeeper on localhost:2181;
117+
* Frontend on http://local.topcoder-dev.com:3000/
118+
119+
To stop the local services:
120+
121+
docker-compose down
122+
123+
Note that when shutting down the containers will be deleted, the next time you start the services you will start with a clean slate, i.e. all the data will be reset to the initial state, and any submissions that you did while testing will be lost.
124+
125+
# Building the Submission System
126+
127+
At this point you could use your favorite IDE to do the next tasks. We will take a command line based approach here, which should provide all details and parameters necessary to do the same through an IDE.
128+
129+
## Build ap-supply-library
130+
131+
Common classes for all member facing services
132+
133+
Unzip it the ap-supply-library folder, cd into it and:
134+
135+
mvn clean compile install
136+
137+
## Build ap-submission-library
138+
139+
Common classes for the submissions system, such as models, service clients, utilities, etc.
140+
141+
Unzip it the ap-submission-library folder, cd into it and:
142+
143+
mvn clean compile install
144+
145+
## Build ap-submission-microservice
146+
147+
Submission API
148+
149+
Unzip it the ap-submission-microservice/service folder, cd into it and:
150+
151+
mvn clean compile package
152+
153+
## Build ap-submission-processor
154+
155+
Processor of submissions.
156+
157+
Unzip it the ap-submission-processor/service folder, cd into it and:
158+
159+
mvn clean compile package
160+
161+
# Executing the Submission Service
162+
163+
cd into the ap-submission-service/service, you will find a run.sh file in there containing all environment variables and the startup command to run the service locally:
164+
165+
export AUTH_DOMAIN=topcoder-dev.com
166+
export DYNAMODB_ENDPOINT=http://localhost:7777
167+
export FILE_SERVICE_ENDPOINT=http://localhost/v3/
168+
export CHALLENGE_SERVICE_ENDPOINT=http://localhost/v3/
169+
export IDENTITY_SERVICE_ENDPOINT=http://localhost/v3/
170+
export SQS_URL=http://localhost:4568/submissions
171+
java -Ddw.server.applicationConnectors[0].port=8180 -Ddw.server.adminConnectors[0].port=8181 -jar target/submission-service-1.0.1-SNAPSHOT.jar server src/main/resources/config.yml
172+
173+
To execute the service:
174+
175+
sh run.sh
176+
177+
# Executing the Submission Processor
178+
179+
cd into the ap-submission-processor/service folder, in there you will find a run.sh file which contains all environment variables and the startup command to run the processor:
180+
181+
SQS_URL=http://localhost:4568/submissions
182+
API_URL=http://localhost/v3
183+
DYNAMODB_URL=http://localhost:7777
184+
ZOOKEEPER_HOSTS_LIST=localhost:2181
185+
java -Ddw.server.applicationConnectors[0].port=8280 -Ddw.server.adminConnectors[0].port=8281 -jar target/ap-submission-processor-1.0-SNAPSHOT.jar server src/main/resources/config.yml
186+
187+
# Accessing the Submission UI
188+
189+
Once the system is up and running, we can access the submission user interface to do a submission, however the system requires authentication, and we’ll have to fake it by setting some cookies and local storage values.
190+
191+
Open your browser and navigate to: http://local.topcoder-dev.com:3000
192+
193+
You should see a login screen. Open the browser’s javascript console, paste and execute the the following code:
194+
195+
document.cookie="tcsso=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJhZG1pbmlzdHJhdG9yIl0sImlzcyI6Imh0dHBzOi8vYXBpLnRvcGNvZGVyLWRldi5jb20iLCJoYW5kbGUiOiJoZWZmYW4iLCJleHAiOjE3NjYyODkyNDYsInVzZXJJZCI6IjEzMjQ1NiIsImlhdCI6MTQ1MDkyOTI0NiwiZW1haWwiOm51bGwsImp0aSI6IjEzNjljNjAwLWUwYTEtNDUyNS1hN2M3LTU2YmU3ZDgxM2Y1MSJ9.hp5peSoj-fh3KFkskvBpfUFIcJNtsv4zIMFV-D8F3JA";
196+
document.cookie="tcjwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJhZG1pbmlzdHJhdG9yIl0sImlzcyI6Imh0dHBzOi8vYXBpLnRvcGNvZGVyLWRldi5jb20iLCJoYW5kbGUiOiJoZWZmYW4iLCJleHAiOjE3NjYyODkyNDYsInVzZXJJZCI6IjEzMjQ1NiIsImlhdCI6MTQ1MDkyOTI0NiwiZW1haWwiOm51bGwsImp0aSI6IjEzNjljNjAwLWUwYTEtNDUyNS1hN2M3LTU2YmU3ZDgxM2Y1MSJ9.hp5peSoj-fh3KFkskvBpfUFIcJNtsv4zIMFV-D8F3JA";
197+
localStorage.setItem("appiriojwt", '"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJhZG1pbmlzdHJhdG9yIl0sImlzcyI6Imh0dHBzOi8vYXBpLnRvcGNvZGVyLWRldi5jb20iLCJoYW5kbGUiOiJoZWZmYW4iLCJleHAiOjE3NjYyODkyNDYsInVzZXJJZCI6IjEzMjQ1NiIsImlhdCI6MTQ1MDkyOTI0NiwiZW1haWwiOm51bGwsImp0aSI6IjEzNjljNjAwLWUwYTEtNDUyNS1hN2M3LTU2YmU3ZDgxM2Y1MSJ9.hp5peSoj-fh3KFkskvBpfUFIcJNtsv4zIMFV-D8F3JA"');
198+
199+
This browser now has the fake authentication, and we can go to the submission UI: http://local.topcoder-dev.com:3000/challenges/11/submit/file/
200+
Note the challenge id there is 11, that is the only challenge that is setup in Informix to allow submissions, if you would like to use other challenges you will have to create resource and phase records in Informix, however, using challenge 11 should be sufficient for our purposes.
201+
202+
You should now see a series of fields that can be populated by the submitter, the only required fields are the files at the top, and the “I UNDERSTAND AND AGREE” checkbox at the bottom:
203+
204+
![Files submission preview](submission_preview.png)
205+
206+
The submission zip is the representation of a design submission, it should be a simple zip file containing images. Any valid png, jpg, gif, or bpm files are allowed. You can download some images, take a few screenshots and create a submission zip file containing those images. The folder structure inside this submission zip doesn’t matter. This file needs to have valid images because those are processed, resized, watermarked, etc. Non image files in the submission.zip should be ignored.
207+
208+
The source zip file contains the source files for the submission, such as a photoshop project. The contents of this file doesn’t impact the process, you could simply have a zip file with just one file inside, or even use the submission zip itself, it doesn’t matter, as long as it is a valid zip.
209+
210+
The preview image is a cover image for the submission, any valid png or jpg file can be submitted as the preview.
211+
212+
Select your 3 files, scroll to the bottom, check the “AGREE” box, and click the submit button, you should see the upload happening, and the process logs flowing in the submission processor console.
213+
214+
If you see a message in the console saying “SUCCESSFUL NOTIFICATION SENT”, it means the processor finished successfully, otherwise you would see the exceptions in the log.
215+
216+
# Known Issues
217+
218+
* Local SQS sometimes stops working, messages don’t get delivered from the submission service to the submission processor. The best way to fix this issue is just to restart the local services with docker-compose down / docker-compose up submissions;
219+
220+
# Happy coding :)

‎submission-system/data/cover.png

157 KB
Loading

‎submission-system/data/default.conf

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
server {
2+
listen 80;
3+
server_name localhost;
4+
5+
#charset koi8-r;
6+
#access_log /var/log/nginx/log/host.access.log main;
7+
8+
location /* {
9+
root /usr/share/nginx/html;
10+
index index.html index.htm;
11+
}
12+
13+
#error_page 404 /404.html;
14+
15+
# redirect server error pages to the static page /50x.html
16+
#
17+
error_page 500 502 503 504 /50x.html;
18+
location = /50x.html {
19+
root /usr/share/nginx/html;
20+
}
21+
22+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
23+
#
24+
location /v3/members {
25+
proxy_pass http://192.168.99.100:8080;
26+
include /data/cors-include.conf;
27+
}
28+
29+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
30+
#
31+
location /v3/authorizations {
32+
proxy_pass http://192.168.99.100:8080;
33+
include /data/cors-include.conf;
34+
}
35+
36+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
37+
#
38+
location /v3/challenges {
39+
proxy_pass http://192.168.99.100:8080;
40+
include /data/cors-include.conf;
41+
}
42+
43+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
44+
#
45+
location /v3/submissions {
46+
proxy_pass http://192.168.99.1:8180;
47+
include /data/cors-include.conf;
48+
}
49+
50+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
51+
#
52+
location /v3/files {
53+
proxy_pass http://192.168.99.100:8080;
54+
include /data/cors-include.conf;
55+
}
56+
57+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
58+
#
59+
location /mock-upload {
60+
proxy_pass http://192.168.99.100:8080;
61+
include /data/cors-include.conf;
62+
}
63+
64+
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
65+
#
66+
location /mock-download {
67+
proxy_pass http://192.168.99.100:8080;
68+
include /data/cors-include.conf;
69+
}
70+
71+
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
72+
#
73+
#location ~ \.php$ {
74+
# root html;
75+
# fastcgi_pass 127.0.0.1:9000;
76+
# fastcgi_index index.php;
77+
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
78+
# include fastcgi_params;
79+
#}
80+
81+
# deny access to .htaccess files, if Apache's document root
82+
# concurs with nginx's one
83+
#
84+
#location ~ /\.ht {
85+
# deny all;
86+
#}
87+
}
88+

‎submission-system/data/source.zip

922 KB
Binary file not shown.

‎submission-system/data/submission.zip

147 KB
Binary file not shown.

‎submission-system/data/tc-constants.coffee

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
configEnvConstants = (ENV) ->
2+
constants = {}
3+
4+
if ENV == 'DEV'
5+
Object.assign constants,
6+
API_URL : 'http://local.topcoder-dev.com/v3'
7+
API_URL_V2 : 'http://local.topcoder-dev.com/v2'
8+
ASSET_PREFIX : 'https://s3.amazonaws.com/app.topcoder-dev.com/'
9+
AUTH_API_URL : 'http://local.topcoder-dev.com/v3'
10+
auth0Callback : 'http://local.topcoder-dev.com/pub/callback.html'
11+
auth0Domain : 'topcoder-dev.auth0.com'
12+
clientId : 'JFDo7HMkf0q2CkVFHojy3zHWafziprhT'
13+
AUTH0_DOMAIN : 'topcoder-dev.auth0.com'
14+
AUTH0_CLIENT_ID : 'JFDo7HMkf0q2CkVFHojy3zHWafziprhT'
15+
domain : 'local.topcoder-dev.com'
16+
ENV : 'DEV'
17+
18+
NEW_RELIC_APPLICATION_ID: if process.env.TRAVIS_BRANCH then '8957921' else ''
19+
20+
ARENA_URL : '//arena.topcoder-dev.com'
21+
BLOG_LOCATION : 'https://www.topcoder-dev.com/feed/?post_type=blog'
22+
COMMUNITY_URL : '//community.topcoder-dev.com'
23+
FORUMS_APP_URL : '//apps.topcoder-dev.com/forums'
24+
HELP_APP_URL : 'help.topcoder-dev.com'
25+
MAIN_URL : 'https://www.topcoder-dev.com'
26+
PHOTO_LINK_LOCATION: 'https://community.topcoder-dev.com'
27+
SWIFT_PROGRAM_URL : 'apple.topcoder-dev.com',
28+
TCO16_URL : 'http://tco16.topcoder-dev.com'
29+
30+
31+
if ENV == 'QA'
32+
Object.assign constants,
33+
API_URL : 'https://api.topcoder-qa.com/v3'
34+
API_URL_V2 : 'https://api.topcoder-qa.com/v2'
35+
ASSET_PREFIX : 'https://s3.amazonaws.com/app.topcoder-qa.com/'
36+
AUTH_API_URL : 'https://api.topcoder-qa.com/v3'
37+
auth0Callback : 'https://api.topcoder-qa.com/pub/callback.html'
38+
auth0Domain : 'topcoder-qa.auth0.com'
39+
clientId : 'EVOgWZlCtIFlbehkq02treuRRoJk12UR'
40+
AUTH0_DOMAIN : 'topcoder-qa.auth0.com'
41+
AUTH0_CLIENT_ID : 'EVOgWZlCtIFlbehkq02treuRRoJk12UR'
42+
domain : 'topcoder-qa.com'
43+
ENV : 'QA'
44+
45+
NEW_RELIC_APPLICATION_ID: if process.env.TRAVIS_BRANCH then '11199233' else ''
46+
47+
ARENA_URL : '//arena.topcoder-qa.com'
48+
BLOG_LOCATION : 'https://www.topcoder-qa.com/feed/?post_type=blog'
49+
COMMUNITY_URL : '//community.topcoder-qa.com'
50+
FORUMS_APP_URL : '//apps.topcoder-qa.com/forums'
51+
HELP_APP_URL : 'help.topcoder-qa.com'
52+
MAIN_URL : 'https://www.topcoder-qa.com'
53+
PHOTO_LINK_LOCATION: 'https://community.topcoder-qa.com'
54+
SWIFT_PROGRAM_URL : 'apple.topcoder-qa.com',
55+
TCO16_URL : 'http://tco16.topcoder-qa.com'
56+
57+
if ENV == 'PROD'
58+
Object.assign constants,
59+
API_URL : 'https://api.topcoder.com/v3'
60+
API_URL_V2 : 'https://api.topcoder.com/v2'
61+
ASSET_PREFIX : 'https://s3.amazonaws.com/app.topcoder.com/'
62+
AUTH_API_URL : 'https://api.topcoder.com/v3'
63+
auth0Callback : 'https://api.topcoder.com/pub/callback.html'
64+
auth0Domain : 'topcoder.auth0.com'
65+
clientId : '6ZwZEUo2ZK4c50aLPpgupeg5v2Ffxp9P'
66+
AUTH0_DOMAIN : 'topcoder.auth0.com'
67+
AUTH0_CLIENT_ID : '6ZwZEUo2ZK4c50aLPpgupeg5v2Ffxp9P'
68+
domain : 'topcoder.com'
69+
ENV : 'PROD'
70+
NODE_ENV : 'production'
71+
72+
NEW_RELIC_APPLICATION_ID: if process.env.TRAVIS_BRANCH then '11352758' else ''
73+
74+
ARENA_URL : '//arena.topcoder.com'
75+
BLOG_LOCATION : 'https://www.topcoder.com/feed/?post_type=blog'
76+
COMMUNITY_URL : '//community.topcoder.com'
77+
FORUMS_APP_URL : '//apps.topcoder.com/forums'
78+
HELP_APP_URL : 'help.topcoder.com'
79+
MAIN_URL : 'https://www.topcoder.com'
80+
PHOTO_LINK_LOCATION: 'https://community.topcoder.com'
81+
SWIFT_PROGRAM_URL : 'apple.topcoder.com',
82+
TCO16_URL : 'http://tco16.topcoder.com'
83+
84+
constants
85+
86+
module.exports = configEnvConstants

0 commit comments

Comments
 (0)