Skip to content

Commit 87be91e

Browse files
ThomasKranitsasThomasKranitsas
committed
Sync with develop
2 parents 27355a6 + 9974b23 commit 87be91e

26 files changed

+4211
-1843
lines changed

‎.dockerignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
node_modules/
2+
mock/node_modules
23
.env
34
coverage/
45
.nyc_output/

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
.idea
22
node_modules
3+
mock/node_modules
34
*.log
45
.DS_Store
56
.env

‎README.md

Lines changed: 130 additions & 306 deletions
Large diffs are not rendered by default.

‎Verification.md

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
## Verification
2+
3+
- start kafka server, start elasticsearch, initialize Elasticsearch, start processor app
4+
- Before testing update message, we need to create a record in ES. If you are using the ES from docker-es and using default configuration variables, use the below command to create a record in ES through curl.
5+
6+
```bash
7+
curl -H "Content-Type: application/json" -X POST "http://localhost:9200/challenge/_doc/173803d3-019e-4033-b1cf-d7205c7f774c" -d "{\"id\":\"173803d3-019e-4033-b1cf-d7205c7f774c\",\"typeId\":\"8e17090c-465b-4c17-b6d9-dfa16300b0ff\",\"track\":\"Code\",\"name\":\"test\",\"description\":\"desc\",\"timelineTemplateId\":\"8e17090c-465b-4c17-b6d9-dfa16300b0aa\",\"phases\":[{\"id\":\"8e17090c-465b-4c17-b6d9-dfa16300b012\",\"phaseId\":\"8e17090c-465b-4c17-b6d9-dfa16300b2ba\",\"isOpen\":true,\"duration\":10000}],\"prizeSets\":[{\"type\":\"prize\",\"prizes\":[{\"type\":\"winning prize\",\"value\":500}]}],\"reviewType\":\"code review\",\"tags\":[\"code\"],\"projectId\":123,\"forumId\":456,\"status\":\"Active\",\"created\":\"2018-01-02T00:00:00\",\"createdBy\":\"admin\"}"
8+
```
9+
10+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the created data, you will see the data are properly created:
11+
12+
```bash
13+
info: Elasticsearch data:
14+
info: {
15+
"id": "173803d3-019e-4033-b1cf-d7205c7f774c",
16+
"typeId": "8e17090c-465b-4c17-b6d9-dfa16300b0ff",
17+
"track": "Code",
18+
"name": "test",
19+
"description": "desc",
20+
"timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0aa",
21+
"phases": [
22+
{
23+
"id": "8e17090c-465b-4c17-b6d9-dfa16300b012",
24+
"phaseId": "8e17090c-465b-4c17-b6d9-dfa16300b2ba",
25+
"isOpen": true,
26+
"duration": 10000
27+
}
28+
],
29+
"prizeSets": [
30+
{
31+
"type": "prize",
32+
"prizes": [
33+
{
34+
"type": "winning prize",
35+
"value": 500
36+
}
37+
]
38+
}
39+
],
40+
"reviewType": "code review",
41+
"tags": [
42+
"code"
43+
],
44+
"projectId": 123,
45+
"forumId": 456,
46+
"status": "Active",
47+
"created": "2018-01-02T00:00:00",
48+
"createdBy": "admin"
49+
}
50+
info: Done!
51+
```
52+
- start kafka-console-producer to write messages to `challenge.notification.update` topic:
53+
`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic challenge.notification.update`
54+
55+
- write message to partially update data:
56+
`{"topic":"challenge.notification.update","originator":"challenge-api","timestamp":"2019-02-17T01:00:00","mime-type":"application/json","payload":{"id":"173803d3-019e-4033-b1cf-d7205c7f774c","typeId":"8e17090c-465b-4c17-b6d9-dfa16300b0ff","track":"Code","name":"test3","description":"desc3","timelineTemplateId":"8e17090c-465b-4c17-b6d9-dfa16300b0dd","groups":["group2","group3"],"updated":"2019-02-17T01:00:00","updatedBy":"admin","terms":[{"id":21343,"agreeabilityType":"DocuSignable","title":"Competition Non-Disclosure Agreement","url":"","templateId":"0c5b7081-1fff-4484-a20f-824c97a03b9b"},{"id":20723,"agreeabilityType":"Non-electronically-agreeable","title":"Subcontractor Services Agreement 2009-09-02","url":"http://www.topcoder.com/i/terms/Subcontractor+Services+Agreement+2009-09-02.pdf"}]}}`
57+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
58+
59+
```bash
60+
info: Elasticsearch data:
61+
info: {
62+
"id": "173803d3-019e-4033-b1cf-d7205c7f774c",
63+
"typeId": "8e17090c-465b-4c17-b6d9-dfa16300b0ff",
64+
"track": "Code",
65+
"name": "test3",
66+
"description": "desc3",
67+
"timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0dd",
68+
"phases": [
69+
{
70+
"id": "8e17090c-465b-4c17-b6d9-dfa16300b012",
71+
"phaseId": "8e17090c-465b-4c17-b6d9-dfa16300b2ba",
72+
"isOpen": true,
73+
"duration": 10000
74+
}
75+
],
76+
"prizeSets": [
77+
{
78+
"type": "prize",
79+
"prizes": [
80+
{
81+
"type": "winning prize",
82+
"value": 500
83+
}
84+
]
85+
}
86+
],
87+
"reviewType": "code review",
88+
"tags": [
89+
"code"
90+
],
91+
"projectId": 123,
92+
"forumId": 456,
93+
"status": "Active",
94+
"created": "2018-01-02T00:00:00",
95+
"createdBy": "admin",
96+
"updatedBy": "admin",
97+
"groups": [
98+
"group2",
99+
"group3"
100+
],
101+
"terms": [
102+
{
103+
"id": 21343,
104+
"agreeabilityType": "DocuSignable",
105+
"title": "Competition Non-Disclosure Agreement",
106+
"url": "",
107+
"templateId": "0c5b7081-1fff-4484-a20f-824c97a03b9b"
108+
},
109+
{
110+
"id": 20723,
111+
"agreeabilityType": "Non-electronically-agreeable",
112+
"title": "Subcontractor Services Agreement 2009-09-02",
113+
"url": "http://www.topcoder.com/i/terms/Subcontractor+Services+Agreement+2009-09-02.pdf"
114+
}],
115+
"updated": "2019-02-16T17:00:00.000Z"
116+
}
117+
info: Done!
118+
```
119+
- write message to update data:
120+
`{"topic":"challenge.notification.update","originator":"challenge-api","timestamp":"2019-02-17T00:00:00","mime-type":"application/json","payload":{"id":"173803d3-019e-4033-b1cf-d7205c7f774c","typeId":"45415132-79fa-4d13-a9ac-71f50020dc10","track":"Code","name":"test","description":"a b c","challengeSettings":[{"type":"2d88c598-70f0-4054-8a45-7da38d0ca424","value":"ab"}],"timelineTemplateId":"8e17090c-465b-4c17-b6d9-dfa16300b0aa","phases":[{"id":"8e17090c-465b-4c17-b6d9-dfa16300b012","phaseId":"8e17090c-465b-4c17-b6d9-dfa16300b013","isOpen":true,"duration":2000}],"prizeSets":[{"type":"prize","prizes":[{"type":"win","value":90}]}],"reviewType":"code","tags":["tag1","tag2"],"projectId":12,"forumId":45,"legacyId":55,"status":"Active","groups":["g2"],"terms":[{"id":21343,"agreeabilityType":"DocuSignable","title":"Competition Non-Disclosure Agreement","url":"","templateId":"0c5b7081-1fff-4484-a20f-824c97a03b9b"}],"startDate":"2019-07-17T00:00:00","updated":"2019-02-17T00:00:00","updatedBy":"user"}}`
121+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
122+
123+
```bash
124+
info: Elasticsearch data:
125+
info: {
126+
"id": "173803d3-019e-4033-b1cf-d7205c7f774c",
127+
"typeId": "45415132-79fa-4d13-a9ac-71f50020dc10",
128+
"track": "Code",
129+
"name": "test",
130+
"description": "a b c",
131+
"timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0aa",
132+
"phases": [
133+
{
134+
"duration": 2000,
135+
"isOpen": true,
136+
"phaseId": "8e17090c-465b-4c17-b6d9-dfa16300b013",
137+
"id": "8e17090c-465b-4c17-b6d9-dfa16300b012"
138+
}
139+
],
140+
"prizeSets": [
141+
{
142+
"prizes": [
143+
{
144+
"type": "win",
145+
"value": 90
146+
}
147+
],
148+
"type": "prize"
149+
}
150+
],
151+
"reviewType": "code",
152+
"tags": [
153+
"tag1",
154+
"tag2"
155+
],
156+
"projectId": 12,
157+
"forumId": 45,
158+
"status": "Active",
159+
"created": "2018-01-02T00:00:00",
160+
"createdBy": "admin",
161+
"updatedBy": "user",
162+
"groups": [
163+
"g2"
164+
],
165+
"updated": "2019-02-16T16:00:00.000Z",
166+
"endDate": "2019-07-16T16:33:20.000Z",
167+
"challengeSettings": [
168+
{
169+
"type": "2d88c598-70f0-4054-8a45-7da38d0ca424",
170+
"value": "ab"
171+
}
172+
],
173+
"terms": [
174+
{
175+
"id": 21343,
176+
"agreeabilityType": "DocuSignable",
177+
"title": "Competition Non-Disclosure Agreement",
178+
"url": "",
179+
"templateId": "0c5b7081-1fff-4484-a20f-824c97a03b9b"
180+
}],
181+
"currentPhase": {
182+
"duration": 2000,
183+
"isOpen": true,
184+
"phaseId": "8e17090c-465b-4c17-b6d9-dfa16300b013",
185+
"id": "8e17090c-465b-4c17-b6d9-dfa16300b012"
186+
},
187+
"legacyId": 55,
188+
"startDate": "2019-07-16T16:00:00.000Z"
189+
}
190+
info: Done!
191+
```
192+
193+
- you may write invalid message like:
194+
`{ "topic": "challenge.notification.update", "originator": "challenge-api", "timestamp": "2019-02-17T01:00:00", "mime-type": "application/json", "payload": { "id": "173803d3-019e-4033-b1cf-d7205c7f774c", "typeId": "123", "track": "Code", "name": "test3", "description": "desc3", "timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0dd", "groups": ["group2", "group3"], "updated": "2019-02-17T01:00:00", "updatedBy": "admin" } }`
195+
196+
`{ "topic": "challenge.notification.update", "originator": "challenge-api", "timestamp": "2019-02-17T01:00:00", "mime-type": "application/json", "payload": { "id": "173803d3-019e-4033-b1cf-d7205c7f774c", "typeId": "8e17090c-465b-4c17-b6d9-dfa16300b0ff", "track": ["Code"], "name": "test3", "description": "desc3", "timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0dd", "groups": ["group2", "group3"], "updated": "2019-02-17T01:00:00", "updatedBy": "admin" } }`
197+
198+
`[ [ [ } } }`
199+
- then in the app console, you will see error messages
200+
201+
- start kafka-console-producer to write messages to `challenge.action.resource.create` topic:
202+
`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic challenge.action.resource.create`
203+
204+
- write message to create resource:
205+
`{ "topic": "challenge.action.resource.create", "originator": "resource-api", "timestamp": "2019-02-17T00:00:00", "mime-type": "application/json", "payload": { "id": "45415132-79fa-4d13-a9ac-71f50020dc10", "challengeId": "173803d3-019e-4033-b1cf-d7205c7f774c", "memberId": "123456", "memberHandle": "test", "roleId": "45415132-79fa-4d13-a9ac-71f50020dc12" } }`
206+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
207+
208+
```bash
209+
...
210+
"numOfRegistrants": 1
211+
...
212+
```
213+
214+
- start kafka-console-producer to write messages to `challenge.action.resource.delete` topic:
215+
`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic challenge.action.resource.delete`
216+
217+
- write message to delete resource:
218+
`{ "topic": "challenge.action.resource.delete", "originator": "resource-api", "timestamp": "2019-02-17T00:00:00", "mime-type": "application/json", "payload": { "id": "45415132-79fa-4d13-a9ac-71f50020dc10", "challengeId": "173803d3-019e-4033-b1cf-d7205c7f774c", "memberId": "123456", "memberHandle": "test", "roleId": "45415132-79fa-4d13-a9ac-71f50020dc12" } }`
219+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
220+
221+
```bash
222+
...
223+
"numOfRegistrants": 0
224+
...
225+
```
226+
227+
- start kafka-console-producer to write messages to `submission.notification.create` topic:
228+
`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.create`
229+
230+
- write message to create submission:
231+
`{ "topic": "submission.notification.create", "originator": "submission-api", "timestamp": "2019-02-17T00:00:00", "mime-type": "application/json", "payload": { "resource": "submission", "id": "45415132-79fa-4d13-a9ac-71f50020dc18", "type": "ContestSubmission", "url": "http://test.com/123", "challengeId": "173803d3-019e-4033-b1cf-d7205c7f774c", "memberId": "123456", "created": "2019-02-03T01:01:00", "createdBy": "test" } }`
232+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
233+
234+
```bash
235+
...
236+
"numOfSubmissions": 1
237+
...
238+
```
239+
240+
- start kafka-console-producer to write messages to `submission.notification.delete` topic:
241+
`bin/kafka-console-producer.sh --broker-list localhost:9092 --topic submission.notification.delete`
242+
243+
- write message to delete submission:
244+
`{ "topic": "submission.notification.delete", "originator": "submission-api", "timestamp": "2019-02-17T00:00:00", "mime-type": "application/json", "payload": { "resource": "submission", "id": "45415132-79fa-4d13-a9ac-71f50020dc18", "type": "ContestSubmission", "url": "http://test.com/123", "challengeId": "173803d3-019e-4033-b1cf-d7205c7f774c", "memberId": "123456", "created": "2019-02-03T01:01:00", "createdBy": "test" } }`
245+
- run command `npm run view-data 173803d3-019e-4033-b1cf-d7205c7f774c` to view the updated data, you will see the data are properly updated:
246+
247+
```bash
248+
...
249+
"numOfSubmissions": 0
250+
...
251+
```
252+
253+
- to test the health check API, run `export PORT=5000`, start the processor, then browse `http://localhost:5000/health` in a browser,
254+
and you will see result `{"checksRun":1}`

‎config/default.js

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,14 @@ module.exports = {
66
DISABLE_LOGGING: process.env.DISABLE_LOGGING ? process.env.DISABLE_LOGGING === 'true' : false, // If true, logging will be disabled
77
LOG_LEVEL: process.env.LOG_LEVEL || 'debug',
88

9+
// used to get M2M token
10+
AUTH0_URL: process.env.AUTH0_URL,
11+
AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE || 'https://www.topcoder-dev.com',
12+
TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME,
13+
AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
14+
AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
15+
AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
16+
917
KAFKA_URL: process.env.KAFKA_URL || 'localhost:9092',
1018
KAFKA_GROUP_ID: process.env.KAFKA_GROUP_ID || 'challenge-processor-es',
1119
// below are used for secure Kafka connection, they are optional
@@ -15,17 +23,26 @@ module.exports = {
1523

1624
UPDATE_DATA_TOPIC: process.env.UPDATE_DATA_TOPIC || 'challenge.notification.update',
1725
CREATE_RESOURCE_TOPIC: process.env.CREATE_RESOURCE_TOPIC || 'challenge.action.resource.create',
26+
UPDATE_RESOURCE_TOPIC: process.env.UPDATE_RESOURCE_TOPIC || 'challenge.action.resource.update',
1827
DELETE_RESOURCE_TOPIC: process.env.DELETE_RESOURCE_TOPIC || 'challenge.action.resource.delete',
1928
CREATE_SUBMISSION_TOPIC: process.env.CREATE_SUBMISSION_TOPIC || 'submission.notification.create',
29+
UPDATE_SUBMISSION_TOPIC: process.env.UPDATE_SUBMISSION_TOPIC || 'submission.notification.update',
2030
DELETE_SUBMISSION_TOPIC: process.env.DELETE_SUBMISSION_TOPIC || 'submission.notification.delete',
2131

22-
// challenge registrant role id, if not provided then any role is considered as registrant
23-
REGISTRANT_ROLE_ID: process.env.REGISTRANT_ROLE_ID,
32+
// challenge registrant resource role id, if not provided then any resource role is considered as registrant
33+
REGISTRANT_RESOURCE_ROLE_ID: process.env.REGISTRANT_RESOURCE_ROLE_ID || '173803d3-019e-4033-b1cf-d7205c7f773a',
34+
SUBMISSIONS_API_URL: process.env.SUBMISSIONS_API_URL || 'http://localhost:4000/v5/submissions',
35+
RESOURCES_API_URL: process.env.RESOURCES_API_URL || 'http://localhost:4000/v5/resources',
36+
CONTEST_SUBMISSION_TYPE: process.env.CONTEST_SUBMISSION_TYPE || 'Contest Submission',
37+
CHECKPOINT_SUBMISSION_TYPE: process.env.CHECKPOINT_SUBMISSION_TYPE || 'Checkpoint Submission',
38+
39+
// superagent request timeout in milliseconds
40+
REQUEST_TIMEOUT: process.env.REQUEST_TIMEOUT ? Number(process.env.REQUEST_TIMEOUT) : 20000,
2441

2542
esConfig: {
2643
HOST: process.env.ES_HOST || 'localhost:9200',
2744
AWS_REGION: process.env.AWS_REGION || 'us-east-1', // AWS Region to be used if we use AWS ES
28-
API_VERSION: process.env.ES_API_VERSION || '6.3',
45+
API_VERSION: process.env.ES_API_VERSION || '6.8',
2946
ES_INDEX: process.env.ES_INDEX || 'challenge',
3047
ES_TYPE: process.env.ES_TYPE || '_doc' // ES 6.x accepts only 1 Type per index and it's mandatory to define it
3148
}

‎config/test.js

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,6 @@
33
*/
44

55
module.exports = {
6-
// challenge registrant role id, if not provided then any role is considered as registrant
7-
REGISTRANT_ROLE_ID: process.env.REGISTRANT_ROLE_ID || '173803d3-019e-4033-b1cf-d7205c7f773a',
8-
96
esConfig: {
107
HOST: process.env.ES_HOST || 'localhost:9200',
118
ES_INDEX: process.env.ES_INDEX_TEST || 'challenge-test',

‎docker-es/docker-compose.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
version: "2"
1+
version: '3'
22
services:
33
esearch:
4-
image: "docker.elastic.co/elasticsearch/elasticsearch:6.3.1"
4+
image: "elasticsearch:6.8.4"
55
ports:
66
- "9200:9200"

‎docker/Dockerfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
# Use the base image with Node.js 8.11.3
2-
FROM node:8.11.3
1+
# Use the base image with Node.js 10.19
2+
FROM node:10.19
33

44
# Copy the current directory into the Docker image
55
COPY . /challenge-processor-es

‎docker/docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,4 @@ services:
77
dockerfile: docker/Dockerfile
88
env_file:
99
- api.env
10-
10+
network_mode: "host"

‎docs/dev.env

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
DISABLE_LOGGING:false,
3+
LOG_LEVEL:'',
4+
5+
AUTH0_URL:'',
6+
AUTH0_AUDIENCE:'',
7+
TOKEN_CACHE_TIME:'',
8+
AUTH0_PROXY_SERVER_URL:'',
9+
AUTH0_CLIENT_ID:'',
10+
AUTH0_CLIENT_SECRET:'',
11+
12+
KAFKA_URL:'',
13+
KAFKA_GROUP_ID:'',
14+
KAFKA_CLIENT_CERT:'',
15+
KAFKA_CLIENT_CERT_KEY:'',
16+
17+
UPDATE_DATA_TOPIC:'',
18+
CREATE_RESOURCE_TOPIC:'',
19+
UPDATE_RESOURCE_TOPIC:'',
20+
DELETE_RESOURCE_TOPIC:'',
21+
CREATE_SUBMISSION_TOPIC:'',
22+
UPDATE_SUBMISSION_TOPIC:'',
23+
DELETE_SUBMISSION_TOPIC:'',
24+
25+
REGISTRANT_RESOURCE_ROLE_ID:'',
26+
27+
SUBMISSIONS_API_URL:'',
28+
RESOURCES_API_URL:'',
29+
CONTEST_SUBMISSION_TYPE:'',
30+
CHECKPOINT_SUBMISSION_TYPE:'',
31+
REQUEST_TIMEOUT:'',
32+
33+
esConfig : {
34+
HOST:'',
35+
AWS_REGION:'',
36+
API_VERSION:'',
37+
ES_INDEX:'',
38+
ES_TYPE:''
39+
}
40+
}

0 commit comments

Comments
 (0)