Skip to content

Commit f9a65a9

Browse files
maxceemmaxceem
authored
Merge pull request #128 from topcoder-platform/feature/data-scripts
Feature/data scripts
2 parents f785c99 + 2fd35a1 commit f9a65a9

21 files changed

+1618
-867
lines changed

‎README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,10 +141,15 @@
141141
| `npm run start` | Start app in the production mode from prebuilt `dist` folder. |
142142
| `npm run dev` | Start app in the development mode using `nodemon`. |
143143
| `npm run test` | Run tests. |
144-
| `npm run test-data` | Clears and imports Data into ES. |
145144
| `npm run init-db` | Initializes Database. |
146-
| `npm run create-index` | Create Elasticsearch indexes. |
147-
| `npm run delete-index` | Delete Elasticsearch indexes. |
145+
| `npm run create-index` | Create Elasticsearch indexes. Use `-- --force` flag to skip confirmation |
146+
| `npm run delete-index` | Delete Elasticsearch indexes. Use `-- --force` flag to skip confirmation |
147+
| `npm run data:import <filePath>` | Imports data into ES and db from filePath (`./data/demo-data.json` is used as default). Use `-- --force` flag to skip confirmation |
148+
| `npm run data:export <filePath>` | Exports data from ES and db into filePath (`./data/demo-data.json` is used as default). Use `-- --force` flag to skip confirmation |
149+
| `npm run index:all` | Indexes all data from db into ES. Use `-- --force` flag to skip confirmation|
150+
| `npm run index:jobs <jobId>` | Indexes job data from db into ES, if jobId is not given all data is indexed. Use `-- --force` flag to skip confirmation |
151+
| `npm run index:job-candidates <jobCandidateId>` | Indexes job candidate data from db into ES, if jobCandidateId is not given all data is indexed. Use `-- --force` flag to skip confirmation |
152+
| `npm run index:resource-bookings <resourceBookingsId>` | Indexes resource bookings data from db into ES, if resourceBookingsId is not given all data is indexed. Use `-- --force` flag to skip confirmation |
148153
| `npm run services:up` | Start services via docker-compose for local development. |
149154
| `npm run services:down` | Stop services via docker-compose for local development. |
150155
| `npm run services:logs -- -f <service_name>` | View logs of some service inside docker-compose. |

‎config/default.js

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ module.exports = {
4747
// the project service url
4848
PROJECT_API_URL: process.env.PROJECT_API_URL || 'https://api.topcoder-dev.com',
4949

50+
// the default path for importing and exporting data
51+
DEFAULT_DATA_FILE_PATH: './data/demo-data.json',
52+
5053
esConfig: {
5154
// the elasticsearch host
5255
HOST: process.env.ES_HOST || 'http://localhost:9200',
@@ -68,7 +71,12 @@ module.exports = {
6871
// the job candidate index
6972
ES_INDEX_JOB_CANDIDATE: process.env.ES_INDEX_JOB_CANDIDATE || 'job_candidate',
7073
// the resource booking index
71-
ES_INDEX_RESOURCE_BOOKING: process.env.ES_INDEX_RESOURCE_BOOKING || 'resource_booking'
74+
ES_INDEX_RESOURCE_BOOKING: process.env.ES_INDEX_RESOURCE_BOOKING || 'resource_booking',
75+
76+
// the max bulk size in MB for ES indexing
77+
MAX_BULK_REQUEST_SIZE_MB: process.env.MAX_BULK_REQUEST_SIZE_MB || 20,
78+
// the max number of documents per bulk for ES indexing
79+
MAX_BULK_NUM_DOCUMENTS: process.env.MAX_BULK_NUM_DOCUMENTS || 100
7280
},
7381

7482
// Topcoder Bus API URL

0 commit comments

Comments
 (0)