-
IRUBIS HDF5 files visualized in react programmatically
- -[](https://badge.fury.io/js/react) -[](https://www.python.org/downloads/release/python-360/) - -This HDF5 visualization project written with Flask web framework(Python) as backend and React (Javascript) as frontend is served in Docker containers. HDF5 format is a hierarchical data structure, consists mainly of the following components: attributes, groups and data sets. Attributes are used for storing meta-data. Groups and data sets could be considered as folders and files, respectively, in classical file system - https://en.wikipedia.org/wiki/Hierarchical_Data_Format. - -Structure of IRUBIS HDF5 files Attributes: - -- wavenumbers, array: Wx1, with W - number of data samples - From here you can extract wavenumbers, which are commonly used as x-axis (horizontal) for plotting spectra. - -Groups: - -- internal -- external - -The internal group has the following structure: - -- glucose -- measurement -- time - -time - integers, size Nx1, with N - number of data samples, consists of UNIX timestamps, at which a spectrum was measured. - -measurement - floats, size NxW, with N - number of data samples, W - number of wavenumber, each row is a spectrum. - -glucose - floats, size Nx1, with N - number of data samples, consists of glucose values that were calculated based on corresponding measurement spectra. - -The goal of this task is to programmaticaly extract data from files and plot two graphs. One graph should contain glucose over time and the second, the corresponding measurement. - -# [📖 Docs] - -- [[📖 Docs]](#-docs) - - [Quick Start](#quick-start) -- [Technology](#technology) - - [Backend](#backend) - - [Frontend](#frontend) -- [Dependencies](#dependencies) - - [Backend](#backend-1) - - [Frontend](#frontend-1) -- [EndPoints](#endpoints) -- [Screenshots](#screenshots) - -## Quick Start - -Get up and running with the following. - -- Clone from Github - -```bash -git clone https://github.com/Zak-Musah/hdf5-data-visualization-react.git - -cd hdf5-data-visualization-react - -``` - -- Docker/Docker Compose should be installed before running the start command -- 3 docker containers were used - Backend (Python) , Frontend (Node) and Nginx - -- To start both backend and frontend and run tests using local docker client, run the following from root of application: - -```bash -sh ./start.sh -``` - -- In case the start.sh does not seem to be runnable, use - -```bash -chmod 400 start.sh -``` - -- To run project without docker , follow these steps; Also change correct path of dataset folder in backend/api/**init**.py - -```bash -cd backend -python3 -m venv env [Create a python environment] -source env/bin/activate -(env) pip install -r requirements.txt -(env) export FLASK_APP=api/__init__.py -(env) export REACT_APP_BACKEND_SERVICE_URL=http://localhost:5000 -(env) python manage.py run [Navigate to http://localhost:5000/api/test for sanity check] - - -cd frontend -npm install -export REACT_APP_BACKEND_SERVICE_URL=http://localhost:5000 -npm start [Automatically launches app in your default browser on http://localhost:3000] -``` - -# Technology - -## Backend - -- Python v3.8.6 -- Flask v1.1.2 -- Docker v19.03.1 -- Docker Compose v1.21.0 -- Docker Compose file v3.3 -- Nginx v1.15.9 -- Guicorn v19.9.0 - -## Frontend - -- Node v12.18.3 -- Npm v6.14.7 -- React v16.14.0 -- React Scripts v2.1.8 -- React Dom v16.14.0 - -# Dependencies - -## Backend - -- Flask Testing v0.8.0 - For unit testing within flask -- h5py v3.1.0 - For extracting of big data from HDF5 binary data format -- numpy v1.19.4 To structure multi-dimensional data extracted by h5py into numpy matrices for data aggregation -- Coverage v5.3 - For checking coverage of code tested -- Flask CORS v3.0.9 - To allow for cross-origin AJAX requests -- tzlocal v2.1 - To convert unix timestamps to local timezone - -## Frontend - -- Axios v0.18.1 - For making AJAX calls to the server(backend) -- react-bootstrap v1.4.0 - For app CSS styling and loading spinner -- React-PlotlyJS/PlotlyJS v2.4.0/1.45.3 - For charts shown on dashboard -- Enzyme v3.9.0 - Utility library used for testing react components - -# EndPoints - -You can test out the following endpoints: - -``` - - Endpoint HTTP Method CRUD Method Result - / GET READ Load React App - /api/test GET READ Sanity Check - /api/get_file_names GET READ Get file names - /api/read_data POST READ Get data from file -``` - -# Screenshots - -## When App First Renders - - - -## Dataset-1 - - - -## Dataset-2 - - ++
IRUBIS HDF5 files visualized in react programmatically
+ +[](https://badge.fury.io/js/react) +[](https://www.python.org/downloads/release/python-360/) + +This HDF5 visualization project written with Flask web framework(Python) as backend and React (Javascript) as frontend is served in Docker containers. HDF5 format is a hierarchical data structure, consists mainly of the following components: attributes, groups and data sets. Attributes are used for storing meta-data. Groups and data sets could be considered as folders and files, respectively, in classical file system - https://en.wikipedia.org/wiki/Hierarchical_Data_Format. + +Structure of IRUBIS HDF5 files Attributes: + +- wavenumbers, array: Wx1, with W - number of data samples + From here you can extract wavenumbers, which are commonly used as x-axis (horizontal) for plotting spectra. + +Groups: + +- internal +- external + +The internal group has the following structure: + +- glucose +- measurement +- time + +time - integers, size Nx1, with N - number of data samples, consists of UNIX timestamps, at which a spectrum was measured. + +measurement - floats, size NxW, with N - number of data samples, W - number of wavenumber, each row is a spectrum. + +glucose - floats, size Nx1, with N - number of data samples, consists of glucose values that were calculated based on corresponding measurement spectra. + +The goal of this task is to programmaticaly extract data from files and plot two graphs. One graph should contain glucose data points and the second, the corresponding spectrum. + +## Demo + +Check out the live app, running on Netlify: +[Demo](https://irubis-dashboard.netlify.app/) + +# [📖 Docs] + +- [[📖 Docs]](#-docs) + - [Quick Start](#quick-start) +- [Technology](#technology) + - [Backend](#backend) + - [Frontend](#frontend) +- [Dependencies](#dependencies) + - [Backend](#backend-1) + - [Frontend](#frontend-1) +- [EndPoints](#endpoints) +- [Screenshots](#screenshots) + +## Quick Start + +Get up and running with the following. + +- Clone from Github + +```bash +git clone https://github.com/Zak-Musah/hdf5-data-visualization-react.git + +cd hdf5-data-visualization-react + +``` + +- Docker/Docker Compose should be installed before running the start command +- 3 docker containers were used - Backend (Python) , Frontend (Node) and Nginx + +- To start both backend and frontend and run tests using local docker client, run the following from root of application: + +```bash +sh ./start.sh +``` + +- In case the start.sh does not seem to be runnable, use + +```bash +chmod 400 start.sh +``` + +- To run project without docker , follow these steps; Also change correct path of dataset folder in backend/api/**init**.py + +```bash +cd backend +python3 -m venv env [Create a python environment] +source env/bin/activate +(env) pip install -r requirements.txt +(env) export FLASK_APP=api/__init__.py +(env) export REACT_APP_BACKEND_SERVICE_URL=http://localhost:5000 +(env) python manage.py run [Navigate to http://localhost:5000/api/test for sanity check] + + +cd frontend +npm install +export REACT_APP_BACKEND_SERVICE_URL=http://localhost:5000 +npm start [Automatically launches app in your default browser on http://localhost:3000] +``` + +# Technology + +## Backend + +- Python v3.8.6 +- Flask v1.1.2 +- Docker v19.03.1 +- Docker Compose v1.21.0 +- Docker Compose file v3.3 +- Nginx v1.15.9 +- Guicorn v19.9.0 + +## Frontend + +- Node v12.18.3 +- Npm v6.14.7 +- React v16.14.0 +- React Scripts v2.1.8 +- React Dom v16.14.0 + +# Dependencies + +## Backend + +- Flask Testing v0.8.0 - For unit testing within flask +- h5py v3.1.0 - For extracting of big data from HDF5 binary data format +- numpy v1.19.4 To structure multi-dimensional data extracted by h5py into numpy matrices for data aggregation +- Coverage v5.3 - For checking coverage of code tested +- Flask CORS v3.0.9 - To allow for cross-origin AJAX requests +- tzlocal v2.1 - To convert unix timestamps to local timezone + +## Frontend + +- Axios v0.18.1 - For making AJAX calls to the server(backend) +- react-bootstrap v1.4.0 - For app CSS styling and loading spinner +- React-PlotlyJS/PlotlyJS v2.4.0/1.45.3 - For charts shown on dashboard +- Enzyme v3.9.0 - Utility library used for testing react components +- react-fontawesome v0.1.13 - For light/dark mode icon + +# EndPoints + +You can test out the following endpoints: + +``` + + Endpoint HTTP Method CRUD Method Result + / GET READ Load React App + /api/test GET READ Sanity Check + /api/get_file_names GET READ Get file names + /api/get_glucose_data POST READ Get Glucose data from file + /api/get_measurement_data POST READ Get measurement data from file +``` + +# Screenshots + +## When App First Renders + + +### Choose data file and follow steps to see glucose and corresponding spectra as shown in Results + + + +## Results + + + diff --git a/backend/Procfile b/backend/Procfile deleted file mode 100644 index 6f18d33..0000000 --- a/backend/Procfile +++ /dev/null @@ -1 +0,0 @@ -web: gunicorn api:app \ No newline at end of file diff --git a/backend/api/__init__.py b/backend/api/__init__.py index c60c510..457a041 100644 --- a/backend/api/__init__.py +++ b/backend/api/__init__.py @@ -9,12 +9,11 @@ import tzlocal -app = Flask(__name__) -cors = CORS(app, resources={r"/*": {"origins": "http//localhost"}}) use this -# /usr/src/app # change directory here for correct path to dataset - without docker -FOLDER_PATH = os.path.dirname(os.path.dirname(__file__))+"/dataset/" #Deployment to Heroku +app = Flask(__name__) +cors = CORS(app, resources={r"/*": {"origins": "http://localhost"}}) +FOLDER_PATH = "/usr/src/app/dataset/" # change directory here for correct path to dataset - without docker @app.route("/api/test") def test_sanity(): @@ -33,7 +32,6 @@ def get_file_names(): @app.route('/api/get_glucose_data', methods=['POST']) def get_glucose_data(): post_data = request.get_json() - print(post_data) file_name = post_data.get("file_name") if file_name: full_file_path = FOLDER_PATH + file_name @@ -78,4 +76,4 @@ def get_data_from_file(path): if __name__ == "__main__": - app.run(host="0.0.0.0", debug=True,port=5000) \ No newline at end of file + app.run(debug=True) \ No newline at end of file diff --git a/backend/api/__pycache__/__init__.cpython-38.pyc b/backend/api/__pycache__/__init__.cpython-38.pyc index 11aab85..32329b5 100644 Binary files a/backend/api/__pycache__/__init__.cpython-38.pyc and b/backend/api/__pycache__/__init__.cpython-38.pyc differ diff --git a/frontend/.dockerignore b/frontend/.dockerignore index 38defd2..52d122e 100644 --- a/frontend/.dockerignore +++ b/frontend/.dockerignore @@ -3,4 +3,3 @@ coverage build .dockerignore Dockerfile -Dockerfile-prod \ No newline at end of file diff --git a/frontend/.gitignore b/frontend/.gitignore deleted file mode 100644 index 4d29575..0000000 --- a/frontend/.gitignore +++ /dev/null @@ -1,23 +0,0 @@ -# See https://help.github.com/articles/ignoring-files/ for more about ignoring files. - -# dependencies -/node_modules -/.pnp -.pnp.js - -# testing -/coverage - -# production -/build - -# misc -.DS_Store -.env.local -.env.development.local -.env.test.local -.env.production.local - -npm-debug.log* -yarn-debug.log* -yarn-error.log* diff --git a/frontend/src/components/Dashboard/Dashboard.jsx b/frontend/src/components/Dashboard/Dashboard.jsx index b14b04a..b8e92f3 100644 --- a/frontend/src/components/Dashboard/Dashboard.jsx +++ b/frontend/src/components/Dashboard/Dashboard.jsx @@ -23,7 +23,7 @@ function Dashboard(props) { useEffect(() => { axios - .get(`https://irubis-dashboard.herokuapp.com/api/get_file_names`) + .get(`${process.env.REACT_APP_BACKEND_SERVICE_URL}/api/get_file_names`) .then((response) => { if (response.data.status === true) { setFileNames(response.data.data); @@ -65,7 +65,7 @@ function Dashboard(props) { const data = { file_name: name, index: 10 }; axios .post( - `https://irubis-dashboard.herokuapp.com/api/get_glucose_data`, + `${process.env.REACT_APP_BACKEND_SERVICE_URL}/api/get_glucose_data`, data, ) .then((response) => { @@ -84,14 +84,13 @@ function Dashboard(props) { let name = eventData.points[0]["data"]["name"]; let index = eventData.points[0]["pointIndex"]; let glucoseVal = eventData.points[0]["y"]; - console.log(glucoseVal, glucoseValue); const data = { file_name: name, index }; if (glucoseValue.length > 0 && glucoseValue.includes(glucoseVal)) { return; } else { axios .post( - `https://irubis-dashboard.herokuapp.com/api/get_measurement_data`, + `${process.env.REACT_APP_BACKEND_SERVICE_URL}/api/get_measurement_data`, data, ) .then((response) => { @@ -117,7 +116,6 @@ function Dashboard(props) { }; setGlucoseValue((p) => [...p, glucoseVal]); setSecondPlot((p) => [...p, meas]); - } else { } }) .catch((err) => console.log(err)); @@ -133,10 +131,7 @@ function Dashboard(props) { const popover = (