BIBBOX developer guide
Basically the BIBBOX itsself consists of a series of docker containers. The basic container structure is given by:
- Apache Proxyserver (Front-End Server and Proxy Server for the Apps)
- Backend (Flask Rest API for data exchange from Front-End)
- Postgres (Permanent Data Storage)
- Celery (Asynchronous Task scheduling)
- Redis (Faster data-store for volatile and cache data)
For more information about each container itsself see the sys-bibbox GitHub repository at sys-bibbox.
Developer Documentation
Project structure and outline
The BIBBOX docker system can mainly be divided into 3 parts:
-
Front End
- Angular Front End (Precompiled and mounted within the proxy container frontend)
-
Back End
- Flask Backend container (backend)
- Celery Container (Celery and backend use the same Docker image)
- Postgres DB (postgresql)
- Redis Data Store (Official Redis Docker image. Used by Celery as cache store)
- Adminer (Official Adminer Docker image for DB debugging)
- Celery Monitor (Celery container with Frotnend to debug asynchronous tasks)
- cadvisor (cadvisor monitoring for the complete docker-compose bibbox chain)
-
Apacheproxy (Main linking centerpoint apacheproxy)
To get an idea how this works in an easier fashion you can have a look at Angular-Flask-Docker-Skeleton, which served as base for the work done on the BIBBOX docker framework. The nginx implementation was replaced by Apache since this offered a more convenient workflow.
Prequisites
The following applications need to be installed or are installed using the INSTALL.sh script:
- Docker and docker-compose
- nodejs, nvm (Node version manager) and npm (node package manager) (see INSTALL.sh for version details)
- Python 3.x (see requirements.txt in the repository for more information on python requirements)
Front-End Details
- The Frontend is angular based and uses the ngrx-store principle to achieve the desired functionality (https://ngrx.io/guide/store). Code Structure:
-
Part I: Everything visible in the Frontend is a component uses functions defined in the ngrx store part.
sys-bibbox/frontend/src
├── app
│ ├── app.module.ts
│ ├── app-routing.module.ts
│ ├── commons.ts
│ ├── components
│ │ ├── about
│ │ │ ├── contact
│ │ │ │ ├── contact.component.html
│ │ │ │ ├── contact.component.scss
│ │ │ │ ├── contact.component.spec.ts
│ │ │ │ └── contact.component.ts
│ │ │ ├── imprint
│ │ │ │ ├── imprint.component.html
│ │ │ │ ├── imprint.component.scss
│ │ │ │ ├── imprint.component.spec.ts
│ │ │ │ └── imprint.component.ts
│ │ │ └── partners
│ │ │ ├── partners.component.html
│ │ │ ├── partners.component.scss
│ │ │ ├── partners.component.spec.ts
│ │ │ └── partners.component.ts
│ │ ├── activities
│ │ │ ├── activities.component.html
│ │ │ ├── activities.component.scss
│ │ │ ├── activities.component.spec.ts
│ │ │ ├── activities.component.ts
│ │ │ └── activity-menu-overlay
│ │ │ ├── activity-menu-overlay.component.html
│ │ │ ├── activity-menu-overlay.component.scss
│ │ │ ├── activity-menu-overlay.component.spec.ts
│ │ │ └── activity-menu-overlay.component.ts
│ │ ├── applications
│ │ │ ├── application-group
│ │ │ │ ├── application-group.component.html
│ │ │ │ ├── application-group.component.scss
│ │ │ │ ├── application-group.component.spec.ts
│ │ │ │ ├── application-group.component.ts
│ │ │ │ └── application-tile
│ │ │ │ ├── application-tile.component.html
│ │ │ │ ├── application-tile.component.scss
│ │ │ │ ├── application-tile.component.spec.ts
│ │ │ │ └── application-tile.component.ts
│ │ │ ├── applications.component.html
│ │ │ ├── applications.component.scss
│ │ │ ├── applications.component.spec.ts
│ │ │ ├── applications.component.ts
│ │ │ ├── install-screen
│ │ │ │ ├── install-screen.component.html
│ │ │ │ ├── install-screen.component.scss
│ │ │ │ ├── install-screen.component.spec.ts
│ │ │ │ └── install-screen.component.ts
│ │ │ └── install-screen-dialog
│ │ │ ├── install-screen-dialog.component.html
│ │ │ ├── install-screen-dialog.component.scss
│ │ │ ├── install-screen-dialog.component.spec.ts
│ │ │ └── install-screen-dialog.component.ts
│ │ ├── app-scaffold
│ │ │ ├── app.component.html
│ │ │ ├── app.component.scss
│ │ │ ├── app.component.spec.ts
│ │ │ ├── app.component.ts
│ │ │ ├── footer
│ │ │ │ ├── footer.component.html
│ │ │ │ ├── footer.component.scss
│ │ │ │ ├── footer.component.spec.ts
│ │ │ │ └── footer.component.ts
│ │ │ └── header
│ │ │ ├── header.component.html
│ │ │ ├── header.component.scss
│ │ │ ├── header.component.spec.ts
│ │ │ └── header.component.ts
│ │ ├── instances
│ │ │ ├── instance-detail-page
│ │ │ │ ├── instance-detail-page.component.html
│ │ │ │ ├── instance-detail-page.component.scss
│ │ │ │ ├── instance-detail-page.component.spec.ts
│ │ │ │ └── instance-detail-page.component.ts
│ │ │ ├── instances.component.html
│ │ │ ├── instances.component.scss
│ │ │ ├── instances.component.spec.ts
│ │ │ ├── instances.component.ts
│ │ │ └── instance-tile
│ │ │ ├── instance-tile.component.html
│ │ │ ├── instance-tile.component.scss
│ │ │ ├── instance-tile.component.spec.ts
│ │ │ └── instance-tile.component.ts
│ │ ├── login TODO
│ │ │ ├── login.component.html
│ │ │ ├── login.component.scss
│ │ │ ├── login.component.spec.ts
│ │ │ └── login.component.ts
│ │ ├── not-found
│ │ │ ├── not-found.component.html
│ │ │ ├── not-found.component.scss
│ │ │ ├── not-found.component.spec.ts
│ │ │ └── not-found.component.ts
│ │ └── sys-logs
│ │ ├── sys-logs.component.html
│ │ ├── sys-logs.component.scss
│ │ ├── sys-logs.component.spec.ts
│ │ └── sys-logs.component.ts
│ ├── httperror.interceptor.ts -
Part II: Store part implement all the actions, effects, models, reducers, selectors and services:
│ └── store
│ ├── actions
│ │ ├── activity.actions.ts
│ │ ├── applications.actions.ts
│ │ └── instance.actions.ts
│ ├── effects
│ │ ├── activity.effects.ts
│ │ ├── applications.effects.ts
│ │ └── instance.effects.ts
│ ├── models
│ │ ├── activity.model.ts
│ │ ├── application-group-item.model.ts
│ │ ├── app-state.model.ts
│ │ └── instance-item.model.ts
│ ├── reducers
│ │ ├── activity.reducer.ts
│ │ ├── application-group.reducer.ts
│ │ └── instance.reducer.ts
│ ├── selectors
│ │ ├── activity.selector.ts
│ │ ├── application-group.selector.ts
│ │ └── instance.selector.ts
│ └── services
│ ├── activity.service.spec.ts
│ ├── activity.service.ts
│ ├── application.service.spec.ts
│ ├── application.service.ts
│ ├── auth.service.spec.ts
│ ├── auth.service.ts
│ ├── instance.service.spec.ts
│ ├── instance.service.ts
│ ├── login.service.spec.ts
│ ├── login.service.ts
│ ├── socketio.service.spec.ts
│ ├── socketio.service.ts
│ ├── validator.service.spec.ts
│ └── validator.service.ts
├── app.config.ts (created by Angular CLI)
Note that this exactly resembels the structure given in the NGRx tutorial (https://ngrx.io/guide/store). - Part III images and build environments
├── assets
│ ├── announced.png
│ ├── b3africa.png
│ ├── bbmri-eric.png
│ ├── close.png
│ ├── done.png
│ ├── error.png
│ ├── external_ref.png
│ ├── favicon.ico
│ ├── furley_bg.png
│ ├── loading.gif
│ ├── loading_old.gif
│ ├── lock.png
│ ├── new.png
│ ├── pawn_small.png
│ ├── silicolab_logo.png
│ ├── silicolab_logo_small.png
│ ├── silicolab_logo.svg
│ └── silicolab_logo_white.png
├── environments
│ ├── environment.prod.ts
│ ├── environment.prod.ts.template
│ ├── environment.ts
│ └── environment.ts.template - Standart angular files mostly generated by the Angular CLI (edited in cases)
├── favicon.ico
├── index.html
├── main.ts
├── polyfills.ts
├── proxy.conf.json
├── styles.scss
├── styles-variables.scss
└── test.ts
Front end TODO'S:
- Authentication for users;
- See tutorial here authentication-in-angular-with-ngrx
- Styling
Back-End Details
The Backend code can be grouped the backend code regarding:
- api: Code implementing the Flask API calls it uses code defined in
- the BIBBOX Folder: Implements the file handling and data I/O providing the neccesary JSON data to the API calls.
- celery config files and tasks
- models contains the DB implementations for the database
- services implements the SQL-Alchemy services consuming the models
- static features Api spec files in different formats
- util features constants and globals in a single script
- Other files cover init and configs for falsk and the neccesary interfaces
/opt/bibbox/sys-bibbox/backend/
├── app
* Part I: Main Code influencing Front end behaviour
│ ├── api
│ │ ├── activity.py
│ │ ├── apps.py
│ │ ├── authentication.py
│ │ ├── default.py
│ │ ├── __init__.py
│ │ └── instance.py
│ ├── bibbox
│ │ ├── app.py
│ │ ├── docker_handler.py
│ │ ├── file_handler.py
│ │ ├── __init__.py
│ │ ├── instance_controler.py
│ │ ├── instance_handler.py
│ │ └── instance.py
* Part II DB and asynchronous Tasks
│ ├── celeryconfig.py
│ ├── celerytasks
│ │ ├── __init__.py
│ │ └── tasks.py
│ ├── dirty_test_code.py
│ ├── __init__.py
│ ├── models
│ │ ├── activity.py
│ │ ├── app.py
│ │ ├── catalogue.py
│ │ ├── __init__.py
│ │ ├── log.py
│ │ └── user.py
│ ├── services
│ │ ├── activity_service.py
│ │ ├── app_service.py
│ │ ├── catalogue_service.py
│ │ ├── db_logger_service.py
│ │ ├── __init__.py
│ │ ├── log_service.py
│ │ ├── socketio_service.py
│ │ └── user_service.py
│ ├── static
│ │ ├── bibbox-api-spec old.yml
│ │ ├── bibbox-api-spec.yml
│ │ ├── main.html
│ │ └── swagger.json
│ ├── utility
│ │ ├── celery_util.py
│ │ └── __init__.py
│ └── utils
│ ├── common.py
│ └── __init__.py
* Part III: Init Stuff for websockets and Falsk + logs
├── celery_worker.py
├── debug-test.py
├── Dockerfile
├── entrypoint.sh
├── __init__.py
├── logs
│ ├── debug.log
│ ├── error.log
│ ├── info.log
│ └── warning.log
├── manage.py
├── requirements.txt
├── runflask.py
├── settings.py
├── uwsgi.ini
├── uwsgi_params
└── wsgi.py
Set up Front-End Developement environment
-
Easiest way (Linux Debian based):
- Install BIBBOX locally under linux (See Install BIBBOX)
- Install an IDE (e.g VSCode)
- Set up local DNS Service (See Install BIBBOX)
- Goto sys-bibbox/frontend
- Type
code .
into bash - After VS Code opens type
ng serve
- Open the link the console shows you (If setup went correctly you should see a BIBBOX interface without any errors)
- Change any code file and watch it change live in your browser.
-
NOTE: Prequesite is that the DNS service and BIBBOX installation fully works
Apps Creators Manual
How to join the team
Contact Heimo Müller, Emilian Jungwirth or Markus Plass. Any of them can add you to the Github team of an App repository.
Anatomy of an App
An App is described within a BIBBOX GitHub repository. By convention the name of the repository starts with the prefix "app-" A template repository can be found at: https://github.com/bibbox/app-template
An App consists at least of the following files and directories, please never change the name of these files!
- README.md
- INSTALL-APP.md
- appinfo.json
- docker-compose.yml.template
- environment-parameters.json
- fileinfo.json
- icon.png
README.md
The default Github readme file shoud contain information about the used official docker images and docker images from the BIBBOX dockerhub/. In addition you should describe all mounted volumes and their content.
INSTALL-APP.md
Install instruction for the App to follow after the first installation step (docker-compose up) is finished. This typically describes application specific configuration tasks. A link to this file is given in the Dashboard of the installed App.
appinfo.json
Specifies information about the App as displayed in the BIBBOX App Store.
{
"name": "A long name of the Application, shown in the App Store",
"short_name": "APP short name",
"version": "V.1.0",
"description": "Long description of the Application shown in Store, can be multiline",
"short_description": "Short description of the Application shown in Store",
"catalogue_url": "URL to biobankapps, if applicaple",
"application_url": "URL to App official gace",
"tags": ["Tag1", "Tag2"]
}
docker-compose.yml.template
Based on this file the docker-compose.yml will be generated. The variables §§INSTANCE and all other environment and configuration variables starting with "§§" will be replaced during the installation. Each template for a docker compose file should link to the bibbox-default-network
network. The proxy
and ports
section is used to create a 005-§§INSTANCE.conf
proxy file.
NOTE:
- Only use tagged images versions (not latest). To ensure that the app stays stable over time.
- Use the docker-compose
links
parameter to ensure that containers can "talk" to each other indent of the name set in §§INSTANCE.
version: '3'
networks:
bibbox-default-network:
external: true
services:
§§INSTANCE-container-frontend:
image: bibbox/app-image:vx.y.z
container_name: §§INSTANCE-container-frontend
restart: unless-stopped
networks:
- bibbox-default-network
links:
- §§INSTANCE-container2:container2
ports:
- "80:80"
depends_on:
- §§INSTANCE-container2
volumes:
- ./data/container/app-data:/app-data
proxy:
TYPE: PRIMARY
URLPREFIX: §§INSTANCE
TEMPLATE: default
DISPLAYNAME: 'APP name'
§§INSTANCE-container2:
image: bibbox/app-image2:va.b.c
container_name: §§INSTANCE-container2
restart: unless-stopped
networks:
- bibbox-default-network
links:
- §§INSTANCE-app-db:app-db
ports:
- "8000:8000"
depends_on:
- §§INSTANCE-app-db
volumes:
- ./data/container/app-data:/app-data
§§INSTANCE-app-db:
image: mysql:8
container_name: §§INSTANCE-app-db
restart: unless-stopped
networks:
- bibbox-default-network
user: root
environment:
- MYSQL_ROOT_PASSWORD=thispasswordisneverusededoutsidethecontainer
- MYSQL_DATABASE=§§MYSQL_DATABASE_NAME
- MYSQL_USER=§§MYSQL_DATABASE_USER
- MYSQL_PASSWORD=§§MYSQL_DATABASE_PASSWORD
volumes:
- ./data/db/var/app-data:/var/app-data
environment-parameters.json
Environment parameters for the docker-compose-template file. The user can set the values for these parameters in a GUI during the installation of an App.
[
{
"id":"NAME_AS_IN_DOCKER_TEMPLATE_1",
"display_name":"Name Displayed in GUI at the Installation",
"type":"text, number, password",
"default_value":"Default Value",
"description":"Description shown in the GUI",
"min_length": "1",
"max_length": "128"
},
{
"id":"MYSQL_DATABASE_NAME",
"display_name":"Name of the mysql database",
"type":"text,",
"default_value":"Default Value",
"description":"Description shown in the GUI",
"min_length": "1",
"max_length": "128"
},
{
"id":"MYSQL_DATABASE_USER",
"display_name":"Name of the mysql user",
"type":"text",
"default_value":"Default Value",
"description":"Description shown in the GUI",
"min_length": "1",
"max_length": "128"
},
{
"id":"MYSQL_DATABASE_PASSWORD",
"display_name":"Password of mySQL user",
"type":"password",
"default_value":"Default Value",
"description":"Description shown in the GUI",
"min_length": "1",
"max_length": "128"
}
]
fileinfo.json
A list of commands to be run when installing an app, consisting of three categories: "environmental_replace", "script_replace" and "permissions".
key | description |
---|---|
environmental_replace | WIP - specifies files/directories where placeholders can be replaced easily |
script_replace | WIP - specifies files/directories where placeholders etc. should be replaced by a script |
permissions | specifies which files/folders in the instance directory should be set to which permissions. \n the path is always relative to the instance directory |
Tool for calculating permissions: chmod-calculator
structure:
{
"environment_replace": {},
"script_replace": [],
"permissions": {
"PATH1": NUMERIC_VALUE_OF_PERMISSION1,
"PATH2": NUMERIC_VALUE_OF_PERMISSION2,
"PATH3": NUMERIC_VALUE_OF_PERMISSION3
}
}
example:
{
"environment_replace": {},
"script_replace": [],
"permissions": {
"assets": 775,
"data" : 777
}
}
icon.png
Icon of the App in the application store und in the application dashboard. Please use a square format, e.g. 512x512 pixeland PNG with a transparency channel.
Versioning
Each App should be versioned. We distinguish between
-
Production Versions generated from a specific release in the repository. For each major release. i.e. given by a new version of dockerized software tool, a new branch should be generated in the repository. Please note, that the tag name for the Github release should contain the APP version and the BIBBOX APP release (adaption of the APP for BIBBOX). The proposed format is vAPP_verion_bibboxrel3 digit bibbox_release_number e.g.
v8.7.2_bibboxrel001
. Thelatest
stable release is in the master branch. -
Development Version comming soon. Branches and tags not specified in a kit can be loaded in a Developer Mode.
When creating a production version, you should make the follwoing steps. Please make it exactly in this order, otherwise the tagging of the bibbox docker/hub and bibbox github will get confused.
-
Generate a new branch in github, if you plan to release a major version.
-
Once a stable bibbox release is ready create a tag in compliance with the naming convention (e.g.
v8.7.2_bibboxrel001
). -
The latest app version should be merged in the master branch if a stable BIBBOX release exists via a pull request.
-
-
Update all the files for the anticipated version, don't forget to update appinfo.json.
-
Set in the docker compose template file (docker-compose.yml.template) the tags for the docker images. Please note, that ALL used images should be tagged with an specific version, don't use the 'latest" tag, as this could break your APP in the future.
-
Add the tag in the DOCKER hub to build tagged images (press SAVE before TRIGGER)
-
Edit in the used kit, e.g. bibbox.json the version information. This file can be found in the application-store repository.
Register an App
For the registration of an App two configuration files have to be extended:
-
applications.json/ maps a Github repository to a human readable name of the App
-
bibbox.json/ puts the App into a specific kit. A kit defines ta group of Apps together with metadata, which are then displayed in the App store and can be installed in a BIBBOX instance.