BIBBOX developer guide

Basically the BIBBOX itsself consists of a series of docker containers. The basic container structure is given by:

• Apache Proxyserver (Front-End Server and Proxy Server for the Apps)
• Backend (Flask Rest API for data exchange from Front-End)
• Postgres (Permanent Data Storage)
• Celery (Asynchronous Task scheduling)
• Redis (Faster data-store for volatile and cache data)

For more information about each container itsself see the sys-bibbox GitHub repository at sys-bibbox.

Developer Documentation

Project structure and outline

The BIBBOX docker system can mainly be divided into 3 parts:

• Front End

  • Angular Front End (Precompiled and mounted within the proxy container frontend)

• Back End

  • Flask Backend container (backend)
  • Celery Container (Celery and backend use the same Docker image)
  • Postgres DB (postgresql)
  • Redis Data Store (Official Redis Docker image. Used by Celery as cache store)
  • Adminer (Official Adminer Docker image for DB debugging)
  • Celery Monitor (Celery container with Frotnend to debug asynchronous tasks)
  • cadvisor (cadvisor monitoring for the complete docker-compose bibbox chain)

• Apacheproxy (Main linking centerpoint apacheproxy)

To get an idea how this works in an easier fashion you can have a look at Angular-Flask-Docker-Skeleton, which served as base for the work done on the BIBBOX docker framework. The nginx implementation was replaced by Apache since this offered a more convenient workflow.

Prequisites

The following applications need to be installed or are installed using the INSTALL.sh script:

• Docker and docker-compose
• nodejs, nvm (Node version manager) and npm (node package manager) (see INSTALL.sh for version details)
• Python 3.x (see requirements.txt in the repository for more information on python requirements)

Front-End Details

• The Frontend is angular based and uses the ngrx-store principle to achieve the desired functionality (https://ngrx.io/guide/store). Code Structure:

• Part I: Everything visible in the Frontend is a component uses functions defined in the ngrx store part.
  sys-bibbox/frontend/src
  ├── app
  │   ├── app.module.ts
  │   ├── app-routing.module.ts
  │   ├── commons.ts
  │   ├── components
  │   │   ├── about
  │   │   │   ├── contact
  │   │   │   │   ├── contact.component.html
  │   │   │   │   ├── contact.component.scss
  │   │   │   │   ├── contact.component.spec.ts
  │   │   │   │   └── contact.component.ts
  │   │   │   ├── imprint
  │   │   │   │   ├── imprint.component.html
  │   │   │   │   ├── imprint.component.scss
  │   │   │   │   ├── imprint.component.spec.ts
  │   │   │   │   └── imprint.component.ts
  │   │   │   └── partners
  │   │   │   ├── partners.component.html
  │   │   │   ├── partners.component.scss
  │   │   │   ├── partners.component.spec.ts
  │   │   │   └── partners.component.ts
  │   │   ├── activities
  │   │   │   ├── activities.component.html
  │   │   │   ├── activities.component.scss
  │   │   │   ├── activities.component.spec.ts
  │   │   │   ├── activities.component.ts
  │   │   │   └── activity-menu-overlay
  │   │   │   ├── activity-menu-overlay.component.html
  │   │   │   ├── activity-menu-overlay.component.scss
  │   │   │   ├── activity-menu-overlay.component.spec.ts
  │   │   │   └── activity-menu-overlay.component.ts
  │   │   ├── applications
  │   │   │   ├── application-group
  │   │   │   │   ├── application-group.component.html
  │   │   │   │   ├── application-group.component.scss
  │   │   │   │   ├── application-group.component.spec.ts
  │   │   │   │   ├── application-group.component.ts
  │   │   │   │   └── application-tile
  │   │   │   │   ├── application-tile.component.html
  │   │   │   │   ├── application-tile.component.scss
  │   │   │   │   ├── application-tile.component.spec.ts
  │   │   │   │   └── application-tile.component.ts
  │   │   │   ├── applications.component.html
  │   │   │   ├── applications.component.scss
  │   │   │   ├── applications.component.spec.ts
  │   │   │   ├── applications.component.ts
  │   │   │   ├── install-screen
  │   │   │   │   ├── install-screen.component.html
  │   │   │   │   ├── install-screen.component.scss
  │   │   │   │   ├── install-screen.component.spec.ts
  │   │   │   │   └── install-screen.component.ts
  │   │   │   └── install-screen-dialog
  │   │   │   ├── install-screen-dialog.component.html
  │   │   │   ├── install-screen-dialog.component.scss
  │   │   │   ├── install-screen-dialog.component.spec.ts
  │   │   │   └── install-screen-dialog.component.ts
  │   │   ├── app-scaffold
  │   │   │   ├── app.component.html
  │   │   │   ├── app.component.scss
  │   │   │   ├── app.component.spec.ts
  │   │   │   ├── app.component.ts
  │   │   │   ├── footer
  │   │   │   │   ├── footer.component.html
  │   │   │   │   ├── footer.component.scss
  │   │   │   │   ├── footer.component.spec.ts
  │   │   │   │   └── footer.component.ts
  │   │   │   └── header
  │   │   │   ├── header.component.html
  │   │   │   ├── header.component.scss
  │   │   │   ├── header.component.spec.ts
  │   │   │   └── header.component.ts
  │   │   ├── instances
  │   │   │   ├── instance-detail-page
  │   │   │   │   ├── instance-detail-page.component.html
  │   │   │   │   ├── instance-detail-page.component.scss
  │   │   │   │   ├── instance-detail-page.component.spec.ts
  │   │   │   │   └── instance-detail-page.component.ts
  │   │   │   ├── instances.component.html
  │   │   │   ├── instances.component.scss
  │   │   │   ├── instances.component.spec.ts
  │   │   │   ├── instances.component.ts
  │   │   │   └── instance-tile
  │   │   │   ├── instance-tile.component.html
  │   │   │   ├── instance-tile.component.scss
  │   │   │   ├── instance-tile.component.spec.ts
  │   │   │   └── instance-tile.component.ts
  │   │   ├── login TODO
  │   │   │   ├── login.component.html
  │   │   │   ├── login.component.scss
  │   │   │   ├── login.component.spec.ts
  │   │   │   └── login.component.ts
  │   │   ├── not-found
  │   │   │   ├── not-found.component.html
  │   │   │   ├── not-found.component.scss
  │   │   │   ├── not-found.component.spec.ts
  │   │   │   └── not-found.component.ts
  │   │   └── sys-logs
  │   │   ├── sys-logs.component.html
  │   │   ├── sys-logs.component.scss
  │   │   ├── sys-logs.component.spec.ts
  │   │   └── sys-logs.component.ts
  │   ├── httperror.interceptor.ts
  

• Part II: Store part implement all the actions, effects, models, reducers, selectors and services:
  │   └── store
  │   ├── actions
  │   │   ├── activity.actions.ts
  │   │   ├── applications.actions.ts
  │   │   └── instance.actions.ts
  │   ├── effects
  │   │   ├── activity.effects.ts
  │   │   ├── applications.effects.ts
  │   │   └── instance.effects.ts
  │   ├── models
  │   │   ├── activity.model.ts
  │   │   ├── application-group-item.model.ts
  │   │   ├── app-state.model.ts
  │   │   └── instance-item.model.ts
  │   ├── reducers
  │   │   ├── activity.reducer.ts
  │   │   ├── application-group.reducer.ts
  │   │   └── instance.reducer.ts
  │   ├── selectors
  │   │   ├── activity.selector.ts
  │   │   ├── application-group.selector.ts
  │   │   └── instance.selector.ts
  │   └── services
  │   ├── activity.service.spec.ts
  │   ├── activity.service.ts
  │   ├── application.service.spec.ts
  │   ├── application.service.ts
  │   ├── auth.service.spec.ts
  │   ├── auth.service.ts
  │   ├── instance.service.spec.ts
  │   ├── instance.service.ts
  │   ├── login.service.spec.ts
  │   ├── login.service.ts
  │   ├── socketio.service.spec.ts
  │   ├── socketio.service.ts
  │   ├── validator.service.spec.ts
  │   └── validator.service.ts
  ├── app.config.ts (created by Angular CLI)
  Note that this exactly resembels the structure given in the NGRx tutorial (https://ngrx.io/guide/store).
  

• Part III images and build environments
  ├── assets
  │   ├── announced.png
  │   ├── b3africa.png
  │   ├── bbmri-eric.png
  │   ├── close.png
  │   ├── done.png
  │   ├── error.png
  │   ├── external_ref.png
  │   ├── favicon.ico
  │   ├── furley_bg.png
  │   ├── loading.gif
  │   ├── loading_old.gif
  │   ├── lock.png
  │   ├── new.png
  │   ├── pawn_small.png
  │   ├── silicolab_logo.png
  │   ├── silicolab_logo_small.png
  │   ├── silicolab_logo.svg
  │   └── silicolab_logo_white.png
  ├── environments
  │   ├── environment.prod.ts
  │   ├── environment.prod.ts.template
  │   ├── environment.ts
  │   └── environment.ts.template
  
• Standart angular files mostly generated by the Angular CLI (edited in cases)
  ├── favicon.ico
  ├── index.html
  ├── main.ts
  ├── polyfills.ts
  ├── proxy.conf.json
  ├── styles.scss
  ├── styles-variables.scss
  └── test.ts
  

Front end TODO'S:

• Authentication for users;
  • See tutorial here authentication-in-angular-with-ngrx
• Styling

Back-End Details

The Backend code can be grouped the backend code regarding:

• api: Code implementing the Flask API calls it uses code defined in
  • the BIBBOX Folder: Implements the file handling and data I/O providing the neccesary JSON data to the API calls.
• celery config files and tasks
• models contains the DB implementations for the database
  • services implements the SQL-Alchemy services consuming the models
• static features Api spec files in different formats
• util features constants and globals in a single script
• Other files cover init and configs for falsk and the neccesary interfaces

/opt/bibbox/sys-bibbox/backend/
├── app
* Part I: Main Code influencing Front end behaviour
│   ├── api
│   │   ├── activity.py
│   │   ├── apps.py
│   │   ├── authentication.py
│   │   ├── default.py
│   │   ├── __init__.py
│   │   └── instance.py
│   ├── bibbox
│   │   ├── app.py
│   │   ├── docker_handler.py
│   │   ├── file_handler.py
│   │   ├── __init__.py
│   │   ├── instance_controler.py
│   │   ├── instance_handler.py
│   │   └── instance.py
* Part II DB and asynchronous Tasks
│   ├── celeryconfig.py
│   ├── celerytasks
│   │   ├── __init__.py
│   │   └── tasks.py
│   ├── dirty_test_code.py
│   ├── __init__.py
│   ├── models
│   │   ├── activity.py
│   │   ├── app.py
│   │   ├── catalogue.py
│   │   ├── __init__.py
│   │   ├── log.py
│   │   └── user.py
│   ├── services
│   │   ├── activity_service.py
│   │   ├── app_service.py
│   │   ├── catalogue_service.py
│   │   ├── db_logger_service.py
│   │   ├── __init__.py
│   │   ├── log_service.py
│   │   ├── socketio_service.py
│   │   └── user_service.py
│   ├── static
│   │   ├── bibbox-api-spec old.yml
│   │   ├── bibbox-api-spec.yml
│   │   ├── main.html
│   │   └── swagger.json
│   ├── utility
│   │   ├── celery_util.py
│   │   └── __init__.py
│   └── utils
│   ├── common.py
│   └── __init__.py
* Part III: Init Stuff for websockets and Falsk + logs
├── celery_worker.py
├── debug-test.py
├── Dockerfile
├── entrypoint.sh
├── __init__.py
├── logs
│   ├── debug.log
│   ├── error.log
│   ├── info.log
│   └── warning.log
├── manage.py
├── requirements.txt
├── runflask.py
├── settings.py
├── uwsgi.ini
├── uwsgi_params
└── wsgi.py

Set up Front-End Developement environment

• Easiest way (Linux Debian based):
  

  • Install BIBBOX locally under linux (See Install BIBBOX)
  • Install an IDE (e.g VSCode)
  • Set up local DNS Service (See Install BIBBOX)
  • Goto sys-bibbox/frontend
  • Type code . into bash
  • After VS Code opens type ng serve
  • Open the link the console shows you (If setup went correctly you should see a BIBBOX interface without any errors)
  • Change any code file and watch it change live in your browser.

• NOTE: Prequesite is that the DNS service and BIBBOX installation fully works

Apps Creators Manual

How to join the team

Contact Heimo Müller, Emilian Jungwirth or Markus Plass. Any of them can add you to the Github team of an App repository.

Anatomy of an App

An App is described within a BIBBOX GitHub repository. By convention the name of the repository starts with the prefix "app-" A template repository can be found at: https://github.com/bibbox/app-template

An App consists at least of the following files and directories, please never change the name of these files!

• README.md
• INSTALL-APP.md
• appinfo.json
• docker-compose.yml.template
• environment-parameters.json
• fileinfo.json
• icon.png

README.md

The default Github readme file shoud contain information about the used official docker images and docker images from the BIBBOX dockerhub/. In addition you should describe all mounted volumes and their content.

INSTALL-APP.md

Install instruction for the App to follow after the first installation step (docker-compose up) is finished. This typically describes application specific configuration tasks. A link to this file is given in the Dashboard of the installed App.

appinfo.json

Specifies information about the App as displayed in the BIBBOX App Store.

    {
            "name": "A long name of the Application, shown in the App Store",
            "short_name": "APP short name",
            "version": "V.1.0",
            "description": "Long description of the Application shown in Store, can be multiline",
            "short_description": "Short description of the Application shown in Store",
            "catalogue_url": "URL to biobankapps, if applicaple",
            "application_url": "URL to App official gace",
            "tags": ["Tag1", "Tag2"]
    }

docker-compose.yml.template

Based on this file the docker-compose.yml will be generated. The variables §§INSTANCE and all other environment and configuration variables starting with "§§" will be replaced during the installation. Each template for a docker compose file should link to the bibbox-default-network network. The proxy and ports section is used to create a 005-§§INSTANCE.conf proxy file.

NOTE:

• Only use tagged images versions (not latest). To ensure that the app stays stable over time.
• Use the docker-compose links parameter to ensure that containers can "talk" to each other indent of the name set in §§INSTANCE.

    version: '3'

    networks:
        bibbox-default-network:
          external: true

    services:

      §§INSTANCE-container-frontend:
        image: bibbox/app-image:vx.y.z
        container_name:  §§INSTANCE-container-frontend
        restart: unless-stopped
        networks:
          - bibbox-default-network
        links:
          - §§INSTANCE-container2:container2
        ports:
          - "80:80"
        depends_on:
          - §§INSTANCE-container2
        volumes: 
          - ./data/container/app-data:/app-data
        proxy:
          TYPE: PRIMARY
          URLPREFIX: §§INSTANCE
          TEMPLATE: default
          DISPLAYNAME: 'APP name'  

    §§INSTANCE-container2:
      image: bibbox/app-image2:va.b.c
      container_name:  §§INSTANCE-container2
      restart: unless-stopped
      networks:
        - bibbox-default-network
      links:
          - §§INSTANCE-app-db:app-db
      ports:
        - "8000:8000"
      depends_on:
          - §§INSTANCE-app-db
      volumes: 
        - ./data/container/app-data:/app-data

      §§INSTANCE-app-db:
        image: mysql:8
        container_name: §§INSTANCE-app-db
        restart: unless-stopped
        networks:
          - bibbox-default-network
        user: root
        environment:
          - MYSQL_ROOT_PASSWORD=thispasswordisneverusededoutsidethecontainer
          - MYSQL_DATABASE=§§MYSQL_DATABASE_NAME
          - MYSQL_USER=§§MYSQL_DATABASE_USER
          - MYSQL_PASSWORD=§§MYSQL_DATABASE_PASSWORD
        volumes: 
          - ./data/db/var/app-data:/var/app-data

environment-parameters.json

Environment parameters for the docker-compose-template file. The user can set the values for these parameters in a GUI during the installation of an App.

    [
       {
          "id":"NAME_AS_IN_DOCKER_TEMPLATE_1",
          "display_name":"Name Displayed in GUI at the Installation",
          "type":"text, number, password",
          "default_value":"Default Value",
          "description":"Description shown in the GUI",
          "min_length": "1",
          "max_length": "128"
       },
       {
          "id":"MYSQL_DATABASE_NAME",
          "display_name":"Name of the mysql database",
          "type":"text,",
          "default_value":"Default Value",
          "description":"Description shown in the GUI",
          "min_length": "1",
          "max_length": "128"
       },
       {
          "id":"MYSQL_DATABASE_USER",
          "display_name":"Name of the mysql user",
          "type":"text",
          "default_value":"Default Value",
          "description":"Description shown in the GUI",
          "min_length": "1",
          "max_length": "128"
       },
       {
          "id":"MYSQL_DATABASE_PASSWORD",
          "display_name":"Password of mySQL user",
          "type":"password",
          "default_value":"Default Value",
          "description":"Description shown in the GUI",
          "min_length": "1",
          "max_length": "128"
       }

    ]

fileinfo.json

A list of commands to be run when installing an app, consisting of three categories: "environmental_replace", "script_replace" and "permissions".

key	description
environmental_replace	WIP - specifies files/directories where placeholders can be replaced easily
script_replace	WIP - specifies files/directories where placeholders etc. should be replaced by a script
permissions	specifies which files/folders in the instance directory should be set to which permissions. \n the path is always relative to the instance directory

Tool for calculating permissions: chmod-calculator

structure:

    {
        "environment_replace": {},
        "script_replace": [],
        "permissions": {
            "PATH1": NUMERIC_VALUE_OF_PERMISSION1,
        "PATH2": NUMERIC_VALUE_OF_PERMISSION2,
        "PATH3": NUMERIC_VALUE_OF_PERMISSION3
        }
    }

example:

    {
        "environment_replace": {},
        "script_replace": [],
        "permissions": {
            "assets": 775,
        "data" : 777
        }
    }

icon.png

Icon of the App in the application store und in the application dashboard. Please use a square format, e.g. 512x512 pixeland PNG with a transparency channel.

Versioning

Each App should be versioned. We distinguish between

• Production Versions generated from a specific release in the repository. For each major release. i.e. given by a new version of dockerized software tool, a new branch should be generated in the repository. Please note, that the tag name for the Github release should contain the APP version and the BIBBOX APP release (adaption of the APP for BIBBOX). The proposed format is vAPP_verion_bibboxrel3 digit bibbox_release_number e.g. v8.7.2_bibboxrel001. The latest stable release is in the master branch.

• Development Version comming soon. Branches and tags not specified in a kit can be loaded in a Developer Mode.

When creating a production version, you should make the follwoing steps. Please make it exactly in this order, otherwise the tagging of the bibbox docker/hub and bibbox github will get confused.

1. Generate a new branch in github, if you plan to release a major version.

   1. Once a stable bibbox release is ready create a tag in compliance with the naming convention (e.g. v8.7.2_bibboxrel001).

   2. The latest app version should be merged in the master branch if a stable BIBBOX release exists via a pull request.

2. Update all the files for the anticipated version, don't forget to update appinfo.json.

3. Set in the docker compose template file (docker-compose.yml.template) the tags for the docker images. Please note, that ALL used images should be tagged with an specific version, don't use the 'latest" tag, as this could break your APP in the future.

4. Add the tag in the DOCKER hub to build tagged images (press SAVE before TRIGGER)

5. Edit in the used kit, e.g. bibbox.json the version information. This file can be found in the application-store repository.

Register an App

For the registration of an App two configuration files have to be extended:

• applications.json/ maps a Github repository to a human readable name of the App

• bibbox.json/ puts the App into a specific kit. A kit defines ta group of Apps together with metadata, which are then displayed in the App store and can be installed in a BIBBOX instance.