# in-house-gin

This repository documents how to set up a custom instance of the [GIN](https://gin.g-node.org) G-Node infrastructure 
service. The project's source code of can be found on [github](https://github.com/G-Node/gogs).

## Set up GIN in your lab
If you plan to set up a GIN Server in your own lab, you are more than welcome to do so. All our software is based on 
open source projects and therefore meant to be used, extended and modified by others. The features we implemented on 
top of [gogs](https://gogs.io/) are available on [Github](https://github.com/G-Node/gogs).

Should you have problems or questions in the process, contact us at [gin@g-node.org](mailto:gin@g-node.org). 
Even if you have no trouble, we would love to receive your feedback or just learn from you that you have just set up 
your own GIN! 

### Support
The in-house GIN has gained a lot of traction in the last year, which we are very 
happy to see and also happy to support as much as we can. With the growing number of
in-house GIN users, we also receive many requests for service set-up support and 
maintenance. Often these are simple matters, and we try to keep the documentation as
up-to date and helpful as possible, but since we are a small open-source team, the 
growing demand for support takes up quite some of our time.

Since we are providing this service for free, we would like to ask you in to support 
your fellow in-house users and help each other out running the service.
To this end we are running a dedicated slack workspace at in-house-gin.slack.com. 
Slack can be used either via the browser or by a dedicated slack app.

To join this workspace, we would ask you to send an email to gin@g-node.org with the 
email address you would like to be invited with. Please note, that this email address 
will be available to all other members in the channel, use an email address you are 
comfortable sharing.

This is also the place to discuss new features or companion services that you are employing 
or would like to try in tandem with GIN. The G-Node team is also hanging around the 
channel to discuss and help out as much as we can.

You can also fork the in-house repository and supply pull requests to update
the in-house documentation with information you would like to see included, open issues
on the in-house issue tracker or post them in the slack channels.

#### Setup prerequisites and preparations

`docker` and `docker-compose` are required on the system. If the gin instance should be available within the network 
or to the outside, we recommend using the GIN instance together with an Apache2 server.
Please see the corresponding installation notes for [docker](https://docs.docker.com/get-docker/) and 
[docker-compose](https://docs.docker.com/compose/install/) for details. Furthermore, a 
[dockerhub account](https://hub.docker.com/) is useful.

As a general note, the documentation is based on OS Debian and relies on docker version >= `20.10.x` 
and docker-compose version >= `1.26.x`. Using a different setup might require additional or differing steps.


## Test deploying GIN using docker on your local machine

If you are new to docker and docker-compose or deploying services with it, it might be good to test the deployment
with a simplified setup using `docker` on your local machine. This deployment is meant as a means to explore the web
frontend of the service.

For a description on how to deploy a full server including a persistent postgres database and customized content,
please see the section "Deploy and run full server" below.

If you are not familiar with docker or docker-compose, please get familiar with it. These setup notes try to be very
thorough, but differences in host machine setup can lead to issues during the service setup process. Decent knowledge
with docker and docker-compose are usually enough to resolve these issues on your own.

#### Set up GIN with Docker
The easiest way to set up a GIN web frontend server is via pre-built docker containers. You will need a working Docker
installation. 

You start by downloading the docker image for GIN by typing:
```bash
docker pull gnode/gin-web:latest
```

You can start GIN with:
```bash
docker run -p 3000:3000 -p 2222:22 -d gnode/gin-web
```
You can now visit the website of your GIN server at port 3000 (see the first -p above) of your local machine (localhost:3000). 
The ssh port can be reached via port 2222 (see the second -p above). You will be greeted with a setup page which will 
ask you for all the necessary information.

To make the start as easy as possible, select 'SQLite3' as database. This will setup a lightweight database within the
docker container for testing purposes. In production use a postgres database as described in the section "Deploy and 
run full server" below.
All other settings can be left with their default value to initialize a test GIN server. Note that if an admin account 
is not specified on the "First-time Run" page, the first account registered on the running service will become a 
service admin. Also note that username "admin" is reserved and cannot be used.

This will map all data and configuration into the docker container, which is fine for playing around. However, should 
you want to use the server for data, it's probably not what you want. Instead, repository data as well as the 
GIN service config and log files can be mapped to directories of the local host:
```bash
docker run -v /somefolder/:/data -p 3000:3000 -p 2222:22 -d gnode/gin-web
```

This will map the data, configs and such into a directory on the host machine (change "somefolder" accordingly).

If you want the container to run the web service with another user and group id than 1000, you can do this by setting 
the corresponding environment variables on docker by running:
```bash
docker run -v /somefolder/:/data -p 3000:3000 -p 2222:22 -e GITUID="1001" -e GITGID="1001" -d gnode/gin-web
```

where `GITUID` and `GITGID` are followed by the numerical user and group ids respectively.

#### Update an existing container
To update an existing GIN docker installation you simply need to pull in the changes:
```bash
docker pull gnode/gin-web:latest
```

afterward, you stop the running instance with:
```bash
docker stop <containername>
```

`<containername>` can be determined with:

```bash
docker ps
```

After that you can start a new version using the same command that you started the original docker container with, e.g.:

```bash
docker run -v /somefolder/:/data -p 3000:3000 -p 2222:22 -d gnode/gin-web
```

#### Using git with a local docker test setup

- add an ssh public key on the running GIN server
- clone repositories when testing on localhost using

  ```bash
  git clone ssh://git@localhost:2222/[ginuser]/[ginrepository].git
  ```


## Deploy and run a full server using docker-compose

The setup for a full server requires `docker-compose` and is more complex than the simple, local `docker` setup.
But it comes with notes on persistent storage, server customization and notes on how to configure the GIN client
to work with your own instance of GIN.
If you are considering this option, it might still be worth to try setting up the local version above as described 
above to familiarize yourself with the basic GIN setup before diving into the more complex one.

This setup will require the files found in the [in-house-gin](https://gin.g-node.org/G-Node/in-house-gin) repository. 
It also assumes, that the service will run behind an http server like [Apache](https://httpd.apache.org/). If you still 
want to test the docker-compose setup on a local machine, you will need to make adjustments to the docker-compose.yml 
file e.g. mapping port 3000 to make it available.

All the necessary steps should be described and available below. If you still find steps missing or the description 
ambiguous, please let us know at [gin@g-node.org](mailto:gin@g-node.org).

#### Required group and user
To smoothly work with a docker deployment, the following groups and users should be set up.
- prepare a dedicated deployment group e.g., "ginservice" group if it does not exist.
- create a dedicated deployment user e.g., "ginuser".
- for any further setup, all required directories and files should be created with this user/group as owner.
- make sure to add all users that are running docker-compose or need access to project files to the docker and 
  deploy groups.

#### Required directories

The current setup requires certain directories to be available before the containers have started. These directories 
will contain and persist
- the postgres database
- all repositories uploaded to the GIN instance
- configuration files
- custom frontend files

This schema notes all required directories and files:
```
$GIN_ROOT_FOLDER
├── config
|   ├── postgres
|   |   └── pgressecrets.env
|   └── gogs
|       ├── notice
|       |   └── banner.md   # GIN page notice banner
|       ├── public          # custom frontend style
|       └── templates       # custom frontend files
├── volumes
|   └── ginweb
├── gindata
|   ├── gin-postgresdb
|   └── gin-repositories
└── gin-dockerfile
    ├── .env
    └── docker-compose.yml
```

The following bash script lines are an example how to set up these required directories.

```bash
# Provide the user that will use docker-compose to start the service
DEPLOY_USER=[provide username]

# Specify the GIN files root location
DIR_GINROOT=[provide path to GIN root directory]

# Postgres database password
PGRES_PASS=[changeMe]

# Prepare a ginservice group if it does not exist
if [ $(getent group ginservice) ]; then
  echo "group ginservice already exists."
else
  groupadd ginservice
fi

# Create dedicated "ginuser" user and add it to the "docker"
# group; and the "ginservice" group; create all required 
# directories with this user and the "ginservice" group permission.
if [ $(getent passwd ginuser) ]; then
    echo "user ginuser already exists"
else
    useradd -M -G docker,ginservice ginuser
    # Disable login for user ginuser
    usermod -L ginuser
fi

# Make sure to add the user running docker-compose to the ginservice group as well!
usermod -a -G ginservice $DEPLOY_USER

# Required directories
DIR_GINCONFIG=$DIR_GINROOT/config
DIR_GINVOLUMES=$DIR_GINROOT/volumes
DIR_GINDATA=$DIR_GINROOT/gindata

# Create gin specific directories
# The 'notice' directory may contain a banner.md file.
# The content of this file is displayed on the GIN page without a service restart e.g.
#  to inform users of an upcoming service downtime.
mkdir -vp $DIR_GINCONFIG/gogs/notice
mkdir -vp $DIR_GINCONFIG/gogs/conf
mkdir -vp $DIR_GINCONFIG/postgres

mkdir -vp $DIR_GINVOLUMES/ginweb

mkdir -vp $DIR_GINDATA/gin-repositories
mkdir -vp $DIR_GINDATA/gin-postgresdb

mkdir -vp $DIR_GINROOT/gin-dockerfile

# Create an env file to specify the docker compose project name
echo "COMPOSE_PROJECT_NAME=gin" > $DIR_GINROOT/gin-dockerfile/.env

# Create postgres secret file
echo "POSTGRES_PASSWORD=${PGRES_PASS}" > $DIR_GINCONFIG/postgres/pgressecrets.env

# make sure an empty gogs config file is available; technically only required for previous versions of gogs
touch $DIR_GINCONFIG/gogs/conf/app.ini
```

If the user running the docker containers is not the `ginuser`, it might be worth adding this user to both the `docker` 
and `ginservice` group as well. Otherwise, the services might run into permission issues.

#### Prepare docker-compose file

The `resources` folder contains an example docker-compose file to set up an in-house GIN instance.

Prepare the `docker-compose.yml` file in the `$DIR_GINROOT/gin-dockerfile` directory. The example docker-compose 
file assumes the directory structure created above to properly map all required volumes.

Update the docker-compose file to match your requirements:
  - adjust `web:environment`: use the IDs of the prepared `ginuser` and `ginservice` group
  - make sure the ssh port is set correctly: use e.g. port 2121:22 since server port 22 needs to be available 
    for normal ssh connections
  - make sure the set IP matches the IP set in your apache2/webserver configuration
  - ensure the following paths are set correctly to match the created directory structure
    - `web:volumes: ../gindata/..`
    - `db:volumes: ../gindata/..`
  - if you plan to run the docker-compose setup on localhost you also need to map the port 3000 in the docker-compose 
    file; if you use an http server like an apache do not expose the port via the docker-compose file but include it 
    in the apache configuration.

#### Adjust file permissions

Change all file permissions of the GIN root directory to ginservice group to ensure the docker services 
have access to the directories and change group permissions to write.

```bash
chown -R ginuser:ginservice $DIR_GINROOT
chmod -R g+rw $DIR_GINROOT
```

#### Docker container and initial GIN setup

You can use the `gnode/gin-web:latest` docker container to run the in-house GIN; this is set as default in the example 
`docker-compose.yml` file but can be changed to another container.

- fetch all required containers from the docker-gin directory; note that this will pull the appropriate docker
  containers for the GIN web service and the postgres database.
  ```bash
  cd $DIR_GINROOT/gin-dockerfile
  docker-compose pull
  ```

- launch the postgres database container for the initial gin setup
  ```bash
  docker-compose up -d db
  docker exec -it gin_db_1 /bin/bash
  su -l postgres
  createuser -P gin
  # enter a password for the new role and note it; it will later be used on the initial gin setup page
  createdb -O gin gin
  exit
  exit
  ```

- launch the gin web docker container for the inital setup; on a local machine it should be accessible in the webbrowser 
  at localhost:3000; if it is set up within a network or via a domain name, it will be available there.

  ```
  docker-compose up -d
  ```

- on the setup page, set the following fields with the corresponding values. For other setup options please refer 
  to the gogs documentation.
    - db:               postgres
    - host:             ginpgres:5432
    - user:             gin
    - password:         [used during database setup]
    - database name:    gin
    - app name:         GIN dev
    - repo root:        as defined in `docker-compose` on the container side e.g. `/data/repos`
    - domain:           your domain or localhost
    - create an administration user; do not use 'admin'; if the database has been restored from a live server, 
      this user will be overwritten.
- save; this might redirect to an error page, but this is inconsequential
- check that the server is running at your specified domain and you can login with the created admin user

- stop all docker containers
  ```bash
  cd $DIR_GINROOT/gin-dockerfile
  docker-compose down
  ```

- replace `config/gogs/conf/app.ini` with the file found in `resources/app.ini`; when in doubt keep the old file and 
  compare entries.
  - adjust all settings to fit your server requirements - the example `app.ini` should fit most settings and 
    directories used so far.
    - in the [server] section check `DOMAIN`, `HTTP_PORT` and `ROOT_URL`
    - in the [mail] section enable or disable a mail server. Depending on the state of the mailserver, 
      you might want to activate or deactivate registration confirmation by mail.
    - in the [security] section, copy the SECRET_KEY value from the automatically create app.ini to this one.

- copy the `public` and `templates` directories of this repository to `$DIR_GINROOT/config/gogs`; this directory 
  is specified via the `docker-compose.yml` file to hold custom frontend templates for the GIN instance. When the 
  container is started, templates and stylesheets from these directories will overrule same-name files on the server.
  - `home.tmpl` is the main page non-logged in users will see.
  - `base/head_gin.tmpl` contains custom links in the page header
  - `base/footer_gin*.tmpl` templates contain custom entries in the page.
  - `explore/navbar.tmpl` contains categories on the `explore` page.
  - for additional custom content changes please see the "Customize the in-house GIN content" section below
  - note that every time the content of the `public` or `tempaltes` folder have been changed, the docker container
    needs to be restarted before the changes take effect.

- adjust file permissions for all added and modified files on the server
  ```bash
  chown -R ginuser:ginservice $DIR_GINCONFIG
  ```


- stop any running container and restart the full service
  ```bash
  docker-compose down
  docker-compose up -d
  docker-compose logs -f
  ```

NOTE: when working with the GIN server (gin.g-node.org) in parallel with a local or in-house version of GIN, it can happen, 
that the services have set up different web browser CSRF tokens. This results in the message `Invalid csrf token.` when submitting 
any webform data via the web browser. In this case either purge the browser cache or use a different 
browser all-together.

#### Customize the in-house GIN content

Apart from changing the main page, as well as header and footer content as described above, most custom pages are hosted 
via the Wiki of a specific GIN repository wiki: `Service/Info`. By default the service links and expects the following
pages in this particular wiki: `Help`, `News`, `about`, `imprint`, `contact`, `terms of use` and `Datenschutz`.

To create and edit these on your GIN instance

- log in with an administrative user
- create a `Service` organisation
- create a `Service/Info` repository
- set this repo to public
- open this repos' wiki page and initialise the wiki by creating a "home.md" page.
- all files required by default can be added via the wiki of this repository at the address "http://[domain]/Service/Info/wiki"

    - "Help" refers to the page "http://[domain]/Service/Info/wiki/Home.md"
    - "News" refers to the page "http://[domain]/Service/Info/wiki/News.md"
    - "About" refers to the page "http://[domain]/Service/Info/wiki/about.md"
    - "Imprint" refers to the page "http://[domain]/Service/Info/wiki/imprint.md"
    - "Contact" refers to the page "http://[domain]/Service/Info/wiki/contact.md"
    - "Terms of use" refers to the page "http://[domain]/Service/Info/wiki/Terms of use.md"
    - "Datenschutz" refers to the page "http://[domain]/Service/Info/wiki/Datenschutz.md"

- alternatively to adding these pages via the web interface, you can add an ssh public key to your user and git clone 
  the wiki repository after it has been initialized:
  `git clone ssh://git@[your domain]:2121/Service/Info.wiki.git`

- further information about server frontend customization can be found in the official gogs/gogs documentation about [custom templates](https://gogs.io/docs/features/custom_template).

#### GIN client setup

By default the [GIN-client](https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup), a command line tool to work with GIN, 
is set up to work with the GIN server at gin.g-node.org. To use the GIN-Client with an in-house version of gin a couple of 
additional steps have to be taken. On the machine where a gin client is installed, additional entries can be added to 
the list of supported servers.

Please note, that you will need to add the port number that is specified as the ssh port in the `docker-compose.yml` file. 
  In the example this has been set to 2121. The name "alias-in-house" can of course be chosen freely.
```bash
gin logout
gin add-server --web https://gin.dev.g-node.org:443 --git git@gin.dev.g-node.org:2121 alias-in-house
gin use-server alias-in-house
gin login
```

#### Avoid exposing debug service

Macaron toolbox, which runs alongside GOGS, exposes a minimal debug control panel at `/debug` (https://github.com/gogs/gogs/issues/5578).  
This control panel can be used to run profiling operations and dump information about the running server.
Currently, there's no way to restrict this in code. For now, adding the following Apache configuration will deny 
all access to the console:
```
<Location /debug>
    Deny from  all
</Location>
```

## Build your own docker container from source

If the customizations above are not sufficient for your service, you can also clone our [GIN github repository](
https://github.com/G-Node/gogs), directly edit the source code and build your own docker container. Run it from the 
root of the repository:
```bash
docker build -t [organization-name]/[container-name] .
```

## Ansible galaxy in-house-gin

Mika Pflüger was so kind to create an Ansible role for in-house-gin. If you want to give it a try, you can find the 
details over at the [ansible galaxy page](https://galaxy.ansible.com/mikapfl/gin). We take neither 
credit nor responsibility for this project.

## Further reading

Since GIN is based on GOGS, the official documentation for running GOGS in a container is also relevant:
[Docker for GOGS](https://github.com/gogs/gogs/tree/main/docker)