Compare commits

...

11 Commits

21 changed files with 842 additions and 2635 deletions

157
paperless_ngx/README.md Normal file
View File

@@ -0,0 +1,157 @@
# Spin it up
## Defaults
We use the [upstream github.com/paperless-ngx/paperless-ngx](https://github.com/paperless-ngx/paperless-ngx) repo assuming you have this checked out at `/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev`. The `COMPOSE_CTX` (`CTX` as in context) is a unique identifier to differentiate one instance from another. This can be for example `hr_juba` to indicate that this particular instance runs on behalf of the Human Resources department based in Juba in South Sudan in Africa.
Examples assume we're using the `docker-compose.postgres-tika.yml` file which spins up PostgreSQL as its database and also provides [Apache Tika](https://tika.apache.org/) and [Gotenberg PDF API](https://gotenberg.dev/docs/about).
## Environment variables
* Set env vars
```
export UPSTREAM_REPO_DIR='/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev'
export UPSTREAM_COMPOSE_DIR="${UPSTREAM_REPO_DIR%/}"'/docker/compose'
export UPSTREAM_COMPOSE_FILE="${UPSTREAM_COMPOSE_DIR%/}"'/docker-compose.postgres-tika.yml'
export COMPOSE_CTX='hr_juba'
export COMPOSE_PROJECT_NAME='paperless_ngx-'"${COMPOSE_CTX}"
export COMPOSE_PROJECT_DIR='/opt/containers/paperless_ngx'
export COMPOSE_OVERRIDE_FILE="${COMPOSE_PROJECT_DIR%/}"'/docker-compose.override.yml'
export COMPOSE_ENV_FILE=<set accordingly>
```
Also check out the example env file at [env/fully.qualified.domain.name_ctx.env.example](env/fully.qualified.domain.name_ctx.env.example).
* Build
```
docker compose --project-directory "${COMPOSE_PROJECT_DIR}" --file "${UPSTREAM_COMPOSE_FILE}" --file "${COMPOSE_OVERRIDE_FILE}" --env-file "${COMPOSE_ENV_FILE}" --profile 'build' build
```
We're building a custom PostgreSQL image with `ping` installed. We're using that to do a health check on our virtual IP (VIP) address. We've added a dependency to the main `paperless-ngx` web server container so that it only starts after the PostgreSQL container with its health check confirms that the VIP is reachable.
* Start containers
```
docker compose --project-name "${COMPOSE_PROJECT_NAME}" --file "${UPSTREAM_COMPOSE_FILE}" --env-file "${COMPOSE_ENV_FILE}" up --detach
```
# Prep work
## Data sets
To get started from scratch create your ZFS datasets and set permissions as needed for `paperless-ngx`.
* Parent dateset
```
zfs create -o mountpoint=/opt/docker-data 'zpool/docker-data'
```
* Container-specific datasets
```
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/broker/data'
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/db/data'
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/webserver/data'
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/webserver/media'
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/webserver/export'
zfs create -p 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/webserver/consume'
```
* Change ownership for all webserver-related dirs
```
chown -R 1000:1000 'zpool/docker-data/paperless_ngx-'"${COMPOSE_CTX}"'/webserver'
```
## Apply patch
Identify yourself to the local `paperless-ngx` repo. Obviously substitute your own name. An e-mail address is optional here. You don't want to contribute upstream, you just want to locally apply a patch file.
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' config user.name "hygienic-books"
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' config user.email ""
```
Apply `paperless_ngx.patch` to Docker Compose file. We use the `docker-compose.postgres-tika.yml` Compose file. Assuming this repo lives at `/opt/containers/paperless_ngx`:
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' am '/opt/containers/paperless_ngx/paperless_ngx.patch'
# Output will be:
Applying: refactor(compose): 4 spaces indentation
Applying: refactor(compose): Harmonize restart and logging settigs
Applying: refactor(compose): Replace static exposed port with environment variable
Applying: refactor(compose): Harmonize container names
Applying: refactor(compose): Replace named volumes with bind mounts
...
```
And then back up to [Environment variables](#environment-variables).
# Upgrade an existing repo
Check [Prep work](#prep-work) for first time steps. On consecutive upgrades proceed as follows.
## Revert unpushed local changes
Return repo state to exactly the upstream repo's original branch state throwing away the commits you added.
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' reset --hard origin
```
Switch to `main` branch, get newest commits from upstream
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' checkout dev
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' pull
```
Pick and checkout new tag
```
while IFS= read -r; do commitDate=$(grep -Pio '^.+?(?=[[:space:]])' <<< "${REPLY}"); commitDate=$(date --date='@'"${commitDate}" +%F-%H%M%S); tagRef="$(cut -d $'\t' -f2 <<< "${REPLY}")"; tagName="$(grep -Pio '(?<=refs/tags/)[^\r\n\f]+' <<<"${tagRef}")"; commitHash="$(git rev-list -n 1 "${tagRef}")"; echo "${commitDate} ${commitHash} ${tagName}"; done < <(git for-each-ref --sort=v:refname --format='%(*creatordate:raw)%00%(creatordate:raw)%00%(refname)' refs/tags | awk -F"\0" 'BEGIN {ORS=""} $1 == "" {print $2} $1 != "" {print $1} {print "\t"$3"\n"}')
# Output goes like:
...
2023-04-27-161244 864e242ed9c454585e236b0c20ccae0927b4c9b2 v1.14.1
2023-04-27-195703 356c26ce848ca5301156a33e9ea75a10255f404b v1.14.2
2023-05-03-155437 4353646b3ac5805f7c582599b784a2fc246b3700 v1.14.3
2023-05-04-164855 ec4814a76e88efa81387316d8c42afc7e220fcbe v1.14.4
2023-05-15-170859 3e129763c799a7141e5ecd04862c0160caeeef5b v1.14.5
...
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' checkout 'tags/vx.y.z'
```
Lastly [apply patch](#apply-patch). If patch does not apply cleanly read on in the next section [Create new patch](#create-new-patch) to find out how to fix your patch.
# Create new patch
## Add your changes as commits
With `paperless-ngx` repo checked out at `/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev` get the it into a state with which you're happy then
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' format-patch 31b7e26f6c4d7111f4f4957996efb9f7a5d06cb9^..64beae08ffe8f9a65208e2567919fd75564b95c6 --stdout > '/opt/containers/paperless_ngx/paperless_ngx.patch'
```
Where the first commit hash is our first commit and the other commit hash is our last commit. Note the caret (`^`) right after the first commit hash.
## Investigation
You may have to try and find out how a known good base commit differs from a newer one in case the newer one does no longer cleanly accept the patch.
Get commit hashes from both affected tags, e.g.
```
while IFS= read -r; do commitDate=$(grep -Pio '^.+?(?=[[:space:]])' <<< "${REPLY}"); commitDate=$(date --date='@'"${commitDate}" +%F-%H%M%S); tagRef="$(cut -d $'\t' -f2 <<< "${REPLY}")"; tagName="$(grep -Pio '(?<=refs/tags/)[^\r\n\f]+' <<<"${tagRef}")"; commitHash="$(git rev-list -n 1 "${tagRef}")"; echo "${commitDate} ${commitHash} ${tagName}"; done < <(git for-each-ref --sort=v:refname --format='%(*creatordate:raw)%00%(creatordate:raw)%00%(refname)' refs/tags | awk -F"\0" 'BEGIN {ORS=""} $1 == "" {print $2} $1 != "" {print $1} {print "\t"$3"\n"}')
# Output goes like:
...
2023-04-27-161244 864e242ed9c454585e236b0c20ccae0927b4c9b2 v1.14.1
2023-04-27-195703 356c26ce848ca5301156a33e9ea75a10255f404b v1.14.2
2023-05-03-155437 4353646b3ac5805f7c582599b784a2fc246b3700 v1.14.3
2023-05-04-164855 ec4814a76e88efa81387316d8c42afc7e220fcbe v1.14.4
2023-05-15-170859 3e129763c799a7141e5ecd04862c0160caeeef5b v1.14.5
...
```
Diff them
```
git -C '/opt/git/github.com/paperless-ngx/paperless-ngx/branches/dev' diff ec4814a76e88efa81387316d8c42afc7e220fcbe 3e129763c799a7141e5ecd04862c0160caeeef5b 'docker/compose/docker-compose.postgres-tika.yml'
```
Output will be empty in case no difference exists in `docker/compose/docker-compose.postgres-tika.yml` between both commit hashes.
Commit your updated patch file into _this_ repo. With a new working patch in hand head back up to [Upgrade an existing repo](#upgrade-an-existing-repo).

View File

@@ -0,0 +1,6 @@
ARG PGSQL_VERSION
FROM "docker.io/library/postgres:${PGSQL_VERSION}"
RUN apt-get update && \
apt-get -y install iputils-ping && \
rm -rf /var/lib/apt/lists/*

View File

@@ -1,11 +1,11 @@
services: services:
common-settings: common-settings:
environment:
TZ: "${TIMEZONE:-Etc/UTC}"
logging: logging:
driver: "json-file" driver: "json-file"
options: options:
max-size: "10m" max-size: "10m"
max-file: "10" max-file: "10"
compress: "true" compress: "true"
environment: restart: "${RESTARTPOLICY:-unless-stopped}"
TZ: Europe/Berlin
restart: unless-stopped

View File

@@ -0,0 +1,9 @@
services:
paperless_ngx-build:
image: "docker.io/library/postgres:${PGSQL_VERSION}"
profiles: ["build"]
build:
context: "build-context"
dockerfile: Dockerfile
args:
PGSQL_VERSION: "${PGSQL_VERSION}"

3
paperless_ngx/env/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
*
!.gitignore
!fully.qualified.domain.name_ctx.env.example

View File

@@ -0,0 +1,21 @@
# This file is maintained by remco and populated with data from HashiCorp
# Vault. Changes not done in Vault will be reverted when file gets rendered.
CONTEXT=hr_juba
PAPERLESS_OCR_CLEAN=clean-final
PAPERLESS_OCR_LANGUAGE=eng
PAPERLESS_OCR_LANGUAGES=bos
PAPERLESS_SECRET_KEY=face-roll-your-keyboard-here
PAPERLESS_TIME_ZONE=Africa/Juba
PAPERLESS_URL=https://fully.qualified.domain.name
PGSQL_VERSION=13
SUBNET=172.22.17.0/24
TIMEZONE=Africa/Juba
WEBSERVER_PORT=8080
WEBSERVER_VIP=192.168.29.103
# Other available defaults
# USERMAP_UID=1000
# USERMAP_GID=1000
# PAPERLESS_FORCE_SCRIPT_NAME=/PATHPREFIX
# PAPERLESS_STATIC_URL=/PATHPREFIX/static/ # trailing slash required

View File

@@ -0,0 +1,641 @@
From 42c196c2c0dcbed0101d653e0ed67a3dfc3f1808 Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:17:21 +0200
Subject: [PATCH 01/12] refactor(compose): 4 spaces indentation
---
.../compose/docker-compose.postgres-tika.yml | 114 +++++++++---------
1 file changed, 57 insertions(+), 57 deletions(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index a451b00d..7423be72 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -32,67 +32,67 @@
version: "3.4"
services:
- broker:
- image: docker.io/library/redis:7
- restart: unless-stopped
- volumes:
- - redisdata:/data
+ broker:
+ image: docker.io/library/redis:7
+ restart: unless-stopped
+ volumes:
+ - redisdata:/data
- db:
- image: docker.io/library/postgres:13
- restart: unless-stopped
- volumes:
- - pgdata:/var/lib/postgresql/data
- environment:
- POSTGRES_DB: paperless
- POSTGRES_USER: paperless
- POSTGRES_PASSWORD: paperless
+ db:
+ image: docker.io/library/postgres:13
+ restart: unless-stopped
+ volumes:
+ - pgdata:/var/lib/postgresql/data
+ environment:
+ POSTGRES_DB: paperless
+ POSTGRES_USER: paperless
+ POSTGRES_PASSWORD: paperless
- webserver:
- image: ghcr.io/paperless-ngx/paperless-ngx:latest
- restart: unless-stopped
- depends_on:
- - db
- - broker
- - gotenberg
- - tika
- ports:
- - "8000:8000"
- healthcheck:
- test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
- interval: 30s
- timeout: 10s
- retries: 5
- volumes:
- - data:/usr/src/paperless/data
- - media:/usr/src/paperless/media
- - ./export:/usr/src/paperless/export
- - ./consume:/usr/src/paperless/consume
- env_file: docker-compose.env
- environment:
- PAPERLESS_REDIS: redis://broker:6379
- PAPERLESS_DBHOST: db
- PAPERLESS_TIKA_ENABLED: 1
- PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
- PAPERLESS_TIKA_ENDPOINT: http://tika:9998
+ webserver:
+ image: ghcr.io/paperless-ngx/paperless-ngx:latest
+ restart: unless-stopped
+ depends_on:
+ - db
+ - broker
+ - gotenberg
+ - tika
+ ports:
+ - "8000:8000"
+ healthcheck:
+ test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ volumes:
+ - data:/usr/src/paperless/data
+ - media:/usr/src/paperless/media
+ - ./export:/usr/src/paperless/export
+ - ./consume:/usr/src/paperless/consume
+ env_file: docker-compose.env
+ environment:
+ PAPERLESS_REDIS: redis://broker:6379
+ PAPERLESS_DBHOST: db
+ PAPERLESS_TIKA_ENABLED: 1
+ PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
+ PAPERLESS_TIKA_ENDPOINT: http://tika:9998
- gotenberg:
- image: docker.io/gotenberg/gotenberg:7.8
- restart: unless-stopped
+ gotenberg:
+ image: docker.io/gotenberg/gotenberg:7.8
+ restart: unless-stopped
- # The gotenberg chromium route is used to convert .eml files. We do not
- # want to allow external content like tracking pixels or even javascript.
- command:
- - "gotenberg"
- - "--chromium-disable-javascript=true"
- - "--chromium-allow-list=file:///tmp/.*"
+ # The gotenberg chromium route is used to convert .eml files. We do not
+ # want to allow external content like tracking pixels or even javascript.
+ command:
+ - "gotenberg"
+ - "--chromium-disable-javascript=true"
+ - "--chromium-allow-list=file:///tmp/.*"
- tika:
- image: ghcr.io/paperless-ngx/tika:latest
- restart: unless-stopped
+ tika:
+ image: ghcr.io/paperless-ngx/tika:latest
+ restart: unless-stopped
volumes:
- data:
- media:
- pgdata:
- redisdata:
+ data:
+ media:
+ pgdata:
+ redisdata:
--
2.39.1
From 6bf9b93ebe48036d67d94a49162c1f6c17eb80de Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:22:06 +0200
Subject: [PATCH 02/12] refactor(compose): Harmonize restart and logging
settigs
---
docker/compose/common-settings.yml | 11 ++++++++++
.../compose/docker-compose.postgres-tika.yml | 20 ++++++++++++++-----
2 files changed, 26 insertions(+), 5 deletions(-)
create mode 100644 docker/compose/common-settings.yml
diff --git a/docker/compose/common-settings.yml b/docker/compose/common-settings.yml
new file mode 100644
index 00000000..9fd26d75
--- /dev/null
+++ b/docker/compose/common-settings.yml
@@ -0,0 +1,11 @@
+services:
+ common-settings:
+ environment:
+ TZ: "${TIMEZONE:-Etc/UTC}"
+ logging:
+ driver: "json-file"
+ options:
+ max-size: "10m"
+ max-file: "10"
+ compress: "true"
+ restart: "${RESTARTPOLICY:-unless-stopped}"
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 7423be72..d19c22d7 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -34,13 +34,17 @@ version: "3.4"
services:
broker:
image: docker.io/library/redis:7
- restart: unless-stopped
+ extends:
+ file: common-settings.yml
+ service: common-settings
volumes:
- redisdata:/data
db:
image: docker.io/library/postgres:13
- restart: unless-stopped
+ extends:
+ file: common-settings.yml
+ service: common-settings
volumes:
- pgdata:/var/lib/postgresql/data
environment:
@@ -50,7 +54,9 @@ services:
webserver:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
- restart: unless-stopped
+ extends:
+ file: common-settings.yml
+ service: common-settings
depends_on:
- db
- broker
@@ -78,7 +84,9 @@ services:
gotenberg:
image: docker.io/gotenberg/gotenberg:7.8
- restart: unless-stopped
+ extends:
+ file: common-settings.yml
+ service: common-settings
# The gotenberg chromium route is used to convert .eml files. We do not
# want to allow external content like tracking pixels or even javascript.
@@ -89,7 +97,9 @@ services:
tika:
image: ghcr.io/paperless-ngx/tika:latest
- restart: unless-stopped
+ extends:
+ file: common-settings.yml
+ service: common-settings
volumes:
data:
--
2.39.1
From 9b7dd19f5c6f410594f35dd233d1d32c9d118dbf Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:28:02 +0200
Subject: [PATCH 03/12] refactor(compose): Replace static exposed port with
environment variable
---
docker/compose/docker-compose.postgres-tika.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index d19c22d7..554d9735 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -63,7 +63,7 @@ services:
- gotenberg
- tika
ports:
- - "8000:8000"
+ - "${WEBSERVER_PORT}":8000
healthcheck:
test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
interval: 30s
--
2.39.1
From aee7de3b1adfb30e7f0f2737d567cfc8190d8302 Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:30:47 +0200
Subject: [PATCH 04/12] refactor(compose): Harmonize container names
---
docker/compose/docker-compose.postgres-tika.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 554d9735..10316fc2 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -34,6 +34,7 @@ version: "3.4"
services:
broker:
image: docker.io/library/redis:7
+ container_name: "paperless_ngx-broker-${CONTEXT}"
extends:
file: common-settings.yml
service: common-settings
@@ -42,6 +43,7 @@ services:
db:
image: docker.io/library/postgres:13
+ container_name: "paperless_ngx-db-${CONTEXT}"
extends:
file: common-settings.yml
service: common-settings
@@ -54,6 +56,7 @@ services:
webserver:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
+ container_name: "paperless_ngx-webserver-${CONTEXT}"
extends:
file: common-settings.yml
service: common-settings
@@ -84,6 +87,7 @@ services:
gotenberg:
image: docker.io/gotenberg/gotenberg:7.8
+ container_name: "paperless_ngx-gotenberg-${CONTEXT}"
extends:
file: common-settings.yml
service: common-settings
@@ -97,6 +101,7 @@ services:
tika:
image: ghcr.io/paperless-ngx/tika:latest
+ container_name: "paperless_ngx-tika-${CONTEXT}"
extends:
file: common-settings.yml
service: common-settings
--
2.39.1
From e600e84d5c6e6c8938498f8cfa8bef32c1aea180 Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:36:29 +0200
Subject: [PATCH 05/12] refactor(compose): Replace named volumes with bind
mounts
---
.../compose/docker-compose.postgres-tika.yml | 18 ++++++------------
1 file changed, 6 insertions(+), 12 deletions(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 10316fc2..eacb76c8 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -39,7 +39,7 @@ services:
file: common-settings.yml
service: common-settings
volumes:
- - redisdata:/data
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/broker/data:/data
db:
image: docker.io/library/postgres:13
@@ -48,7 +48,7 @@ services:
file: common-settings.yml
service: common-settings
volumes:
- - pgdata:/var/lib/postgresql/data
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/db/data:/var/lib/postgresql/data
environment:
POSTGRES_DB: paperless
POSTGRES_USER: paperless
@@ -73,10 +73,10 @@ services:
timeout: 10s
retries: 5
volumes:
- - data:/usr/src/paperless/data
- - media:/usr/src/paperless/media
- - ./export:/usr/src/paperless/export
- - ./consume:/usr/src/paperless/consume
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/data:/usr/src/paperless/data
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/media:/usr/src/paperless/media
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/export:/usr/src/paperless/export
+ - /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/consume:/usr/src/paperless/consume
env_file: docker-compose.env
environment:
PAPERLESS_REDIS: redis://broker:6379
@@ -105,9 +105,3 @@ services:
extends:
file: common-settings.yml
service: common-settings
-
-volumes:
- data:
- media:
- pgdata:
- redisdata:
--
2.39.1
From a1ff3cdfd84a45e0a383168470a5028246577a7d Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:37:44 +0200
Subject: [PATCH 06/12] refactor(compose): Introduce dedicated network
---
.../compose/docker-compose.postgres-tika.yml | 21 +++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index eacb76c8..05885149 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -35,6 +35,8 @@ services:
broker:
image: docker.io/library/redis:7
container_name: "paperless_ngx-broker-${CONTEXT}"
+ networks:
+ paperless_ngx-default:
extends:
file: common-settings.yml
service: common-settings
@@ -44,6 +46,8 @@ services:
db:
image: docker.io/library/postgres:13
container_name: "paperless_ngx-db-${CONTEXT}"
+ networks:
+ paperless_ngx-default:
extends:
file: common-settings.yml
service: common-settings
@@ -57,6 +61,8 @@ services:
webserver:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
container_name: "paperless_ngx-webserver-${CONTEXT}"
+ networks:
+ paperless_ngx-default:
extends:
file: common-settings.yml
service: common-settings
@@ -88,6 +94,8 @@ services:
gotenberg:
image: docker.io/gotenberg/gotenberg:7.8
container_name: "paperless_ngx-gotenberg-${CONTEXT}"
+ networks:
+ paperless_ngx-default:
extends:
file: common-settings.yml
service: common-settings
@@ -102,6 +110,19 @@ services:
tika:
image: ghcr.io/paperless-ngx/tika:latest
container_name: "paperless_ngx-tika-${CONTEXT}"
+ networks:
+ paperless_ngx-default:
extends:
file: common-settings.yml
service: common-settings
+
+networks:
+ paperless_ngx-default:
+ name: paperless_ngx-${CONTEXT}
+ driver: bridge
+ driver_opts:
+ com.docker.network.enable_ipv6: "false"
+ ipam:
+ driver: default
+ config:
+ - subnet: ${SUBNET}
--
2.39.1
From fd35306130b35d9c4d77a3ca6849e96222a244cb Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:39:23 +0200
Subject: [PATCH 07/12] refactor(compose): No double quotes for port variable
needed
---
docker/compose/docker-compose.postgres-tika.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 05885149..f1dd4913 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -72,7 +72,7 @@ services:
- gotenberg
- tika
ports:
- - "${WEBSERVER_PORT}":8000
+ - ${WEBSERVER_PORT}:8000
healthcheck:
test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
interval: 30s
--
2.39.1
From 0defc3f29d607e8c81c655b15fb9e92cf3921fdb Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 22:53:20 +0200
Subject: [PATCH 08/12] refactor(compose): Replace hard-linked env file with
variables
---
docker/compose/docker-compose.postgres-tika.yml | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index f1dd4913..1a5abbb8 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -83,13 +83,17 @@ services:
- /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/media:/usr/src/paperless/media
- /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/export:/usr/src/paperless/export
- /opt/docker-data/paperless_ngx-${CONTEXT}/webserver/consume:/usr/src/paperless/consume
- env_file: docker-compose.env
environment:
PAPERLESS_REDIS: redis://broker:6379
PAPERLESS_DBHOST: db
PAPERLESS_TIKA_ENABLED: 1
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998
+ PAPERLESS_OCR_LANGUAGE: ${PAPERLESS_OCR_LANGUAGE}
+ PAPERLESS_OCR_LANGUAGES: ${PAPERLESS_OCR_LANGUAGES}
+ PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY}
+ PAPERLESS_TIME_ZONE: ${PAPERLESS_TIME_ZONE}
+ PAPERLESS_URL: ${PAPERLESS_URL}
gotenberg:
image: docker.io/gotenberg/gotenberg:7.8
--
2.39.1
From c044e323b9d09a95c782b0a216c02874be52611c Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Tue, 16 May 2023 23:15:26 +0200
Subject: [PATCH 09/12] feat(compose): Bind to a virtual IP address
---
docker/compose/docker-compose.postgres-tika.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 1a5abbb8..3cb20287 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -72,7 +72,7 @@ services:
- gotenberg
- tika
ports:
- - ${WEBSERVER_PORT}:8000
+ - ${WEBSERVER_VIP}:${WEBSERVER_PORT}:8000
healthcheck:
test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
interval: 30s
--
2.39.1
From 51744302beb94ff44d348f3abf834e5222aca675 Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Wed, 17 May 2023 23:33:44 +0200
Subject: [PATCH 10/12] fix(compose): Apply unpaper, use cleaned images to
build output file
---
docker/compose/docker-compose.postgres-tika.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index 3cb20287..ed60ce40 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -89,6 +89,7 @@ services:
PAPERLESS_TIKA_ENABLED: 1
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998
+ PAPERLESS_OCR_CLEAN: ${PAPERLESS_OCR_CLEAN}
PAPERLESS_OCR_LANGUAGE: ${PAPERLESS_OCR_LANGUAGE}
PAPERLESS_OCR_LANGUAGES: ${PAPERLESS_OCR_LANGUAGES}
PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY}
--
2.39.1
From 516c326ba44243c768b94b604d627616c580386c Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Wed, 17 May 2023 23:37:44 +0200
Subject: [PATCH 11/12] fix(compose): Let ping tell us when our virtual IP
address is bound
---
docker/compose/docker-compose.postgres-tika.yml | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index ed60ce40..e4e2b895 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -51,6 +51,12 @@ services:
extends:
file: common-settings.yml
service: common-settings
+ healthcheck:
+ test: ["CMD", "ping", "-qnc", "1", "-W", "0.2", "${WEBSERVER_VIP}"]
+ interval: 2s
+ timeout: 2s
+ retries: 3
+ start_period: 10s
volumes:
- /opt/docker-data/paperless_ngx-${CONTEXT}/db/data:/var/lib/postgresql/data
environment:
@@ -67,10 +73,11 @@ services:
file: common-settings.yml
service: common-settings
depends_on:
- - db
- - broker
- - gotenberg
- - tika
+ db:
+ condition: service_healthy
+ broker:
+ gotenberg:
+ tika:
ports:
- ${WEBSERVER_VIP}:${WEBSERVER_PORT}:8000
healthcheck:
--
2.39.1
From 249e2190fba5abd4a13c1311fe089936c3affe87 Mon Sep 17 00:00:00 2001
From: hygienic-books <>
Date: Wed, 17 May 2023 23:51:15 +0200
Subject: [PATCH 12/12] fix(compose): Unify depends_on
---
docker/compose/docker-compose.postgres-tika.yml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/docker/compose/docker-compose.postgres-tika.yml b/docker/compose/docker-compose.postgres-tika.yml
index e4e2b895..d3ec434d 100644
--- a/docker/compose/docker-compose.postgres-tika.yml
+++ b/docker/compose/docker-compose.postgres-tika.yml
@@ -76,8 +76,11 @@ services:
db:
condition: service_healthy
broker:
+ condition: service_started
gotenberg:
+ condition: service_started
tika:
+ condition: service_started
ports:
- ${WEBSERVER_VIP}:${WEBSERVER_PORT}:8000
healthcheck:
--
2.39.1

View File

@@ -1,12 +1,12 @@
# How to # How to
Add a `COMPOSE_ENV_FILE` and export it along with the location where this repo subdirectory lives, here for example `/opt/containers/snipeit` plus all other variables. At [env/fully.qualified.domain.name_ctx.example](env/fully.qualified.domain.name_ctx.example) you'll find an example file. For parameters that aren't self-explanatory check out [Environment variables](#environment-variables) below. Add a `COMPOSE_ENV_FILE` and export it along with the location where this repo subdirectory lives, here for example `/opt/containers/snipeit` plus all other variables. At [env/fully.qualified.domain.name_ctx.example](env/fully.qualified.domain.name_ctx.example) you'll find an example file. For parameters that aren't self-explanatory check out [Environment variables](#environment-variables) below. The `COMPOSE_CTX` here will be equal to whatever's in your `CONTEXT` down in [Environment variables](#environment-variables).
## Prep ## Prep
``` ```
export COMPOSE_DIR='/opt/containers/snipeit' export COMPOSE_DIR='/opt/containers/snipeit'
export COMPOSE_CTX='loft' export COMPOSE_CTX='ux_edmonton'
export COMPOSE_PROJECT_NAME='snipeit-'"${COMPOSE_CTX}" export COMPOSE_PROJECT_NAME='snipeit-'"${COMPOSE_CTX}"
export COMPOSE_FILE="${COMPOSE_DIR}"'/docker-compose.yml' export COMPOSE_FILE="${COMPOSE_DIR}"'/docker-compose.yml'
export COMPOSE_ENV_FILE= export COMPOSE_ENV_FILE=

View File

@@ -1,195 +0,0 @@
# Upstream repo
We use the [official Zabbix Docker GitHub repo](https://github.com/zabbix/zabbix-docker) for Docker Compose deployment, we add a few local changes.
Create dir
```
mkdir -p '/opt/git/github.com/zabbix/zabbix-docker/branches/latest'
```
Pull repo
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' clone 'https://github.com/zabbix/zabbix-docker' .
```
# Docker Compose
## Base setup
When everything's ready start Zabbix with Docker Compose, otherwise head down to [Initial setup](#initial-setup) or [Upgrade an existing repo](#upgrade-an-existing-repo) first.
Define variables assuming the official Zabbix Docker repo lives at `/opt/git/github.com/zabbix/zabbix-docker/branches/latest`:
```
export COMPOSE_DIR='/opt/git/github.com/zabbix/zabbix-docker/branches/latest'
export COMPOSE_FILE="${COMPOSE_DIR}"'/docker-compose_v3_alpine_pgsql_latest.yaml'
export COMPOSE_ENV_FILE=<add accordingly>
```
Run Zabbix like so
```
docker compose --file "${COMPOSE_FILE}" --env-file "${COMPOSE_ENV_FILE}" up --detach
```
## Additional files
- [common-settings.yml](common-settings.yml)
This file will be auto-created as part of the patch. Use it as an example in case patching fails
- [env/fqdn_context.env.example](env/fqdn_context.env.example)
An example env file with all currently used variables after `docker-compose_v3_alpine_pgsql_latest.yaml` is patched
- Directory tree underneath [build-context](build-context)
```
build-context/
└── docker-data
├── postgres
│   └── config
│   ├── cert
│   │   ├── ZBX_PGSQL_TLS_CA_CERT_FILE
│   │   ├── ZBX_PGSQL_TLS_CERT_FILE
│   │   └── ZBX_PGSQL_TLS_KEY_FILE
│   └── docker-entrypoint-initdb.d
│   └── init-user-db.sh
├── zabbixserver
│   └── config
│   └── cert
│   ├── ZBX_SERVER_TLS_CA_CERT_FILE
│   ├── ZBX_SERVER_TLS_CERT_FILE
│   └── ZBX_SERVER_TLS_KEY_FILE
└── zabbixwebnginx
└── config
└── cert
├── dhparam.pem
├── ZBX_WEBNGINX_TLS_CERT_FULLCHAIN_FILE
└── ZBX_WEBNGINX_TLS_KEY_FILE
```
Example data you're going to want to physically place on your deployment machine. SSL certs and keys are blank files each of which has the exact same name used in env file `fqdn_context.env.example`.
In [postgres/config/docker-entrypoint-initdb.d](build-context/docker-data/postgres/config/docker-entrypoint-initdb.d) a PostgreSQL initialization script - when this container is run on a completely empty data directory - will create an additional read-only user `ZBX_DB_USERNAME_RO` with password `ZBX_DB_USERNAME_PW`. The example's intended to grant a Grafana daemon direct PostgreSQL database read access.
# Upgrade an existing repo
Check [Initial setup](#initial-setup) below for first time steps. On consecutive upgrades proceed as follows.
## Revert unpushed local changes
Return repo state to exactly the upstream repo's original branch state throwing away the commits you added.
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' reset --hard origin
```
Switch to `trunk` branch, get newest commits from upstream
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' checkout trunk
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' pull
```
Pick and checkout new tag
```
while IFS= read -r tag; do printf -- '%s %s\n' "$(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' rev-list -n 1 'refs/tags/'"${tag}")" "${tag}"; done < <(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' --no-pager tag --list --sort=v:refname)
# Output goes like:
...
89511f06ad4de6b373f10b06604dc5d8e1da02df 6.2.7
2025ec8ad74f59981ad6598e9f6cd2a5c9c99f6b 6.2.8
59a91bfbb6e46885f201e50f9197a7a44d3ba3ac 6.2.9
9f2e726e554b23595489eb66c8e11e5d114b573f 6.4.0
9f16f6d773a2a46f1595c86077899d1e040db283 6.4.1
0fa87156974e799e04bf99e5300bad6830d754ab 6.4.2
...
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' checkout 'tags/x.y.z'
```
Lastly [apply patch](#apply-patch). If patch does not apply cleanly read on in the next section [Create new patch](#create-new-patch) to find out how to fix your patch.
# Create new patch
## Add your changes as commits
Get `zabbix-docker` repo into a state with which you're happy then
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' format-patch 7665739620ba6d99090838d502ab76d2f5a47e96^..a17380598ca66153ddc2a42eb618d906d4f582e6 --stdout > '/opt/containers/zabbixserver/zabbix-docker.patch'
```
Where the first commit hash is our first commit and the other commit hash is our last commit. Note the caret (`^`) right after the first commit hash.
## Investigation
You may have to try and find out how a known good base commit differs from a newer one in case the newer one does no longer cleanly accept the patch.
Get commit hashes from both affected tags, e.g.
```
while IFS= read -r tag; do printf -- '%s %s\n' "$(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' rev-list -n 1 'refs/tags/'"${tag}")" "${tag}"; done < <(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' --no-pager tag --list --sort=v:refname)
# Output goes like:
...
89511f06ad4de6b373f10b06604dc5d8e1da02df 6.2.7
2025ec8ad74f59981ad6598e9f6cd2a5c9c99f6b 6.2.8
59a91bfbb6e46885f201e50f9197a7a44d3ba3ac 6.2.9
9f2e726e554b23595489eb66c8e11e5d114b573f 6.4.0
9f16f6d773a2a46f1595c86077899d1e040db283 6.4.1
0fa87156974e799e04bf
# Run
99e5300bad6830d754ab 6.4.2
...
```
Diff them
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' diff 9f16f6d773a2a46f1595c86077899d1e040db283 0fa87156974e799e04bf99e5300bad6830d754ab 'docker-compose_v3_alpine_pgsql_latest.yaml'
```
Output will be empty in case no difference exists in `docker-compose_v3_alpine_pgsql_latest.yaml` between both commit hashes.
Commit your updated patch file into _this_ repo. With a new working patch in hand head back up to [Upgrade an existing repo](#upgrade-an-existing-repo).
# Initial setup
## Prep
Get desired tag e.g. from version-sorted tags list
```
while IFS= read -r tag; do printf -- '%s %s\n' "$(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' rev-list -n 1 'refs/tags/'"${tag}")" "${tag}"; done < <(git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' --no-pager tag --list --sort=v:refname)
# Output goes like:
...
89511f06ad4de6b373f10b06604dc5d8e1da02df 6.2.7
2025ec8ad74f59981ad6598e9f6cd2a5c9c99f6b 6.2.8
59a91bfbb6e46885f201e50f9197a7a44d3ba3ac 6.2.9
9f2e726e554b23595489eb66c8e11e5d114b573f 6.4.0
9f16f6d773a2a46f1595c86077899d1e040db283 6.4.1
0fa87156974e799e04bf99e5300bad6830d754ab 6.4.2
...
```
Switch to desired tag
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' checkout 'tags/6.4.2'
```
## Apply patch
Identify yourself to the local `zabbix-docker` repo
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' config user.name "hygienic-books"
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' config user.email "hygienic-books@tentic.net"
```
Apply `zabbix-docker.patch` to Docker Compose file. We use Zabbix' `docker-compose_v3_alpine_pgsql_latest.yaml` Compose file. Assuming this repo lives at `/opt/containers/zabbixserver`:
```
git -C '/opt/git/github.com/zabbix/zabbix-docker/branches/latest' am '/opt/containers/zabbixserver/zabbix-docker.patch'
# Output will be:
Applying: refactor(compose): Remove trailing whitespace
Applying: refactor(compose): 4 leading spaces
Applying: refactor(compose): Indent comments
Applying: refactor(zabbix-server): Set correct libs paths
Applying: refactor(zabbix-server): Set TLS cert file names
Applying: feat(zabbix-server): Replace env files with variables
...
```
And now back up to [Docker Compose](#docker-compose).

View File

@@ -1,21 +0,0 @@
#!/bin/bash
set -e
sleep 20
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
DO
\$block\$
BEGIN
IF EXISTS (
SELECT FROM pg_catalog.pg_roles WHERE rolname = '${ZBX_DB_USERNAME_RO}'
)
THEN
RAISE NOTICE 'Role ''${ZBX_DB_USERNAME_RO}'' already exists. Skipping ...';
ELSE
CREATE ROLE "${ZBX_DB_USERNAME_RO}" WITH PASSWORD '${ZBX_DB_USERNAME_PW}';
GRANT CONNECT ON DATABASE "${POSTGRES_DB}" TO "${ZBX_DB_USERNAME_RO}";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "${ZBX_DB_USERNAME_RO}";
END IF;
END
\$block\$;
EOSQL

View File

@@ -1,13 +0,0 @@
-----BEGIN DH PARAMETERS-----
MIICCAKCAgEA1shg4Gf/2rG+kllZ1qE2or0BHGqhDdjw0DlwNlPL9qVaiqmU/TRq
LCxr0ZloKa8dwImvEtwxy8bJROMW7gcVfYebsOwTnNbQGePkQ3OSKyyBBG+A04rx
QAT6mxgG84ydQOicu42mK0lRwWeFUzZFauZa8CWEcaLcKBUxYQWN6QXOAk7pUQ32
3vAjUKL8+dYUINCna5QXOPmNgnSmXJfjPEnLwveDUTj6IaXFLvWmJm4yRgi7AvXF
r85aAKl9FgT7e5+BntpJAP4Mj7TYxVyHHq7BLZAke7slwe6bkFLxQ6H3INlTYWgp
QEmALgW+KjiARTTh12NJgJvT0ti4ck7VA6P9eN5kw4FCEg1hZbMLFQg7asUWq9tV
7usrDC971W46YsrBstQg851Vbs64ZMf5+knHYJIWaUF5ZTQ1cHihKhEfGJOdRvxU
Py2q192knNzXwroqi/q22iUe9zu4kPRI3qLjR1brVcf8mkUGnMtkIZsO6cdHdvf9
+2De05V57/yCp8R1QUY/UErdDSO+ey+gNFVfpIBdUIoy8+bG1Dcz70X8DDHXD+4+
DJXeajEWS4xkHEB8kaoYGHS6dDJpQk/nsk2H4Mdb1M/uYDedLdMh3FVjH40lzQzR
oRYpzgieag0RPJcaxi6z8PN0HEuVpPA8EbOvxwDMR2zp4zJxHuA0inMCAQI=
-----END DH PARAMETERS-----

View File

@@ -1,27 +0,0 @@
# This file is maintained by remco and populated with data from HashiCorp
# Vault. Changes not done in Vault will be reverted when file gets rendered.
CTX=
PHP_TZ=
POSTGRES_DB=
POSTGRES_PASSWORD=
POSTGRES_USER=
VAULT_TOKEN=
ZBX_DB_USERNAME_PW=
ZBX_DB_USERNAME_RO=
ZBX_PGSQL_TLS_CA_CERT_FILE=
ZBX_PGSQL_TLS_CERT_FILE=
ZBX_PGSQL_TLS_KEY_FILE=
ZBX_SERVER_HOST=
ZBX_SERVER_NAME=
ZBX_SERVER_TLS_CA_CERT_FILE=
ZBX_SERVER_TLS_CERT_FILE=
ZBX_SERVER_TLS_KEY_FILE=
ZBX_VAULTDBPATH=
ZBX_VAULTURL=
ZBX_WEBNGINX_EXPOSED_HTTPS_PORT=
ZBX_WEBNGINX_EXPOSED_HTTP_PORT=
ZBX_WEBNGINX_TLS_CERT_FULLCHAIN_FILE=
ZBX_WEBNGINX_TLS_KEY_FILE=
# When needed for temporary debugging
# ZBX_DEBUGLEVEL=4

File diff suppressed because it is too large Load Diff