diff --git a/README.md b/README.md index 55d9d65..c0a0a4c 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,199 @@ -# haproxy +# HAProxy Docker Compose files -HAProxy container setup \ No newline at end of file +Docker Compose files to spin up an instance of HAProxy. + +# How to run + +Add a `COMPOSE_ENV` file and save its location as a shell variable along with the location where this repo lives, here for example `/opt/containers/haproxy` plus all other variables. At [env/fqdn_context.env.example](env/fqdn_context.env.example) you'll find an example environment file. + +When everything's ready start HAProxy with Docker Compose, otherwise head down to [Initial setup](#initial-setup) first. + +## Environment +``` +export COMPOSE_DIR='/opt/containers/haproxy' +export COMPOSE_CTX='ux_vilnius' +export COMPOSE_PROJECT='haproxy-'"${COMPOSE_CTX}" +export COMPOSE_FILE="${COMPOSE_DIR}"'/docker-compose.yml' +export COMPOSE_ENV= +``` + +## Context + +On your deployment machine create the necessary Docker context to connect to and control the Docker daemon on whatever target host you'll be using, for example: +``` +docker context create fully.qualified.domain.name --docker 'host=ssh://root@fully.qualified.domain.name' +``` + +## Start + +``` +docker --context 'fully.qualified.domain.name' compose --project-name "${COMPOSE_PROJECT}" --file "${COMPOSE_FILE}" --env-file "${COMPOSE_ENV}" up --detach +``` + +## Reload + +HAProxy lends itself well to graceful reloads in a similar fashion to Nginx. Considering a running HAProxy container named `containername` check config file syntax: + +``` +docker exec -i containername haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg +``` + +Reload HAProxy + +``` +docker kill --signal SIGHUP containername +``` + +# Initial setup + +We're assuming you run Docker Compose workloads with ZFS-based bind mounts. ZFS management, creating a zpool and setting adequate properties for its datasets is out of scope of this document. + +## Datasets + +Create ZFS datasets. + +* Parent dateset + ``` + zfs create -o mountpoint=/opt/docker-data 'zpool/docker-data' + ``` + +* Container-specific dataset + ``` + zfs create -p 'zpool/docker-data/haproxy-'"${COMPOSE_CTX}"'/haproxy/config' + ``` + +* Create a subdir for certificates + ``` + mkdir -p '/opt/docker-data/haproxy-'"${COMPOSE_CTX}"'/haproxy/config/certs' + ``` + +A change in file or directory ownership is typically not needed. HAProxy runs by default as user ID 99 and group ID 99, it has no problem accessing a bind-mounted file owned by `root:root` with typical `0644` (`rw-r--r--`) permissions. + +## Additional files + +Place the following files on target server. Use the directory structure at [build-context](build-context) as a guide, specifically at `docker-data`. + +``` +build-context/ +├── docker-data +│   └── config +│   ├── certs +│   │   ├── example.com.pem +│   │   ├── example.com.pem.key +│   │   ├── example.net.pem +│   │   └── example.net.pem.key +│   ├── haproxy.cfg +│   └── hosts.map +├── Dockerfile +└── extras +``` + +In our example config (see next section [HAProxy config example](#haproxy-config-example)) each certificate is stored in what HAProxy calls a cert bundle: one complete PEM-formatted plain text SSL certificate chain file `something.pem` plus its matching key named `something.pem.key`. The chain file from top to bottom contains domain certificate, optionally intermediate certificate and lastly Certificate Authority certificate. With this naming scheme HAProxy finds keys and matching certs automatically. + +When done head back up to [How to run](#how-to-run). + +# HAProxy config example + +Proper config for your HAProxy use case is beyond the scope of this document. The example [build-context/docker-data/config/haproxy.cfg](build-context/docker-data/config/haproxy.cfg) file is a courtesy config to get you started. Your real world file will likely turn out differently. + +Here's a quick rundown of what you see in the example file. + +## Section `global` + +``` +global + daemon + maxconn 200 + log stdout format raw local0 + ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 + ssl-default-bind-options prefer-client-ciphers no-tls-tickets ssl-min-ver TLSv1.3 + ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 + ssl-default-server-options no-tls-tickets ssl-min-ver TLSv1.3 +``` + +We follow recommendations from [Mozilla's SSL Configuration Generator](https://ssl-config.mozilla.org/) for SSL settings. The container outputs logging data to `stdout`. We're setting a global connection limit of `maxconn 200`. Keeping some limit in place makes HAProxy behavior predictable, it doesn't have to be this low. This argument goes hand in hand with the `ulimits` directive in [docker-compose.yml](docker-compose.yml). + +``` +... + ... + ulimits: + nproc: ${ULIMIT_NPROC-65535} + nofile: + soft: ${ULIMIT_NPROC-65535} + hard: ${ULIMIT_NPROC-65535} +``` + +If you don't specify a sensible limit HAProxy will try and allocate whatever your platform allows. + +## Sections `defaults` + +``` +defaults generic_defaults + log global + maxconn 50 + + # Irrelevant on frontends + timeout connect 5s + timeout server 5s + + # Irrelevant on backends + timeout client 30s + +defaults http_defaults from generic_defaults + mode http + option forwardfor + option httplog +``` + +Set generic defaults and supplement them with HTTP-specific defaults. + +## Statistics + +``` +frontend fe_stats from http_defaults + bind :"${STATS_PORT-61000}" + stats enable + stats uri / + stats refresh 10s + acl ips_allowed src 10.10.10.0/24 10.10.1.0/24 + http-request deny if !ips_allowed +``` + +Enable statistics output. Access to this endpoint is permitted only from `acl ips_allowed` which can be for example network prefixes in your local network. + +## A proxy example + +``` +frontend fe_https-proxy from http_defaults + bind :80 + bind :443 ssl strict-sni crt /usr/local/etc/haproxy/certs alpn h2,http/1.1 + redirect scheme https code 301 if !{ ssl_fc } + http-response set-header Strict-Transport-Security max-age=63072000 + use_backend %[req.hdr(host),lower,map_dom(/usr/local/etc/haproxy/hosts.map,be_no-match)] +``` + +We use a `hosts.map` file to map HTTP `host` header information for incoming requests to the backend that should serve them. Check out [build-context/docker-data/config/hosts.map](build-context/docker-data/config/hosts.map) for an example file. + +Also HTTP traffic gets redirected to HTTPS and we specify more best practices suggested by [Mozilla's SSL Configuration Generator](https://ssl-config.mozilla.org/). + +``` +backend be_example.net from http_defaults + compression algo gzip + compression type text/plain text/css + option httpchk + http-check send meth HEAD uri /status hdr Host example.net hdr User-Agent "${HEALTH_CHECK_USER_AGENT-HAProxy} health check" + server example.net example.net:8080 check inter 10s fall 3 rise 3 +``` + +In our backend example we do HTTP health checks by sending a `HEAD` request every 10 seconds. When doing so we identify ourselves as an HAProxy instance so the downstream application can recognize legitimate health checks in its logs. + +Note that both `frontend` and `backend` inherit settings from the `http_defaults` section defined earlier. + +## No match + +``` +backend be_no-match from http_defaults + http-request deny deny_status 404 +``` + +To requests without a match we serve an HTTP status code 404. diff --git a/build-context/Dockerfile b/build-context/Dockerfile new file mode 100644 index 0000000..67530b5 --- /dev/null +++ b/build-context/Dockerfile @@ -0,0 +1,14 @@ +# For the remainder of this Dockerfile EXAMPLE_ARG_FOR_DOCKERFILE will be +# available with a value of 'must_be_available_in_dockerfile', check out the env +# file at 'env/fully.qualified.domain.name.example' for reference. +# ARG EXAMPLE_ARG_FOR_DOCKERFILE + +# Another env var, this one's needed in the example build step below: +# ARG HAPROXY_VERSION + +# Example +# FROM "haproxy:${HAPROXY_VERSION}" +# RUN apt-get update && \ +# apt-get -y install \ +# somepackage-6.q16-6-extra && \ +# rm -rf /var/lib/apt/lists/* diff --git a/build-context/docker-data/.gitkeep b/build-context/docker-data/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/build-context/docker-data/config/certs/example.com.pem b/build-context/docker-data/config/certs/example.com.pem new file mode 100644 index 0000000..e69de29 diff --git a/build-context/docker-data/config/certs/example.com.pem.key b/build-context/docker-data/config/certs/example.com.pem.key new file mode 100644 index 0000000..e69de29 diff --git a/build-context/docker-data/config/certs/example.net.pem b/build-context/docker-data/config/certs/example.net.pem new file mode 100644 index 0000000..e69de29 diff --git a/build-context/docker-data/config/certs/example.net.pem.key b/build-context/docker-data/config/certs/example.net.pem.key new file mode 100644 index 0000000..e69de29 diff --git a/build-context/docker-data/config/haproxy.cfg b/build-context/docker-data/config/haproxy.cfg new file mode 100644 index 0000000..71e8ab9 --- /dev/null +++ b/build-context/docker-data/config/haproxy.cfg @@ -0,0 +1,49 @@ +global + daemon + maxconn 200 + log stdout format raw local0 + ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 + ssl-default-bind-options prefer-client-ciphers no-tls-tickets ssl-min-ver TLSv1.3 + ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 + ssl-default-server-options no-tls-tickets ssl-min-ver TLSv1.3 + +defaults generic_defaults + log global + maxconn 50 + + # Irrelevant on frontends + timeout connect 5s + timeout server 5s + + # Irrelevant on backends + timeout client 30s + +defaults http_defaults from generic_defaults + mode http + option forwardfor + option httplog + +frontend fe_stats from http_defaults + bind :"${STATS_PORT-61000}" + stats enable + stats uri / + stats refresh 10s + acl ips_allowed src 10.10.10.0/24 10.10.1.0/24 + http-request deny if !ips_allowed + +frontend fe_https-proxy from http_defaults + bind :80 + bind :443 ssl strict-sni crt /usr/local/etc/haproxy/certs alpn h2,http/1.1 + redirect scheme https code 301 if !{ ssl_fc } + http-response set-header Strict-Transport-Security max-age=63072000 + use_backend %[req.hdr(host),lower,map_dom(/usr/local/etc/haproxy/hosts.map,be_no-match)] + +backend be_example.net from http_defaults + compression algo gzip + compression type text/plain text/css + option httpchk + http-check send meth HEAD uri /status hdr Host example.net hdr User-Agent "${HEALTH_CHECK_USER_AGENT-HAProxy} health check" + server nextcloud-loft example.net:8080 check inter 10s fall 3 rise 3 + +backend be_no-match from http_defaults + http-request deny deny_status 404 diff --git a/build-context/docker-data/config/hosts.map b/build-context/docker-data/config/hosts.map new file mode 100644 index 0000000..50ecf4a --- /dev/null +++ b/build-context/docker-data/config/hosts.map @@ -0,0 +1,6 @@ +# See https://www.haproxy.com/blog/how-to-map-domain-names-to-backend-server-pools-with-haproxy +# +# domain backend +fully.qualified.domain.name be_example.net +example.net be_example.net +another.domain.com be_example.net diff --git a/build-context/extras/.gitkeep b/build-context/extras/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/common-settings.yml b/common-settings.yml new file mode 100644 index 0000000..9fd26d7 --- /dev/null +++ b/common-settings.yml @@ -0,0 +1,11 @@ +services: + common-settings: + environment: + TZ: "${TIMEZONE:-Etc/UTC}" + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "10" + compress: "true" + restart: "${RESTARTPOLICY:-unless-stopped}" diff --git a/docker-compose.override.yml b/docker-compose.override.yml new file mode 100644 index 0000000..13dff84 --- /dev/null +++ b/docker-compose.override.yml @@ -0,0 +1,10 @@ +services: + haproxy-build: + image: "haproxy:${HAPROXY_VERSION}" + profiles: ["build"] + build: + context: "build-context/haproxy" + dockerfile: Dockerfile + args: + EXAMPLE_ARG_FOR_DOCKERFILE: "${EXAMPLE_ARG_FROM_ENV_FILE}" + HAPROXY_VERSION: "${HAPROXY_VERSION}" diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..dfa271c --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,35 @@ +services: + haproxy: + image: "haproxy:${HAPROXY_VERSION}" + container_name: "haproxy-${CONTEXT}" + networks: + haproxy-default: + ulimits: + nproc: ${ULIMIT_NPROC-65535} + nofile: + soft: ${ULIMIT_NPROC-65535} + hard: ${ULIMIT_NPROC-65535} + extends: + file: common-settings.yml + service: common-settings + ports: + - 80:80 + - 443:443 + - ${STATS_PORT}:${STATS_PORT} + volumes: + - /opt/docker-data/haproxy-${CONTEXT}/haproxy/config/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg + - /opt/docker-data/haproxy-${CONTEXT}/haproxy/config/hosts.map:/usr/local/etc/haproxy/hosts.map + - /opt/docker-data/haproxy-${CONTEXT}/haproxy/config/certs:/usr/local/etc/haproxy/certs + environment: + STATS_PORT: ${STATS_PORT} + HEALTH_CHECK_USER_AGENT: "haproxy-${CONTEXT}" +networks: + haproxy-default: + name: haproxy-${CONTEXT} + driver: bridge + driver_opts: + com.docker.network.enable_ipv6: "false" + ipam: + driver: default + config: + - subnet: ${SUBNET} diff --git a/env/fqdn_context.env.example b/env/fqdn_context.env.example new file mode 100644 index 0000000..4187dd9 --- /dev/null +++ b/env/fqdn_context.env.example @@ -0,0 +1,36 @@ +CONTEXT=ux_vilnius + + + +# Set something sensible here and uncomment +# --- +# HAPROXY_VERSION=x.y.z + + + +# On which port do you want HAProxy's statistics page exposed? +STATS_PORT=61000 + + + +# Feel free to leave defaults. They apply while these vars are commented out +# --- +# RESTARTPOLICY=unless-stopped +# TIMEZONE=Etc/UTC + + + +# Subnet to use for this Docker Compose project. Docker defaults to +# container networks in prefix 172.16.0.0/12 which is 1 million addresses in +# the range from 172.16.0.0 to 172.31.255.255. Docker uses 172.17.0.0/16 for +# itself. Use any sensible prefix in 172.16.0.0/12 here except for Docker's +# own 172.17.0.0/16. +# --- +SUBNET=172.30.95.0/24 + + + +# See 'docker-compose.override.yml' for how to make a variable available in +# a Dockerfile +# --- +# EXAMPLE_ARG_FROM_ENV_FILE=must_be_available_in_dockerfile