bolhaverso

TL;DR

Sobre os recentes acontecimentos, estamos aqui para esclarecer que vamos continuar.

Ouvimos os apelos de nossos usuários, entendemos que a bolha deixou de ser um simples side-project e se tornou referência no universo Mastodon, e por isso deve continuar.

Seguir sim, mas com melhorias

Depois de algumas considerações com os usuários, parceiros e apoiadores, entendemos que podemos seguir operando, contudo, algumas coisas vão mudar.

Novo Conselho

Decidimos criar um conselho para gerir os serviços da bolha e decidir seu funcionamento e expansão de forma coletiva e responsável.

Mais ajuda

Algumas pessoas nos ofereceram ajuda para manter a bolha rodando e dividir o trabalho conosco, estamos organizando e estruturando esse novo formato de equipe.

Guardians & Pilots

Quem já era Guardian (moderador) e pilot (admin) continua com os mesmos privilégios e contamos com a ajuda sempre.

Estamos também recrutando novos moderadores e pilots com mais tempo para atuar de forma mais regular na bolha.

Serviços da Bolha

Vamos manter no ar os seguintes serviços de forma permanente:

  • bolha.us
  • bolha.chat

E o serviços beta abaixo serão alçados a estáveis e mantidos como tal:

  • bolha.blog
  • bolha.photos
  • bolha.social

A bolha.tools também segue

  • libretranslate.bolha.tools (tradutor)
  • video.bolha.tools (jitsi)
  • tube.bolha.tools (invidious youtube client)
  • notes.bolha.tools (hedgedoc)

Bolha.chat

A bolha.chat passe a ter como co-owner o sr. @robertomurta, que irá cuidar desse ambiente com muito carinho e atenção.

  • bolha.chat
  • element.bolha.chat
  • hydrogen.bolha.chat
  • cinny.bolha.chat

Bolha.video

O Jitsi será integrado ao bolha.chat como seu canal de vídeo oficial.

Bookwyrm chegando

Esse será o novo projeto a ser integrado a bolha em tempo, estava na fila e segue em testes e homologação na GCN.

Quem é a GCN.sh?

A GCN.sh passa a ser a madrinha e responsável por manter a plataforma e a infraestrutura que o coletivo bolha usa para rodar seus serviços.

A GCN é um projeto do founder Guto Carvalho.

Podemos dizer que a bolha segue rodando nos servidores da GCN.

A GCN é um novo projeto que estará focado em estudar e disseminar conhecimento técnico relativo ao Fediverso, bem como testar, homologar e documentar ferramentas que poderão ser adotadas pela bolha no futuro.

A GCN não é só Fediverso, também estudamos Cloud Native, Kubernetes, DevOps, Infra as Code e Software Engineering.

Nosso founder agora vai se comunicar via mastodon na conta @gutocarvalho@gcn.sh, você pode encontrá-lo por lá e segui-lo.

Siga a conta avisos@bolha.us para mais informações.

[s]

Introduction

This post is a Howto to install Mastodon 4.1.2 using Docker.

We hope you can install Mastodon 4.1.2 using docker like we did.

This is almost the same setup we use to run the instance https://bolha.us.

Information Section

Base Linux System

Ubuntu 20.04 or higher, always.

We're running a Virtual Machine (KVM) inside an open-source Hypervisor.

Hardware size

This proposal uses only docker to handle 500 active users in a 1500 registered server running on a single node (KVM).

  • vpcu: 8 (12 ideal)
  • memory: 12 gb ram (16 gb ideal)
  • network: 500 mbits network minimal (1 gbit ideal)
  • disk: 670 gb

Partition layout

  • root (50g)
  • /var/lib/docker (50g)
  • /var/log (50g)
  • /opt (500g)
  • /tmp (10g)
  • /swapfile (10g)

OPT reserves

  • 250 gb reserved for mastodon upload files
  • 50 gb reserved for elastic
  • 50 gb reserved for postgres
  • 25 gb reserved for redis
  • 25 gb reserved for nginx cache (if in the same server)
  • 100 gb reserved for normal growth of your mastodon instance

Our Baremetal Provider

  • OVH Canada
    • OVH BareMetal ECO
    • Running ProxMox 7.1

We have our own BareMetal Server with ProxMox 7.

We have several Virtual Machines running different Fediverse Tools.

Our VPS Providers

We use OVH/Canada VPS for

  • Load Balancer (NGINX Primary)
  • Video Conference (Our Jitsi instance)

We use VULTR/VPS

  • Load Balancer (NGINX standby)
  • Monitoring (Status Kuma)
  • Other notification services

Other Providers

  • Namecheap to register our domains.
  • CloudFlare to configure and serve DNS Records.
  • Wasabi as our Object Storage/CDN for Mastodon Media
  • BlackBlaze as our Object Storage for Backup

OnPrem Services

  • We're using Uptime Kuma to monitor our instance.
  • We're running our own SMTP Mail Server (Zimbra 8).

PreReqs Section

IPTABLES Config

if you are running nginx on the same machine

  • 22, 80, 443 TCP opened
  • all other traffic blocked on the filter input table

if you are running nginx externally

  • 22 TCP opened
  • 3000 and 4000 TCP only to your NGINX IP
  • all other traffic blocked on the filter input table

in your nginx server (if it's dedicated to mastodon)

  • 22, 80, 443 TCP opened
  • all other traffic blocked on the filter input table

Fail2ban Config

Use it

  • get your port 22 (ssh) protected always

App-armor

First, It's essential to have it up and running continuously.

However, it can cause abnormal behaviors in some scenarios. It's best to keep this component disabled – during the installation, especially if you don't know how to use it or how to configure profiles in case of a problem between docker, mastodon, and app-armor.

It usually won't interfere with the docker or mastodon configuration, but, in case of problems with aa-profiles, we won't cover the solution here. It's best for you to disable it during the installation, and you can re-enable it after, if you want, and know what you are doing.

During the how-to validation, the default Ubuntu app-armor config was enabled, and everything worked fine. However, it's important to mention this in the case of different app-armor configs.

Installation section

1. Installing docker

installing

curl https://get.docker.com | bash

enabling

systemctl enable docker

starting

systemctl start docker

2. Installing docker-compose

installing

curl -s https://api.github.com/repos/docker/compose/releases/latest | grep browser_download_url  | grep docker-compose-linux-x86_64 | cut -d '"' -f 4 | wget -qi -

enabling

chmod +x docker-compose-linux-x86_64

moving to /usr/local/bin, make sure that the dir it's in the PATH var.

mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose

3. Creating directories

main dirs

mkdir -p /opt/mastodon/
mkdir -p /opt/mastodon/{docker,data}

sub-dirs

mkdir -p /opt/mastodon/data/{app,web,database}
mkdir -p /opt/mastodon/data/database/{postgresql,redis,elasticsearch}
mkdir -p /opt/mastodon/data/web/{public,system}

4. Configuring permissions

creating user and groups

groupadd -g 991 mastodon
useradd mastodon -u 991 -g 991

fixing web perms

chown -R mastodon:mastodon /opt/mastodon/data/web

fixing database perms

chown -R 1000:1000 /opt/mastodon/data/database/elasticsearch

5. Creating docker config

create the file

vim /opt/mastodon/docker/docker-compose.yml

content

version: '3'

services:
  postgresql:
    image: "postgres:${POSTGRESQL_VERSION}"
    container_name: mastodon_postgresql
    restart: always
    env_file: 
      - database.env
      - versions.env
    shm_size: 256mb
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
    volumes:
      - postgresql:/var/lib/postgresql/data
    networks:
      - internal_network

  redis:
    image: "redis:${REDIS_VERSION}"
    container_name: mastodon_redis
    restart: always
    env_file: 
      - database.env
      - versions.env
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    volumes:
      - redis:/data
    networks:
      - internal_network

  redis-volatile:
    image: "redis:${REDIS_VERSION}"
    container_name: mastodon_redis_cache
    restart: always
    env_file: 
      - database.env
      - versions.env
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    networks:
      - internal_network

  elasticsearch:
    image: "elasticsearch:${ELASTICSEARCH_VERSION}"
    container_name: mastodon_elastisearch
    restart: always
    env_file: 
      - database.env
      - versions.env
    environment:
      - cluster.name=elasticsearch-mastodon
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ingest.geoip.downloader.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "nc -z elasticsearch 9200"]
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    networks:
      - internal_network

  website:
    image: "tootsuite/mastodon:${MASTODON_VERSION}"
    container_name: mastodon_website
    env_file: 
      - application.env
      - database.env
      - versions.env
    command: bash -c "bundle exec rails s -p 3000"
    restart: always    
    depends_on:
      - postgresql
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '3000:3000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
    volumes:
      - public:/opt/mastodon/public
      - uploads:/opt/mastodon/public/system
      - app:/opt/mastodon/app
       
  streaming:
    image: "tootsuite/mastodon:${MASTODON_VERSION}"
    container_name: mastodon_streaming
    env_file: 
      - application.env
      - database.env
      - versions.env
    command: node ./streaming
    environment:
      - DB_POOL=4
    restart: always
    depends_on:
      - postgresql
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '4000:4000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1']
    volumes:
      - public:/opt/mastodon/public
      - uploads:/opt/mastodon/public/system
      - app:/opt/mastodon/app

  sidekiq:
    image: "tootsuite/mastodon:${MASTODON_VERSION}"
    container_name: mastodon_sidekiq
    env_file: 
      - application.env
      - database.env
      - versions.env
    restart: always
    depends_on:
      - postgresql
      - redis
      - redis-volatile
      - website
    networks:
      - internal_network
      - external_network
    environment:
      - DB_POOL=18
    healthcheck:
      test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
    command: bundle exec sidekiq -c 18
    volumes:
      - public:/opt/mastodon/public
      - uploads:/opt/mastodon/public/system
      - app:/opt/mastodon/app

  shell:
    image: "tootsuite/mastodon:${MASTODON_VERSION}"
    env_file: 
      - application.env
      - database.env
      - versions.env
    command: /bin/bash 
    restart: "no"
    networks:
      - internal_network
      - external_network
    volumes:
      - public:/opt/mastodon/public
      - uploads:/opt/mastodon/public/system
      - app:/opt/mastodon/app

networks:
  external_network:
  internal_network:
    internal: true

volumes:
  postgresql:
    driver_opts:
      type: none
      device: /opt/mastodon/data/database/postgresql
      o: bind    
  redis:
    driver_opts:
      type: none
      device: /opt/mastodon/data/database/redis
      o: bind    
  elasticsearch:
    driver_opts:
      type: none
      device: /opt/mastodon/data/database/elasticsearch
      o: bind    
  uploads:
    driver_opts:
      type: none
      device: /opt/mastodon/data/web/system
      o: bind
  app:
    driver_opts:
      type: none
      device: /opt/mastodon/data/app
      o: bind
  public:
    driver_opts:
      type: none
      device: /opt/mastodon/data/web/public
      o: bind

6. Env Files

6.1 versions.env

create the versions.env file

vim /opt/mastodon/docker/versions.env

content

MASTODON_VERSION=v4.1.2
POSTGRESQL_VERSION=14
ELASTICSEARCH_VERSION=7.17.10
REDIS_VERSION=7

creating a symbolic link to load the versions properly

cd /opt/mastodon/docker; ln -s versions.env .env

6.2 application.env

create the application.env file

vim /opt/mastodon/docker/application.env

content

# environment config

RAILS_ENV=production
NODE_ENV=production

# web performance/tuning/concurrency

WEB_CONCURRENCY=2
MAX_THREADS=4

# locale config

DEFAULT_LOCALE=en

# local domain of your instance

LOCAL_DOMAIN=dev.bolha.us

# redirect to the first profile?

SINGLE_USER_MODE=false

# rails will serve static files?

RAILS_SERVE_STATIC_FILES=true

# email config

SMTP_SERVER=smtp.provider.tld
SMTP_PORT=587
SMTP_LOGIN=mastodon@provider.tld
SMTP_AUTH_METHOD=plain
SMTP_FROM_ADDRESS=mastodon@provider.tld
SMTP_PASSWORD=change_this_password_to_the_real_one

# secrets

SECRET_KEY_BASE=YOU_WILL_GENERATE_AND_REPLACE_HERE_LATER_CONTINUE_THE_DOC
OTP_SECRET=YOU_WILL_GENERATE_AND_REPLACE_HERE_LATER_CONTINUE_THE_DOC

# web push

VAPID_PRIVATE_KEY=YOU_WILL_GENERATE_AND_REPLACE_HERE_LATER_CONTINUE_THE_DOC
VAPID_PUBLIC_KEY=YOU_WILL_GENERATE_AND_REPLACE_HERE_LATER_CONTINUE_THE_DOC

6.3 database.env

create the database.env file

vim /opt/mastodon/docker/database.env

content

# postgresql config

POSTGRES_HOST=postgresql
POSTGRES_USER=mastodon
POSTGRES_DB=mastodon_production
POSTGRES_PASSWORD=you_will_change_this_password_ahead_on_this_doc

# elasticsearch config

ES_JAVA_OPTS="-Xms1024m -Xmx2048m"
ELASTIC_PASSWORD=you_will_change_this_password_ahead_on_this_doc

# redis config

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_URL=redis://redis:6379

# redis cache config

CACHE_REDIS_HOST=redis-volatile
CACHE_REDIS_PORT=6379
CACHE_REDIS_URL=redis://redis-volatile:6379

# postgresql mastodon integration

DB_HOST=postgresql
DB_USER=mastodon
DB_NAME=mastodon_production
DB_PASS=you_will_change_this_password_ahead_on_this_doc
DB_PORT=5432

# elasticsearch mastodon integration

ES_ENABLED=true
ES_HOST=elasticsearch
ES_PORT=9200
ES_USER=elastic
ES_PASS=you_will_change_this_password_ahead_on_this_doc

7. generating secrets and passwords

7.1 secret key and OTP secret

Now we need to generate two secrets, SECRETKEYBASE and OTP_SECRET.

From the docker host run

$ docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell bundle exec rake secret

Yes, you need to run two times, one for each secret and then update the application.env file.

7.2 generate vapid secrets

Now we need to generate the VAPID secrets, VAPIDPRIVATEKEY and VAPIDPUBLICKEY.

From the docker host run

$ docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell bundle exec rake mastodon:webpush:generate_vapid_key

Get the output and append the file application.env

7.3 generate elasticsearch password

Generate a password for the var ELASTIC_PASSWORD

$ openssl rand -hex 15

Update the database.env file, don't forget to update ES_PASS, it's the same password.

7.4 generate postgresql password

Generate a password for the var POSTGRES_PASSWORD

$ openssl rand -hex 15

Update the database.env file.

8. configuring a local nginx to serve mastodon

Here we'll configure an NGINX on the same docker host of our mastodon.

8.1 extracting the static files for nginx cache

This procedure is only used by NGINX for CACHE purposes, mastodon do not use this static files, it's used by NGINX, and NGINX Only.

let's create a docker volume in YOUR NGINX SERVER to store the static mastodon files

$ mkdir -p /opt/www/mastodon/dev.bolha.us/public
$ chown -R mastodon:mastodon /opt/www/mastodon
$ docker volume create --opt type=none --opt device=/opt/www/mastodon/dev.bolha.us/public --opt o=bind mastodon_public_dev-bolha-us_temp_files

now let's copy the files from the container to the device

$ docker run --rm -v "mastodon_public_dev-bolha-us_temp_files:/static" tootsuite/mastodon:v4.1.2 bash -c "cp -r /opt/mastodon/public/* /static/"

nice, check if the files were copied properly

$ ls /opt/www/mastodon/dev.bolha.us/public

output expected

500.html  avatars    embed.js  favicon.ico  inert.css  oops.gif  packs       sounds  sw.js.map                 web-push-icon_favourite.png
assets    badge.png  emoji     headers      ocr        oops.png  robots.txt  sw.js   web-push-icon_expand.png  web-push-icon_reblog.png

now we can remove our docker volume; we only needed the volume for the copy.

$ docker volume rm mastodon_public_dev-bolha-us_temp_files

ok, now we have a directory to be used by nginx cache to serve the static files.

8.2 Creating directories

$ mkdir -p /opt/nginx/{docker,html,vhost,conf,stream,certbot,cache,logs}
$ mkdir -p /opt/nginx/certbot/{html,conf}
$ mkdir -p /opt/nginx/cache/public/4.1.2

8.3 Creating nginx.conf

Let's create our nginx.conf

$ vim /opt/nginx/conf/nginx.conf

The contents of the file

user nginx;
worker_processes auto;

pid        /var/run/nginx.pid;

include /etc/nginx/conf.d/*.conf;

events {
	worker_connections 2048;
}

http {

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;

	keepalive_timeout 65;
	types_hash_max_size 2048;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;
	
	log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	gzip on;

	include /etc/nginx/vhosts/*.conf;
	include /etc/nginx/sites-enabled/*;

}

stream {

 log_format proxy '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time "$upstream_addr" '
                 '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time" "$upstream_addr"';

   include /etc/nginx/stream/*.conf;

}

8.4 Creating default.conf

Let's create our default.conf to avoid nginx container default configs conflict.

$ vim /opt/nginx/vhost/default.conf

The contents of the file

server {
    listen       80;
    server_name  localhost;

    location /.well-known/acme-challenge/ {
      root /var/www/certbot;
    }
  
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

}

8.5 Creating dev.bolha.us.conf

Now we can create the config of our mastodon instance; we will use dev.bolha.us as our domain, just as an example.

$ vim /opt/nginx/vhost/dev-bolha-us.conf

With this content

# connection configuration

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

# upstream configuration

upstream backend {
    server localhost:3000 fail_timeout=0;
}

upstream streaming {
    server localhost:4000 fail_timeout=0;
}

# cache for static files

proxy_cache_path /var/cache/mastodon/public/4.1.2 levels=1:2 keys_zone=MASTODON_CACHE_v412:10m inactive=7d max_size=3g;

# server configuration

server {
  listen 80;
  server_name dev.bolha.us;

  location /.well-known/acme-challenge/ {
      root /var/www/certbot;
  }
  
  location / { 
    return 301 https://dev.bolha.us$request_uri; 
  }
  
}

server {
  listen 443 ssl http2;
  server_name dev.bolha.us;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers on;
  ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';

  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;

  access_log /var/log/nginx/mastodon-dev-bolha-us-access.log;
  error_log /var/log/nginx/mastodon-dev-bolha-us-error.log;

  ssl_certificate /etc/letsencrypt/live/bolha.us/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/bolha.us/privkey.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  root /var/www/mastodon/dev.bolha.us/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;

  add_header Strict-Transport-Security "max-age=31536000" always;

  location / {
    try_files $uri @proxy;
  }

  location ~ ^/(system/accounts/avatars|system/media_attachments/files) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    root /opt/mastodon/;
    try_files $uri @proxy;
  }

  location ~ ^/(emoji|packs) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
  }

  location /sw.js {
    add_header Cache-Control "public, max-age=0";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy"";
    proxy_pass_header Server;

    proxy_pass http://backend;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache MASTODON_CACHE_v412;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Strict-Transport-Security "max-age=31536000" always;

    tcp_nodelay on;
  }

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy"";

    proxy_pass http://streaming;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  error_page 500 501 502 503 504 /500.html;
}

What do you need to change here?

  • server_name dev.bolha.us;
  • return 301 https://dev.bolha.us$request_uri;
  • server_name dev.bolha.us;
  • access_log /var/log/nginx/mastodon-dev-bolha-us-access.log;
  • error_log /var/log/nginx/mastodon-dev-bolha-us-error.log;
  • ssl_certificate /etc/letsencrypt/live/bolha.us/fullchain.pem;
  • sslcertificatekey /etc/letsencrypt/live/bolha.us/privkey.pem;
  • root /var/www/mastodon/dev.bolha.us/public;

To use this config, we expect that you have it before starting your nginx

  • the directory root with the static files
  • the ssl certificates generated already

Wait and follow the instructions carefully, baby steps!

8.6 Creating the nginx docker-compose configuration

After that, we can create the docker-compose file.

cd /opt/nginx/docker
vim docker-compose.yml

here are the contents of the file

version: '3'

services:
  nginx:
    image: nginx:latest
    container_name: nginx
    restart: always
    network_mode: host
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /opt/nginx/conf/nginx.conf:/etc/nginx/nginx.conf
      - nginx_log:/var/log/nginx
      - nginx_vhost:/etc/nginx/vhosts
      - nginx_stream:/etc/nginx/stream
      - nginx_html:/usr/share/nginx/html
      - nginx_cache:/var/cache/mastodon
      - certbot_conf:/etc/letsencrypt
      - certbot_html:/var/www/certbot
      - mastodon_public:/var/www/mastodon
    healthcheck:
      test: ["CMD-SHELL", "wget -O /dev/null http://localhost || exit 1"]
      timeout: 10s
  certbot:
    image: certbot/certbot:latest
    restart: no
    container_name: certbot
    network_mode: host
    volumes:
      - certbot_conf:/etc/letsencrypt
      - certbot_html:/var/www/certbot

volumes:
  mastodon_public:
    driver_opts:
      type: none
      device: /opt/www/mastodon
      o: bind
  nginx_log:
    driver_opts:
      type: none
      device: /opt/nginx/logs
      o: bind
  nginx_cache:
    driver_opts:
      type: none
      device: /opt/nginx/cache
      o: bind
  nginx_vhost:
    driver_opts:
      type: none
      device: /opt/nginx/vhost
      o: bind
  nginx_stream:
    driver_opts:
      type: none
      device: /opt/nginx/stream
      o: bind
  nginx_html:
    driver_opts:
      type: none
      device: /opt/nginx/html
      o: bind
  certbot_conf:
    driver_opts:
      type: none
      device: /opt/nginx/certbot/conf
      o: bind
  certbot_html:
    driver_opts:
      type: none
      device: /opt/nginx/certbot/html
      o: bind

8.7 Creating the letsencrypt certificate

let's create the certificate first. It would be best to ensure that your domain points to your nginx docker server or cerbot will fail during the certificate generation.

$ cd /opt/nginx/docker
$ docker-compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot/ -d dev.bolha.us

add the command to renew to your crontab

$ crontab -e

add this command

00 3 * * * docker-compose /opt/nginx/docker/docker-compose.yml run --rm certbot renew

8.8 Starting NGINX

now let's run our nginx :)

$ cd /opt/nginx/docker
$ docker-compose -up -d

check if your nginx is running accordingly

$ docker ps
$ docker logs -f

9. Configuring an external NGINX dedicated to mastodon

Point the domain to the external nginx server.

Follow section 8 in your NGINX Server.|

The static files are only needed by NGINX, you do not need to do that part on the mastodon server.

Open the ports 3000 and 4000 TCP for the NGINX IP on your Mastodon Server.

10. Configuring an existing (external) non-docker NGINX

Follow the instructions from the sections:

  • 8.1
  • 8.5

You need to create the cache directory for nginx

proxy_cache_path /var/cache/mastodon/public/4.1.2 levels=1:2 keys_zone=MASTODON_CACHE_v412:10m inactive=7d max_size=3g;

You need to generate your certs.

You need to adjust to having it running in your setup, we don't know your structure, so we can't cover this here, sorry.

11. Starting your Mastodon (Finally, right? :)

If you get here, you're already a warrior :)

Let's do it!

Go to your mastodon server docker config directory.

$ cd /opt/mastodon/docker

11.1 pulling the images

pulling

$ docker-compose pull

output expected

 ✔ postgresql Pulled                                                                                
 ✔ redis  Pulled
 ✔ redis-volatile Skipped - Image is already being pulled by redis                                                                                                     
 ✔ elasticsearch Pulled
 ✔ website Pulled                                                                                                                                                    
 ✔ streaming Skipped - Image is already being pulled by website                                                                                                       
 ✔ sidekiq Skipped - image is already being pulled by website
✔ shell Skipped - Image is already being pulled by website

11.2 starting postgresql and redis

starting databases

$ docker-compose up -d postgresql redis redis-volatile

output expected

 ✔ Network docker_internal_network  Created                                                                                                                            0.0s
 ✔ Network docker_external_network  Created                                                                                                                            0.0s
 ✔ Volume "docker_elasticsearch"    Created                                                                                                                            0.0s
 ✔ Volume "docker_public"           Created                                                                                                                            0.0s
 ✔ Volume "docker_uploads"          Created                                                                                                                            0.0s
 ✔ Volume "docker_app"              Created                                                                                                                            0.0s
 ✔ Volume "docker_postgresql"       Created                                                                                                                            0.0s
 ✔ Volume "docker_redis"            Created                                                                                                                            0.0s
 ✔ Container mastodon_redis         Started                                                                                                                            0.5s
 ✔ Container mastodon_redis_cache   Started                                                                                                                            0.5s
 ✔ Container mastodon_postgresql    Started                                                                                                                            0.5s

11.2 running the database setup

$ docker-compose run --rm shell bundle exec rake db:setup

output expected

Database 'mastodon_production' already exists

11.3 starting remaining services

$ docker-compose up -d

output expected

+] Running 8/8
 ✔ Container mastodon_elastisearch  Started                                                                                                                                                                                                                                            1.1s
 ✔ Container mastodon_website       Started                                                                                                                            1.6s
 ✔ Container mastodon_streaming     Started                                                                                                                            1.6s
 ✔ Container mastodon_sidekiq       Started                                                                                                                            2.2s

11.4 Checking everything

let's verify our containers

root@dev:/opt/nginx/docker# docker ps

output expected

CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS                            PORTS                                NAMES
ac17eef8d384   nginx:latest "/docker-entrypoint.…" 6 seconds ago    Up 5 seconds (health: starting)                                        mastodon_nginx
adabce4171e2   tootsuite/mastodon:v4.1.2 "/usr/bin/tini -- bu…" 25 minutes ago   Up 25 minutes (healthy)           3000/tcp, 4000/tcp                   mastodon_sidekiq
47d8b9720fc4   tootsuite/mastodon:v4.1.2   "/usr/bin/tini -- ba…"   25 minutes ago   Up 25 minutes (healthy)           127.0.0.1:3000->3000/tcp, 4000/tcp   mastodon_website
f55bea899b31   tootsuite/mastodon:v4.1.2   "/usr/bin/tini -- no…"   25 minutes ago   Up 25 minutes (healthy)           3000/tcp, 127.0.0.1:4000->4000/tcp   mastodon_streaming
e2ffd239c210   redis:7 "docker-entrypoint.s…" 25 minutes ago   Up 25 minutes (healthy)                                                mastodon_redis_cache
3914fc3f784b   postgres:14 "docker-entrypoint.s…" 25 minutes ago   Up 25 minutes (healthy)                                                mastodon_postgresql
d02fffd8f108   elasticsearch:7.17.10       "/bin/tini -- /usr/l…"   25 minutes ago   Up 25 minutes (healthy)                                                mastodon_elastisearch
e500590b9a20   redis:7 "docker-entrypoint.s…" 25 minutes ago   Up 25 minutes (healthy)                                                mastodon_redis

11.5 Enabling registration with approval

$ cd /opt/mastodon/docker
$ docker-compose run --rm shell bin/tootctl settings registrations approved

12. Creating users via terminal

Creating an owner

$ docker-compose run --rm shell bin/tootctl accounts create gutocarvalho --email gutocarvalho@bolha.us --confirmed --role Owner
$ docker-compose run --rm shell bin/tootctl accounts approve gutocarvalho

Creating an admin

$ docker-compose run --rm shell bin/tootctl accounts create joseaugusto --email joseaaugusto@bolha.us --confirmed --role Admin
$ docker-compose run --rm shell bin/tootctl accounts approve joseaugusto

Creating an moderator

$ docker-compose run --rm shell bin/tootctl accounts create augustocarvalho --email augustocarvalho@bolha.us --confirmed --role Moderator
$ docker-compose run --rm shell bin/tootctl accounts approve augustocarvalho

Creating a normal user

$ docker-compose run --rm shell bin/tootctl accounts create arturcarvalho --email arturcarvalho@bolha.us --confirmed
$ docker-compose run --rm shell bin/tootctl accounts approve arturcarvalho

13. Accessing your mastodon!

Go to your mastodon!

https://dev.bolha.us

It's ready and should work, you just need to login.

14. Creating maintenance tasks using crontab

open your root crontab

$ crontab -e

indexing data of elasticsearch to create the 'https://dev.bolha.us/explore' content

00 */6 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl search deploy --concurrency 4

this will recount all accounts numbers of the instance daily

30 1 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl cache recount accounts --concurrency 4

this will force all users to follow special instance accounts

00 2 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl accounts follow status

00 2 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl accounts follow news

00 2 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl accounts follow tips

00 2 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl accounts follow gutocarvalho

this will clean media from external instances in our local cache, it will clean everything older than 15 days in the cache

30 2 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl media remove --days=15 --concurrency 4

this will clean local thumbnails for preview cards in our local cache; it will clean everything older than 15 days in the cache

00 3 * * * docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl preview_cards remove --days=15 --concurrency 4

this will regenerate all user feeds every Sunday

00 1 * * 0 docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl feeds clear
    
00 2 * * 0 docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl feeds build --concurrency 4

this will remove statuses without users/references every Sunday

00 3 * * 0 docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl statuses remove --days 60

References!

15. Maintenance tasks

Remember to add this before the command; you will use tootctl inside the mastodon_shell container

$ docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell COMMAND

Example

$ docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell tootctl accounts modify user@domain.tld --approve

I'll remove the first part of the command to make the examples easily understood.

15.1 Accounts Tasks

Reset password

$ tootctl accounts modify user@domain.tld --reset-password

Disable user

$ tootctl accounts modify user@domain.tld --disable

Approve user

$ tootctl accounts modify user@domain.tld --approve

Disable 2FA in case someone forgets the 2FA code or device

$ tootctl accounts modify user@domain.tld --disable-2fa

If you are not seeing user data from a specific user or domain

$ tootctl accounts refresh user@domain.tld

Delete

$ tootctl accounts delete user@domain.tld

15.2 Other tasks

If you are not seeing images from a specific domain

$ tootctl media refresh domain.tld --concurrency 4

Remove all accounts from a given DOMAIN without leaving behind any records. Unlike a suspension, if the DOMAIN still exists in the wild, it means the accounts could return if they are resolved again.

$ tootctl domains purge domain.tld --concurrency 4

Remove remote accounts that no longer exist. Queries every single remote account in the database to determine if it still exists on the origin server, and if it doesn't, then remove it from the database.

Accounts with confirmed activity within the last week are excluded from the checks, in case the server is down.

$ tootctl accounts cull domain.tld --concurrency 4

16. Backup

You can use whatever you want to protect your data; we'll focus on what you need to do before the backup and what you need to backup to have your data replicated to a safe location.

16.1 Cold Backup

  1. stop all containers

  2. run a backup of your /opt/mastodon, /opt/nginx, and /opt/www

16.2 Hot Backup

  1. run a backup of your postgresql

  2. run a backup of your /opt/mastodon/data/web directory

  3. run a backup of your /opt/mastodon/docker directory

  4. run a backup of your /opt/nginx directory

16.3 Our Backup Provider

Our automation sends our backup to blackblaze object storage, it's a good and cheap provider; we do full backups of pgsql and configs every day.

17. API Information

You cant get some info from the API using curl and jq

$ curl -s https://dev.bolha.us/api/v1/instance | jq
$ curl -s https://dev.bolha.us/api/v2/instance | jq

Let's create a script to get some information from mastodon:

    $ vim mastodon_api_info.sh

Script Content

#!/bin/bash

INSTANCE_ENDPOINT_V1="https://dev.bolha.us/api/v1/instance"
INSTANCE_ENDPOINT_V2="https://dev.bolha.us/api/v2/instance"
INSTANCE_URL="https://dev.bolha.us"

COUNT_TOTAL_USERS=$(curl -s $INSTANCE_ENDPOINT_V1 | jq '.stats.user_count')
COUNT_TOTAL_STATUS=$(curl -s $INSTANCE_ENDPOINT_V1 | jq '.stats.status_count')
COUNT_TOTAL_DOMAINS=$(curl -s $INSTANCE_ENDPOINT_V1 | jq '.stats.domain_count')
COUNT_ACTIVE_USERS=$(curl -s $INSTANCE_ENDPOINT_V2 | jq '.usage.users.active_month')
COUNT_POOL_LIMIT=$(curl -s $INSTANCE_ENDPOINT_V2 | jq '.configuration.polls.max_options')
COUNT_CHAR_LIMIT=$(curl -s $INSTANCE_ENDPOINT_V2 | jq '.configuration.statuses.max_characters')
INSTANCE_VERSION=$(curl -s $INSTANCE_ENDPOINT_V2 | jq '.version')

echo "Number of total registered users: $COUNT_TOTAL_USERS"
echo "Number of total statuses: $COUNT_TOTAL_STATUS"
echo "Number of total known domains: $COUNT_TOTAL_DOMAINS"
echo "Number of active users (this month): $COUNT_ACTIVE_USERS"
echo "Number of pool options: $COUNT_POOL_LIMIT"
echo "Number of char limit: $COUNT_CHAR_LIMIT"
echo "Mastodon instance version: $INSTANCE_VERSION"

Execute

$ bash mastodon_api_info.sh

Expected output (from bolha.us)

Number of total registered users: 1464
Number of total statuses: 51191
Number of total known domains: 16237
Number of active users (this month): 618
Number of pool options: 4
Number of char limit: 500
Mastodon instance version: "4.1.2"

:)

18. Upgrade

This process will cover the upgrade for minor versions only.

4.1.x to 4.1.y for example.

Here we'll cover the scenario where mastodon and nginx run on the same server.

18.1 pre-upgrade

  1. run a backup of your postgresql

  2. run a backup of your /opt/mastodon/data/web/system directory

  3. run a backup of your /opt/mastodon/docker directory

  4. run a backup of your /opt/nginx directory

  5. stop mastodon service

  6. stop nginx services

18.2 upgrading

  1. Clear the directory /opt/www/mastodon/dev.bolha.us/public/

  2. Repeat section 8.1 using the new docker image version on the command line to copy the files.

....
....
....

$ docker run --rm -v "mastodon_public_dev-bolha-us_temp_files:/static" tootsuite/mastodon:v4.1.NEW_VERSION_HERE bash -c "cp -r /opt/mastodon/public/* /static/"

....
....
....

Follow section 8.1 properly. It's essential to have a solid upgrade.

  1. Create a directory for the new mastodon version /opt/nginx/cache/public/4.1.x

  2. Update the line proxycachepath with the new directory on the file /opt/nginx/vhost/via.bolha.us.conf.

  3. Start nginx

$ cd /opt/nginx/docker
$ docker-compose up -d

  1. Update the file /opt/mastodon/docker/versions.env with the new mastodon version.

  2. Start all mastodon services

$ cd /opt/mastodon/docker
$ docker-compose up -d
  1. Run the database migration
$ docker-compose -f /opt/mastodon/docker/docker-compose.yml run --rm shell bundle exec rake db:migrate

That's it!

:)

19. Final notes

I hope you can use this doc to configure your own instance like we did.

I wish to find something like this one year ago when I started my studies in the fediverse tools and Mastodon. :)

If you need assistance, you can reach me on the matrix @gutocarvalho@bolha.chat or mastodon @gutocarvalho@bolha.us.

:)

This post inspired our post:

Their post inspired the work, and we tried to expand it the best we could.

The first version of bolha.us followed their instructions, and this new post is just to keep the information flowing and updated.

I want to thank the “sleeplessbeastie.eu” team for the excellent work; we are running in prod because of you :)

20. Next posts

  1. Enabling Object Storage using Wasabi

  2. Individual Sidekiqs Containers For Queues

  3. LibreTranslate installation and integration

  4. Observability with Prometheus + Grafana

  5. Mastodon With TOR

About us

Bolha Means Bubble.

We're a collective of Open Source/DevOps/CoudNative IT people from Brazil :)

Visit our site https://bolha.io

Visit our Mastodon Instance https://bolha.us

Visit our PixelFed Instance https://bolha.photos

Visit our Matrix Instance

Visit our Lemmy Instance https://bolha.social

Visit our Jitsi: https://video.bolha.tools

Visit our WriteFreely: https://bolha.blog

Visit our LibreTranslate: https://libretranslate.bolha.tools

If you want, you can thank me via mastodon!

@gutocarvalho@bolha.us

Cheers!

Sabia que o Ilustre Andre Noel tem conta na bolha?

Segue lá

Veja o vídeo do canal em que ele fala da Bolha :)

Vista o site dele também :)

Mãeeeeee, estamos no Youtube!

:P

Pessoal, nosso antigo blog ghost que rodava no endereço hub.bolha.us foi desligado.

Agora o blog oficial é o @bolhaverso@bolha.blog :)

https://bolha.blog/bolhaverso

O antigo redireciona para cá!

[s] Guto

Subimos o Mobilizon mas eu não achei que o projeto está maduro o suficiente para manter, no futuro vamos revisitar.

Post apenas para registro!

:)

Video

Instalamos o Jitsi, uma ferramenta de vídeo conferência opensource

Alternativa ao Google Meet, Teams e Zoom.

Notes

Instalamos o HedgeDoc, um editor web para textos em formato markdown. Ele oferece edição em tempo real colaborativa de múltiplos usuários.

Parece um Google Docs para Markdown.

O foco é texto puro.

Tube

Instalamos o Invidious, um frontend alternativo para o Youtube.

Translator

Instalamos o Lingva, um frontend alternativo para o Google Translator.

[s] Guto

![]()

Devido a um teste que fiz antes de habilitar o HTTPS algumas coisas deram errado, mas foi devidamente resolvido.

Rodei o comando

tootctl domain purge bolha.blog

No mastodon da bolha e depois interagi e segui novamente as contas do bolha.blog.

Aparentemente tá zero.

:)

![]()

Ainda ajustando aqui e ali, a entrega de mensagens do mastodon pra cá parece que não está ok, mas vamos ver.

@gutocarvalho@bolha.us :)

Se liga agora temos duas novas contas com informações bacanas!

@status@bolha.us @backup@bolha.us

@backup

Essa conta vai trazer informações do backup diário de todas as ferramentas do bolhaverso.

@status

Essa conta vai trazer estatísticas da instância mastodon bolha.us 2x ao dia.

[s] Guto

Agora a bolha tem um nova ferramenta do fediverso :)

O WriteFreely é uma ferramenta para criação do BLOG, tem uma ideia similar do wordpress/medium, mas o foco está mais no conteúdo do que no estilo.

O WriteFreely é o projeto upstream da rede social https://write.as.

Como ele tem suporte ao protocolo ActivityPub, consegue interagir com outras ferramentas do Fediverso, tal qual o mastodon.

Acesse já

Cada usuário pode ter até 8 blogs :)

Sobre os projetos.

Siga os meus blogs!

@gutocarvalho@bolha.blog @mindnotes@bolha.blog @prontofalei@bolha.blog @bolhaverso@bolha.blog

Ainda estamos em Beta, reporte qualquer comportamento inesperado.

[s] Guto