Simon Kuehn
371213dbbb
- Remove Ollama as AI backend (never used); Mistral is primary throughout - Add infrastructure.md: full Docker setup, queue architecture, prod deployment steps, env vars — prod has no staging ERP container Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
546 lines
16 KiB
Markdown
546 lines
16 KiB
Markdown
# SuperSeller3000 — Infrastruktur & Deployment
|
||
|
||
**Zuletzt aktualisiert:** 2026-05-19
|
||
**Zielumgebung:** VPS (Ubuntu/Debian), Docker Compose
|
||
**Domain:** `ss3k.schaunwama.de`
|
||
|
||
> Dieses Dokument beschreibt den vollständigen Stand der Docker-Infrastruktur und alle Schritte, um das System auf einem neuen Server hochzuziehen.
|
||
|
||
---
|
||
|
||
## 1. Übersicht der Services
|
||
|
||
| Service | Image / Build | Zweck |
|
||
|---|---|---|
|
||
| `app` | Build `docker/app/Dockerfile` | PHP 8.4-FPM — Symfony-Applikation |
|
||
| `caddy` | `caddy:2-alpine` | Reverse Proxy, Auto-HTTPS (Let's Encrypt) |
|
||
| `postgres` | `postgres:17-alpine` | Datenbank (Schemas: `app`, `logs`, `logs_archive`) |
|
||
| `redis` | `redis:7-alpine` | Queue-Backend + Session-Cache |
|
||
| `worker-ai` | Build `docker/app/Dockerfile` | Messenger Worker — Transport `ai_pipeline` |
|
||
| `worker-orders` | Build `docker/app/Dockerfile` | Messenger Worker — Transport `orders` |
|
||
| `worker-channel` | Build `docker/app/Dockerfile` | Messenger Worker — Transport `channel_sync` |
|
||
| `cron` | Build `docker/app/Dockerfile` | Log-Rotation täglich (`app:logs:rotate`) |
|
||
|
||
**Nicht im Prod-Setup:** kein Staging-ERP-Container. Frappe ERP läuft extern (eigene Instanz, `FRAPPE_ERP_BASE_URL` in `.env.local`).
|
||
|
||
---
|
||
|
||
## 2. Queue-Architektur (Symfony Messenger + Redis Streams)
|
||
|
||
Drei isolierte Redis-Streams — ein ausgefallener Worker blockiert die anderen nie.
|
||
|
||
### Transports & Retry-Strategie
|
||
|
||
| Transport | Redis Stream | Worker | max_retries | Delay / Backoff |
|
||
|---|---|---|---|---|
|
||
| `ai_pipeline` | `ai_pipeline` | `worker-ai` | 3 | 2 s · ×2 |
|
||
| `orders` | `orders` | `worker-orders` | 5 | 1 s · ×2 |
|
||
| `channel_sync` | `channel_sync` | `worker-channel` | 5 | 2 s · ×2 · max 60 s |
|
||
| `failed` | `failed` | — | — | persistent, nie verloren |
|
||
|
||
Nach Erschöpfung aller Retries landet die Message im `failed`-Transport. Replay: `messenger:failed:retry`.
|
||
|
||
### Messages pro Transport
|
||
|
||
#### `ai_pipeline` → `worker-ai`
|
||
|
||
Die Pipeline-Schritte werden sequenziell als separate Messages dispatcht — jeder Handler dispatcht den Nachfolger.
|
||
|
||
| Message | Handler-Logik |
|
||
|---|---|
|
||
| `PhotoUploadMessage` | Startet Pipeline A: `VisionAgent` (Mistral Pixtral) liest Typenschild → Hersteller, Modell, SN |
|
||
| `SpecsResearchMessage` | `SpecsResearchAgent`: Tavily-Suche mit Modellbezeichnung → Specs-Freitext; Pflichtfelder aus `ArticleType.AttributeDefinitions` (`{{fields}}`-Platzhalter im Prompt) |
|
||
| `JsonCodingMessage` | `JsonCodingAgent`: Specs-Text → strukturiertes JSON gegen ArticleType-Schema |
|
||
| `ValidationMessage` | Pflichtfelder vollständig? → weiter; sonst Retry ab `JsonCodingMessage` (max. 3×, `missing_fields` im Prompt) → `needs_review` |
|
||
| `DraftArticleMessage` | Article anlegen (status: `draft`), Inventurnummer vergeben, Attribute + Foto persistieren |
|
||
| `EbayTextMessage` | `EbayTextAgent`: eBay-Titel + Beschreibung aus Attributen generieren, an Article speichern |
|
||
| `PxeInventoryMessage` | Startet Pipeline B (PXE): PXE-Dump direkt an `JsonCodingAgent` — SpecsResearch entfällt |
|
||
|
||
**Model-Cache:** Nach `PhotoUploadMessage` prüft `findCompletedByModelNumber()` die DB. Treffer → alle Daten kopieren, `SpecsResearch`/`JsonCoding`/`EbayText` überspringen.
|
||
**Fehlerfall:** `PipelineJobFailureListener` fängt `WorkerMessageFailedEvent` ab und setzt `AIPipelineJob.status = failed` mit dem echten Fehlertext.
|
||
|
||
#### `orders` → `worker-orders`
|
||
|
||
| Message | Handler-Logik |
|
||
|---|---|
|
||
| `OrderReceivedMessage` | Vollständiger 13-Schritt-Order-Flow: Idempotenz-Check → atomarer Inventory-Lock (`stock - 1`) → `CustomerResolver` (Matching-Kaskade) → Order anlegen → Frappe Sales Invoice → PDF abrufen → `StorageManager` → Invoice-Record → PDF per SMTP an Lieferant → `UpdateStockOnChannelsMessage` oder `DeactivateListingMessage` dispatchen → Order: `completed` |
|
||
|
||
#### `channel_sync` → `worker-channel`
|
||
|
||
| Message | Handler-Logik |
|
||
|---|---|
|
||
| `PublishToChannelMessage` | `ChannelAdapterRegistry` → richtiger Adapter (z.B. `EbayAdapter`) → `publishListing()` → `ebay_listing_id` an Article zurückschreiben |
|
||
| `UpdateStockOnChannelsMessage` | `updateStock()` auf allen aktiven Plattformen des Artikels |
|
||
| `DeactivateListingMessage` | `deactivateListing()` — wird ausgelöst wenn `stock = 0` nach Verkauf |
|
||
| `TrackingPushMessage` | `pushTracking()` → Plattform als versandt markieren → `tracking_pushed_to_ebay_at` setzen |
|
||
|
||
### AI-Backend-Ausfall
|
||
|
||
AI-Worker-Messages verbleiben in der Queue. Messenger wiederholt mit Backoff, nach 3 Versuchen → `failed`-Transport. Operativer Betrieb (Verkäufe, Rechnungen, Bestand) läuft völlig unabhängig durch. Replay via `messenger:failed:retry` wenn das AI-Backend wieder erreichbar ist.
|
||
|
||
---
|
||
|
||
## 3. Netzwerk
|
||
|
||
Alle Services kommunizieren über ein internes Docker-Bridge-Network. Nur Caddy exponiert Ports nach außen.
|
||
|
||
```yaml
|
||
networks:
|
||
default:
|
||
driver: bridge
|
||
ipam:
|
||
config:
|
||
- subnet: 172.18.0.0/24
|
||
gateway: 172.18.0.1
|
||
```
|
||
|
||
Das Gateway `172.18.0.1` ist die Host-Adresse innerhalb des Docker-Netzes (erreichbar von allen Containern).
|
||
|
||
---
|
||
|
||
## 4. docker-compose.yml (Prod)
|
||
|
||
```yaml
|
||
services:
|
||
app:
|
||
build:
|
||
context: .
|
||
dockerfile: docker/app/Dockerfile
|
||
user: "1000:1000"
|
||
environment:
|
||
HOME: /tmp
|
||
volumes:
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
- ./docker/app/zz-fpm-pool.conf:/usr/local/etc/php-fpm.d/zzz-pool.conf:ro
|
||
depends_on:
|
||
postgres:
|
||
condition: service_healthy
|
||
redis:
|
||
condition: service_healthy
|
||
env_file:
|
||
- path: .env
|
||
required: true
|
||
- path: .env.local
|
||
required: false
|
||
|
||
caddy:
|
||
image: caddy:2-alpine
|
||
ports:
|
||
- "80:80"
|
||
- "443:443"
|
||
volumes:
|
||
- ./docker/caddy/Caddyfile:/etc/caddy/Caddyfile
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
- caddy_data:/data
|
||
depends_on:
|
||
- app
|
||
|
||
postgres:
|
||
image: postgres:17-alpine
|
||
env_file: .env
|
||
volumes:
|
||
- postgres_data:/var/lib/postgresql/data
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
||
interval: 5s
|
||
timeout: 5s
|
||
retries: 5
|
||
|
||
redis:
|
||
image: redis:7-alpine
|
||
command: redis-server --requirepass ${REDIS_PASSWORD}
|
||
volumes:
|
||
- redis_data:/data
|
||
healthcheck:
|
||
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
|
||
interval: 5s
|
||
timeout: 5s
|
||
retries: 5
|
||
|
||
worker-ai:
|
||
build:
|
||
context: .
|
||
dockerfile: docker/app/Dockerfile
|
||
user: "1000:1000"
|
||
environment:
|
||
HOME: /tmp
|
||
command: php bin/console messenger:consume ai_pipeline --time-limit=3600 --memory-limit=256M
|
||
volumes:
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
depends_on:
|
||
- postgres
|
||
- redis
|
||
env_file:
|
||
- path: .env
|
||
required: true
|
||
- path: .env.local
|
||
required: false
|
||
restart: unless-stopped
|
||
|
||
worker-orders:
|
||
build:
|
||
context: .
|
||
dockerfile: docker/app/Dockerfile
|
||
user: "1000:1000"
|
||
environment:
|
||
HOME: /tmp
|
||
command: php bin/console messenger:consume orders --time-limit=3600 --memory-limit=256M
|
||
volumes:
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
depends_on:
|
||
- postgres
|
||
- redis
|
||
env_file:
|
||
- path: .env
|
||
required: true
|
||
- path: .env.local
|
||
required: false
|
||
restart: unless-stopped
|
||
|
||
worker-channel:
|
||
build:
|
||
context: .
|
||
dockerfile: docker/app/Dockerfile
|
||
user: "1000:1000"
|
||
environment:
|
||
HOME: /tmp
|
||
command: php bin/console messenger:consume channel_sync --time-limit=3600 --memory-limit=256M
|
||
volumes:
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
depends_on:
|
||
- postgres
|
||
- redis
|
||
env_file:
|
||
- path: .env
|
||
required: true
|
||
- path: .env.local
|
||
required: false
|
||
restart: unless-stopped
|
||
|
||
cron:
|
||
build:
|
||
context: .
|
||
dockerfile: docker/app/Dockerfile
|
||
user: "1000:1000"
|
||
environment:
|
||
HOME: /tmp
|
||
command: >
|
||
sh -c "while true; do
|
||
php bin/console app:logs:rotate;
|
||
sleep 86400;
|
||
done"
|
||
volumes:
|
||
- .:/var/www
|
||
- ./docker/app/php.ini:/usr/local/etc/php/conf.d/app.ini:ro
|
||
depends_on:
|
||
- postgres
|
||
env_file:
|
||
- path: .env
|
||
required: true
|
||
- path: .env.local
|
||
required: false
|
||
restart: unless-stopped
|
||
|
||
volumes:
|
||
postgres_data:
|
||
redis_data:
|
||
caddy_data:
|
||
|
||
networks:
|
||
default:
|
||
driver: bridge
|
||
ipam:
|
||
config:
|
||
- subnet: 172.18.0.0/24
|
||
gateway: 172.18.0.1
|
||
```
|
||
|
||
**Hinweis:** `docker-compose.override.yml` existiert nur für lokale Entwicklung (Port-Bindings für Postgres/Redis zum direkten Zugriff vom Host). Auf dem Prod-Server nicht verwenden.
|
||
|
||
---
|
||
|
||
## 4. Dockerfile — `docker/app/Dockerfile`
|
||
|
||
```dockerfile
|
||
FROM php:8.4-fpm-alpine
|
||
|
||
RUN apk add --no-cache \
|
||
postgresql-dev \
|
||
icu-dev \
|
||
libzip-dev \
|
||
unzip \
|
||
git \
|
||
$PHPIZE_DEPS \
|
||
&& docker-php-ext-install \
|
||
pdo_pgsql \
|
||
intl \
|
||
zip \
|
||
opcache \
|
||
&& pecl install redis \
|
||
&& docker-php-ext-enable redis \
|
||
&& apk del $PHPIZE_DEPS
|
||
|
||
COPY --from=composer:2 /usr/bin/composer /usr/bin/composer
|
||
|
||
WORKDIR /var/www
|
||
|
||
COPY docker/app/php.ini /usr/local/etc/php/conf.d/app.ini
|
||
```
|
||
|
||
---
|
||
|
||
## 5. PHP-Konfiguration
|
||
|
||
### `docker/app/php.ini`
|
||
|
||
```ini
|
||
opcache.enable=0
|
||
memory_limit=256M
|
||
upload_max_filesize=20M
|
||
post_max_size=20M
|
||
```
|
||
|
||
`opcache.enable=0` weil der Dev-Build keinen Preload hat — in Prod kann auf `1` + `opcache.preload` umgestellt werden, wenn gewünscht.
|
||
|
||
### `docker/app/zz-fpm-pool.conf`
|
||
|
||
```ini
|
||
[www]
|
||
; SSE connections hold a worker for up to 90 s each — raise the pool ceiling
|
||
; so regular requests are not starved.
|
||
pm = dynamic
|
||
pm.max_children = 30
|
||
pm.start_servers = 4
|
||
pm.min_spare_servers = 2
|
||
pm.max_spare_servers = 8
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Caddyfile (Prod) — `docker/caddy/Caddyfile`
|
||
|
||
```
|
||
{
|
||
admin off
|
||
}
|
||
|
||
ss3k.schaunwama.de {
|
||
root * /var/www/public
|
||
php_fastcgi app:9000
|
||
file_server
|
||
encode gzip
|
||
|
||
header {
|
||
Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||
X-Content-Type-Options "nosniff"
|
||
X-Frame-Options "DENY"
|
||
}
|
||
}
|
||
```
|
||
|
||
**Prod enthält keinen `erpstaging`-Block.** Frappe ERP läuft auf einer eigenen Instanz (extern, `FRAPPE_ERP_BASE_URL` in `.env.local`).
|
||
|
||
> Aktueller Dev-Stand hat noch den `erpstaging.schaunwama.de`-Block für die Staging-ERP-Weiterleitung — der muss vor dem ersten Prod-Deploy entfernt werden.
|
||
|
||
---
|
||
|
||
## 7. Umgebungsvariablen
|
||
|
||
### `.env` (ohne Credentials, in Git)
|
||
|
||
```dotenv
|
||
APP_ENV=prod
|
||
APP_SECRET=change_me_in_env_local
|
||
POSTGRES_DB=superseller
|
||
POSTGRES_USER=superseller
|
||
POSTGRES_PASSWORD=change_me
|
||
DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?serverVersion=17&charset=utf8"
|
||
REDIS_PASSWORD=change_me
|
||
REDIS_URL=redis://:${REDIS_PASSWORD}@redis:6379
|
||
MESSENGER_TRANSPORT_DSN=redis://:${REDIS_PASSWORD}@redis:6379/messages
|
||
MAILER_DSN=smtp://localhost
|
||
TAVILY_API_KEY=
|
||
MISTRAL_BASE_URL=https://api.mistral.ai
|
||
MISTRAL_API_KEY=
|
||
MISTRAL_VISION_MODEL=pixtral-12b-2409
|
||
MISTRAL_TEXT_MODEL=mistral-large-latest
|
||
AI_TEXT_MODEL=${MISTRAL_TEXT_MODEL}
|
||
AI_VISION_MODEL=${MISTRAL_VISION_MODEL}
|
||
EBAY_CLIENT_ID=
|
||
EBAY_CLIENT_SECRET=
|
||
EBAY_MARKETPLACE_ID=EBAY_DE
|
||
EBAY_API_BASE_URL=https://api.ebay.com
|
||
EBAY_OAUTH_BASE_URL=https://api.ebay.com
|
||
EBAY_VERIFICATION_TOKEN=
|
||
EBAY_ENDPOINT_URL=https://ss3k.schaunwama.de/webhooks/ebay
|
||
FRAPPE_ERP_BASE_URL=https://erp.example.com
|
||
FRAPPE_ERP_API_KEY=changeme
|
||
FRAPPE_ERP_API_SECRET=changeme
|
||
FRAPPE_GENERIC_ITEM_CODE=REFURB-HW
|
||
SUPPLIER_EMAIL=lieferant@example.com
|
||
SENDER_EMAIL=noreply@superseller3000.de
|
||
```
|
||
|
||
### `.env.local` (nur auf Server, nie in Git)
|
||
|
||
Alle echten Credentials überschreiben hier die Platzhalter aus `.env`:
|
||
|
||
```dotenv
|
||
APP_SECRET=<random-32-char-hex>
|
||
POSTGRES_PASSWORD=<sicheres-passwort>
|
||
REDIS_PASSWORD=<sicheres-passwort>
|
||
MAILER_DSN=smtp://user:pass@mailserver:587
|
||
TAVILY_API_KEY=tvly-...
|
||
MISTRAL_API_KEY=sk-...
|
||
EBAY_CLIENT_ID=...
|
||
EBAY_CLIENT_SECRET=...
|
||
EBAY_VERIFICATION_TOKEN=... # aus eBay Developer Portal
|
||
FRAPPE_ERP_BASE_URL=https://erp.meinefirma.de
|
||
FRAPPE_ERP_API_KEY=...
|
||
FRAPPE_ERP_API_SECRET=...
|
||
SUPPLIER_EMAIL=lieferant@meinefirma.de
|
||
SENDER_EMAIL=noreply@meinefirma.de
|
||
```
|
||
|
||
---
|
||
|
||
## 8. VPS-Voraussetzungen
|
||
|
||
```bash
|
||
# Docker + Compose Plugin
|
||
apt install docker.io docker-compose-plugin
|
||
|
||
# User 1000 muss existieren (PHP läuft als 1000:1000)
|
||
id -u superseller # sollte 1000 sein; ggf. anpassen
|
||
|
||
# UFW: nur Caddy-Ports + SSH
|
||
ufw allow 22/tcp
|
||
ufw allow 80/tcp
|
||
ufw allow 443/tcp
|
||
ufw enable
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Erste Inbetriebnahme (Schritt für Schritt)
|
||
|
||
```bash
|
||
# 1. Repo klonen
|
||
git clone <gitea-url>/superseller3000 /home/superseller/SuperSeller3000
|
||
cd /home/superseller/SuperSeller3000
|
||
|
||
# 2. .env.local mit echten Credentials anlegen (siehe Abschnitt 7)
|
||
nano .env.local
|
||
|
||
# 3. Caddyfile prüfen — kein erpstaging-Block!
|
||
# (docker/caddy/Caddyfile muss nur den ss3k.schaunwama.de-Block enthalten)
|
||
|
||
# 4. Images bauen + DB hochfahren
|
||
docker compose up -d postgres redis
|
||
# warten bis postgres healthy
|
||
|
||
# 5. Composer install
|
||
docker compose run --rm app composer install --no-dev --optimize-autoloader
|
||
|
||
# 6. Datenbankmigrationen ausführen
|
||
docker compose run --rm app php bin/console doctrine:migrations:migrate --no-interaction
|
||
|
||
# 7. Cache aufwärmen
|
||
docker compose run --rm app php bin/console cache:warmup --env=prod
|
||
|
||
# 8. Ersten Admin-User anlegen
|
||
docker compose run --rm app php bin/console app:users:create --env=prod
|
||
|
||
# 9. Ersten API-Key anlegen (Klartext wird nur einmal angezeigt)
|
||
docker compose run --rm app php bin/console app:api-keys:create --env=prod
|
||
|
||
# 10. Alle Services starten
|
||
docker compose up -d
|
||
|
||
# 11. eBay-Webhook registrieren (URL in .env.local: EBAY_ENDPOINT_URL)
|
||
# Im eBay Developer Portal: Notification URL = https://ss3k.schaunwama.de/webhooks/ebay
|
||
# Verification Token = EBAY_VERIFICATION_TOKEN
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Laufender Betrieb
|
||
|
||
### Worker-Status prüfen
|
||
|
||
```bash
|
||
docker compose ps
|
||
docker compose logs -f worker-ai
|
||
docker compose logs -f worker-orders
|
||
docker compose logs -f worker-channel
|
||
```
|
||
|
||
### Failed Messages (nach Erschöpfung aller Retries)
|
||
|
||
```bash
|
||
# Übersicht
|
||
docker compose exec app php bin/console messenger:failed:show
|
||
|
||
# Einzelne Message erneut versuchen
|
||
docker compose exec app php bin/console messenger:failed:retry
|
||
|
||
# Alle nochmal
|
||
docker compose exec app php bin/console messenger:failed:retry --all
|
||
```
|
||
|
||
### Updates deployen
|
||
|
||
```bash
|
||
git pull
|
||
docker compose build app worker-ai worker-orders worker-channel cron
|
||
docker compose run --rm app composer install --no-dev --optimize-autoloader
|
||
docker compose run --rm app php bin/console doctrine:migrations:migrate --no-interaction
|
||
docker compose run --rm app php bin/console cache:warmup --env=prod
|
||
docker compose up -d
|
||
```
|
||
|
||
### Logs (Admin-Panel)
|
||
|
||
`https://ss3k.schaunwama.de/admin` → **Log Entries** — Fulltext-Suche, Filter nach Level/Channel/Zeitraum.
|
||
Rotation läuft täglich automatisch (Cron-Service): Einträge > 90 Tage → `logs_archive`, danach gelöscht.
|
||
|
||
### Backup
|
||
|
||
```bash
|
||
# PostgreSQL-Dump
|
||
docker compose exec postgres pg_dump -U superseller superseller | gzip > backup_$(date +%Y%m%d).sql.gz
|
||
|
||
# Artikel-Fotos und Rechnungs-PDFs (Docker Volumes)
|
||
docker run --rm -v superseller3000_postgres_data:/data -v $(pwd):/backup \
|
||
alpine tar czf /backup/volume_postgres_$(date +%Y%m%d).tar.gz /data
|
||
```
|
||
|
||
---
|
||
|
||
## 11. AI-Konfiguration
|
||
|
||
### Primäres Backend: Mistral Cloud API
|
||
|
||
Standardkonfiguration — `config/services.yaml` bereits so ausgeliefert:
|
||
|
||
```yaml
|
||
App\Infrastructure\AI\OllamaClientInterface:
|
||
alias: App\Infrastructure\AI\MistralClient
|
||
```
|
||
|
||
`.env.local`:
|
||
```dotenv
|
||
MISTRAL_API_KEY=sk-...
|
||
# Modelle sind in .env vorbelegt:
|
||
# AI_TEXT_MODEL=${MISTRAL_TEXT_MODEL} → mistral-large-latest
|
||
# AI_VISION_MODEL=${MISTRAL_VISION_MODEL} → pixtral-12b-2409
|
||
```
|
||
|
||
Web-Suche (SpecsResearchAgent): **Tavily API** — `TAVILY_API_KEY=tvly-...` in `.env.local`.
|
||
|
||
Nach Schlüsseländerung: `docker compose exec app php bin/console cache:clear`.
|