nchebrov/AzerothCore-RealmMaster
1
0
Fork 0
mirror of https://github.com/uprightbass360/AzerothCore-RealmMaster.git synced 2026-01-13 00:58:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: comprehensive module system and database management improvements

Browse Source 
This commit introduces major enhancements to the module installation system,
database management, and configuration handling for AzerothCore deployments.

## Module System Improvements

### Module SQL Staging & Installation
- Refactor module SQL staging to properly handle AzerothCore's sql/ directory structure
- Fix SQL staging path to use correct AzerothCore format (sql/custom/db_*/*)
- Implement conditional module database importing based on enabled modules
- Add support for both cpp-modules and lua-scripts module types
- Handle rsync exit code 23 (permission warnings) gracefully during deployment

### Module Manifest & Automation
- Add automated module manifest generation via GitHub Actions workflow
- Implement Python-based module manifest updater with comprehensive validation
- Add module dependency tracking and SQL file discovery
- Support for blocked modules and module metadata management

## Database Management Enhancements

### Database Import System
- Add db-guard container for continuous database health monitoring and verification
- Implement conditional database import that skips when databases are current
- Add backup restoration and SQL staging coordination
- Support for Playerbots database (4th database) in all import operations
- Add comprehensive database health checking and status reporting

### Database Configuration
- Implement 10 new dbimport.conf settings from environment variables:
  - Database.Reconnect.Seconds/Attempts for connection reliability
  - Updates.AllowedModules for module auto-update control
  - Updates.Redundancy for data integrity checks
  - Worker/Synch thread settings for all three core databases
- Auto-apply dbimport.conf settings via auto-post-install.sh
- Add environment variable injection for db-import and db-guard containers

### Backup & Recovery
- Fix backup scheduler to prevent immediate execution on container startup
- Add backup status monitoring script with detailed reporting
- Implement backup import/export utilities
- Add database verification scripts for SQL update tracking

## User Import Directory

- Add new import/ directory for user-provided database files and configurations
- Support for custom SQL files, configuration overrides, and example templates
- Automatic import of user-provided databases and configs during initialization
- Documentation and examples for custom database imports

## Configuration & Environment

- Eliminate CLIENT_DATA_VERSION warning by adding default value syntax
- Improve CLIENT_DATA_VERSION documentation in .env.template
- Add comprehensive database import settings to .env and .env.template
- Update setup.sh to handle new configuration variables with proper defaults

## Monitoring & Debugging

- Add status dashboard with Go-based terminal UI (statusdash.go)
- Implement JSON status output (statusjson.sh) for programmatic access
- Add comprehensive database health check script
- Add repair-storage-permissions.sh utility for permission issues

## Testing & Documentation

- Add Phase 1 integration test suite for module installation verification
- Add comprehensive documentation for:
  - Database management (DATABASE_MANAGEMENT.md)
  - Module SQL analysis (AZEROTHCORE_MODULE_SQL_ANALYSIS.md)
  - Implementation mapping (IMPLEMENTATION_MAP.md)
  - SQL staging comparison and path coverage
  - Module assets and DBC file requirements
- Update SCRIPTS.md, ADVANCED.md, and troubleshooting documentation
- Update references from database-import/ to import/ directory

## Breaking Changes

- Renamed database-import/ directory to import/ for clarity
- Module SQL files now staged to AzerothCore-compatible paths
- db-guard container now required for proper database lifecycle management

## Bug Fixes

- Fix module SQL staging directory structure for AzerothCore compatibility
- Handle rsync exit code 23 gracefully during deployments
- Prevent backup from running immediately on container startup
- Correct SQL staging paths for proper module installation
This commit is contained in:
uprightbass360
2025-11-20 18:26:00 -05:00
committed by Deckard
parent 0d83f01995
commit e6231bb4a4
56 changed files with 11298 additions and 487 deletions
Download Patch File Download Diff File Expand all files Collapse all files

204
docker-compose.yml
View File

@@ -23,7 +23,7 @@ services:
- /usr/local/bin/mysql-entrypoint.sh
volumes:
- ./scripts/bash/mysql-entrypoint.sh:/usr/local/bin/mysql-entrypoint.sh:ro
- ${STORAGE_PATH_LOCAL}/mysql-data:/var/lib/mysql-persistent
- mysql-data:/var/lib/mysql-persistent
- ${BACKUP_PATH}:/backups
- ${HOST_ZONEINFO_PATH}:/usr/share/zoneinfo:ro
- ${MYSQL_CONFIG_DIR:-${STORAGE_PATH}/config/mysql/conf.d}:/etc/mysql/conf.d
@@ -65,9 +65,13 @@ services:
volumes:
- ${STORAGE_PATH}/config:/azerothcore/env/dist/etc
- ${STORAGE_PATH}/logs:/azerothcore/logs
- ${STORAGE_PATH_LOCAL}/mysql-data:/var/lib/mysql-persistent
- ${AC_SQL_SOURCE_PATH:-${STORAGE_PATH_LOCAL}/source/azerothcore-playerbots/data/sql}:/azerothcore/data/sql:ro
- ${MODULE_SQL_STAGE_PATH:-${STORAGE_PATH}/module-sql-updates}:/modules-sql
- mysql-data:/var/lib/mysql-persistent
- ${STORAGE_PATH}/modules:/modules
- ${BACKUP_PATH}:/backups
- ./scripts/bash/db-import-conditional.sh:/tmp/db-import-conditional.sh:ro
- ./scripts/bash/restore-and-stage.sh:/tmp/restore-and-stage.sh:ro
environment:
AC_DATA_DIR: "/azerothcore/data"
AC_LOGS_DIR: "/azerothcore/logs"
@@ -89,6 +93,16 @@ services:
DB_WORLD_NAME: ${DB_WORLD_NAME}
DB_CHARACTERS_NAME: ${DB_CHARACTERS_NAME}
CONTAINER_USER: ${CONTAINER_USER}
DB_RECONNECT_SECONDS: ${DB_RECONNECT_SECONDS}
DB_RECONNECT_ATTEMPTS: ${DB_RECONNECT_ATTEMPTS}
DB_UPDATES_ALLOWED_MODULES: ${DB_UPDATES_ALLOWED_MODULES}
DB_UPDATES_REDUNDANCY: ${DB_UPDATES_REDUNDANCY}
DB_LOGIN_WORKER_THREADS: ${DB_LOGIN_WORKER_THREADS}
DB_WORLD_WORKER_THREADS: ${DB_WORLD_WORKER_THREADS}
DB_CHARACTER_WORKER_THREADS: ${DB_CHARACTER_WORKER_THREADS}
DB_LOGIN_SYNCH_THREADS: ${DB_LOGIN_SYNCH_THREADS}
DB_WORLD_SYNCH_THREADS: ${DB_WORLD_SYNCH_THREADS}
DB_CHARACTER_SYNCH_THREADS: ${DB_CHARACTER_SYNCH_THREADS}
entrypoint:
- sh
- -c
@@ -97,6 +111,81 @@ services:
/tmp/db-import-conditional.sh
restart: "no"
ac-db-guard:
profiles: ["db"]
image: ${AC_DB_IMPORT_IMAGE}
container_name: ${CONTAINER_DB_GUARD}
user: "${CONTAINER_USER}"
userns_mode: "keep-id"
depends_on:
ac-mysql:
condition: service_healthy
ac-storage-init:
condition: service_completed_successfully
ac-db-import:
condition: service_completed_successfully
networks:
- azerothcore
volumes:
- ${STORAGE_PATH}/config:/azerothcore/env/dist/etc
- ${STORAGE_PATH}/logs:/azerothcore/logs
- ${AC_SQL_SOURCE_PATH:-${STORAGE_PATH_LOCAL}/source/azerothcore-playerbots/data/sql}:/azerothcore/data/sql:ro
- ${MODULE_SQL_STAGE_PATH:-${STORAGE_PATH}/module-sql-updates}:/modules-sql
- mysql-data:/var/lib/mysql-persistent
- ${STORAGE_PATH}/modules:/modules
- ${BACKUP_PATH}:/backups
- ./scripts/bash/db-import-conditional.sh:/tmp/db-import-conditional.sh:ro
- ./scripts/bash/restore-and-stage.sh:/tmp/restore-and-stage.sh:ro
- ./scripts/bash/db-guard.sh:/tmp/db-guard.sh:ro
environment:
AC_DATA_DIR: "/azerothcore/data"
AC_LOGS_DIR: "/azerothcore/logs"
AC_LOGIN_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_AUTH_NAME}"
AC_WORLD_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_WORLD_NAME}"
AC_CHARACTER_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_CHARACTERS_NAME}"
CONTAINER_MYSQL: ${CONTAINER_MYSQL}
MYSQL_PORT: ${MYSQL_PORT}
MYSQL_USER: ${MYSQL_USER}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
DB_AUTH_NAME: ${DB_AUTH_NAME}
DB_WORLD_NAME: ${DB_WORLD_NAME}
DB_CHARACTERS_NAME: ${DB_CHARACTERS_NAME}
DB_PLAYERBOTS_NAME: ${DB_PLAYERBOTS_NAME}
DB_GUARD_RECHECK_SECONDS: ${DB_GUARD_RECHECK_SECONDS}
DB_GUARD_RETRY_SECONDS: ${DB_GUARD_RETRY_SECONDS}
DB_GUARD_WAIT_ATTEMPTS: ${DB_GUARD_WAIT_ATTEMPTS}
DB_RECONNECT_SECONDS: ${DB_RECONNECT_SECONDS}
DB_RECONNECT_ATTEMPTS: ${DB_RECONNECT_ATTEMPTS}
DB_UPDATES_ALLOWED_MODULES: ${DB_UPDATES_ALLOWED_MODULES}
DB_UPDATES_REDUNDANCY: ${DB_UPDATES_REDUNDANCY}
DB_LOGIN_WORKER_THREADS: ${DB_LOGIN_WORKER_THREADS}
DB_WORLD_WORKER_THREADS: ${DB_WORLD_WORKER_THREADS}
DB_CHARACTER_WORKER_THREADS: ${DB_CHARACTER_WORKER_THREADS}
DB_LOGIN_SYNCH_THREADS: ${DB_LOGIN_SYNCH_THREADS}
DB_WORLD_SYNCH_THREADS: ${DB_WORLD_SYNCH_THREADS}
DB_CHARACTER_SYNCH_THREADS: ${DB_CHARACTER_SYNCH_THREADS}
entrypoint:
- /bin/bash
- -c
- |
chmod +x /tmp/db-import-conditional.sh /tmp/restore-and-stage.sh 2>/dev/null || true
exec /bin/bash /tmp/db-guard.sh
restart: unless-stopped
healthcheck:
test:
- "CMD"
- "sh"
- "-c"
- >
file=/tmp/db-guard.ready;
[ -f "$${file}" ] || exit 1;
now=$$(date +%s);
mod=$$(stat -c %Y "$${file}" 2>/dev/null) || exit 1;
[ $$(( now - mod )) -lt ${DB_GUARD_HEALTH_MAX_AGE} ] || exit 1
interval: ${DB_GUARD_HEALTHCHECK_INTERVAL}
timeout: ${DB_GUARD_HEALTHCHECK_TIMEOUT}
retries: ${DB_GUARD_HEALTHCHECK_RETRIES}
ac-db-init:
profiles: ["db"]
image: ${MYSQL_IMAGE}
@@ -106,7 +195,7 @@ services:
ac-db-import:
condition: service_completed_successfully
volumes:
- ${STORAGE_PATH_LOCAL}/mysql-data:/var/lib/mysql-persistent
- mysql-data:/var/lib/mysql-persistent
- ${BACKUP_PATH}:/backups
networks:
- azerothcore
@@ -168,6 +257,7 @@ services:
CONTAINER_USER: ${CONTAINER_USER}
volumes:
- ${BACKUP_PATH}:/backups
- ${STORAGE_PATH}/modules/.modules-meta:/modules-meta:ro
- ./scripts:/tmp/scripts:ro
working_dir: /tmp
command:
@@ -234,7 +324,7 @@ services:
profiles: ["client-data", "client-data-bots"]
image: ${ALPINE_IMAGE}
container_name: ac-volume-init
user: "0:0"
user: "${CONTAINER_USER}"
volumes:
- ${CLIENT_DATA_PATH:-${STORAGE_PATH}/client-data}:/azerothcore/data
- client-data-cache:/cache
@@ -242,11 +332,16 @@ services:
- sh
- -c
- |
mkdir -p /azerothcore/data
echo "🔧 Fixing Docker volume permissions..."
chown -R ${CONTAINER_USER} /azerothcore/data /cache
chmod -R 755 /azerothcore/data /cache
echo "✅ Docker volume permissions fixed"
mkdir -p /azerothcore/data /cache
if [ "$(id -u)" -eq 0 ]; then
echo "🔧 Normalizing client-data volume ownership..."
chown -R ${CONTAINER_USER} /azerothcore/data /cache
chmod -R 755 /azerothcore/data /cache
echo "✅ Docker volume permissions fixed"
else
echo "ℹ️ Running as $(id -u):$(id -g); skipping ownership changes."
fi
echo "📦 Client data volumes ready"
restart: "no"
networks:
- azerothcore
@@ -255,7 +350,7 @@ services:
profiles: ["db", "modules"]
image: ${ALPINE_IMAGE}
container_name: ac-storage-init
user: "0:0"
user: "${CONTAINER_USER}"
volumes:
- ${STORAGE_PATH}:/storage-root
- ${STORAGE_PATH_LOCAL}:/local-storage-root
@@ -267,11 +362,20 @@ services:
mkdir -p /storage-root/config /storage-root/logs /storage-root/modules /storage-root/lua_scripts /storage-root/install-markers
mkdir -p /storage-root/config/mysql/conf.d
mkdir -p /storage-root/client-data
mkdir -p /storage-root/backups /local-storage-root/mysql-data
mkdir -p /storage-root/backups
# Copy core config files if they don't exist
if [ -f "/local-storage-root/source/azerothcore-playerbots/src/tools/dbimport/dbimport.conf.dist" ] && [ ! -f "/storage-root/config/dbimport.conf.dist" ]; then
echo "📄 Copying dbimport.conf.dist..."
cp /local-storage-root/source/azerothcore-playerbots/src/tools/dbimport/dbimport.conf.dist /storage-root/config/
fi
# Fix ownership of root directories and all contents
chown -R ${CONTAINER_USER} /storage-root /local-storage-root
chmod -R 755 /storage-root /local-storage-root
echo "✅ Storage permissions initialized"
if [ "$(id -u)" -eq 0 ]; then
chown -R ${CONTAINER_USER} /storage-root /local-storage-root
chmod -R 755 /storage-root /local-storage-root
echo "✅ Storage permissions initialized"
else
echo "ℹ️ Running as $(id -u):$(id -g); assuming host permissions are already correct."
fi
restart: "no"
networks:
- azerothcore
@@ -294,7 +398,7 @@ services:
working_dir: /tmp
environment:
- CONTAINER_USER=${CONTAINER_USER}
- CLIENT_DATA_VERSION=${CLIENT_DATA_VERSION}
- CLIENT_DATA_VERSION=${CLIENT_DATA_VERSION:-}
command:
- sh
- -c
@@ -325,23 +429,24 @@ services:
working_dir: /tmp
environment:
- CONTAINER_USER=${CONTAINER_USER}
- CLIENT_DATA_VERSION=${CLIENT_DATA_VERSION}
- CLIENT_DATA_VERSION=${CLIENT_DATA_VERSION:-}
command:
- sh
- -c
- |
echo "📦 Installing 7z for faster extraction..."
apt-get update -qq && apt-get install -y p7zip-full
mkdir -p /cache
if [ -f /tmp/scripts/bash/download-client-data.sh ]; then
chmod +x /tmp/scripts/bash/download-client-data.sh 2>/dev/null || true
bash /tmp/scripts/bash/download-client-data.sh
echo "🔧 Fixing ownership of extracted files..."
chown -R ${CONTAINER_USER} /azerothcore/data
echo "✅ Client data extraction and ownership setup complete"
else
echo "No local client-data script"
fi
echo "📦 Installing 7z + gosu for client data extraction..."
apt-get update -qq && apt-get install -y p7zip-full gosu
gosu ${CONTAINER_USER} bash -c '
set -e
mkdir -p /cache
if [ -f /tmp/scripts/bash/download-client-data.sh ]; then
chmod +x /tmp/scripts/bash/download-client-data.sh 2>/dev/null || true
bash /tmp/scripts/bash/download-client-data.sh
echo "✅ Client data extraction completed under UID $(id -u)"
else
echo "No local client-data script"
fi
'
restart: "no"
networks:
- azerothcore
@@ -400,7 +505,7 @@ services:
AC_LOGIN_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_AUTH_NAME}"
AC_WORLD_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_WORLD_NAME}"
AC_CHARACTER_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_CHARACTERS_NAME}"
AC_UPDATES_ENABLE_DATABASES: "0"
AC_UPDATES_ENABLE_DATABASES: "7"
AC_BIND_IP: "0.0.0.0"
AC_DATA_DIR: "/azerothcore/data"
AC_SOAP_PORT: "7878"
@@ -447,8 +552,8 @@ services:
depends_on:
ac-mysql:
condition: service_healthy
ac-db-import:
condition: service_completed_successfully
ac-db-guard:
condition: service_healthy
ac-db-init:
condition: service_completed_successfully
environment:
@@ -483,13 +588,13 @@ services:
depends_on:
ac-mysql:
condition: service_healthy
ac-db-import:
condition: service_completed_successfully
ac-db-guard:
condition: service_healthy
ac-db-init:
condition: service_completed_successfully
environment:
AC_LOGIN_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_AUTH_NAME}"
AC_UPDATES_ENABLE_DATABASES: "0"
AC_UPDATES_ENABLE_DATABASES: "7"
AC_BIND_IP: "0.0.0.0"
AC_LOG_LEVEL: "1"
AC_LOGGER_ROOT_CONFIG: "1,Console"
@@ -522,11 +627,13 @@ services:
condition: service_healthy
ac-client-data-playerbots:
condition: service_completed_successfully
ac-db-guard:
condition: service_healthy
environment:
AC_LOGIN_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_AUTH_NAME}"
AC_WORLD_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_WORLD_NAME}"
AC_CHARACTER_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_CHARACTERS_NAME}"
AC_UPDATES_ENABLE_DATABASES: "0"
AC_UPDATES_ENABLE_DATABASES: "7"
AC_BIND_IP: "0.0.0.0"
AC_DATA_DIR: "/azerothcore/data"
AC_SOAP_PORT: "7878"
@@ -575,11 +682,13 @@ services:
condition: service_healthy
ac-client-data-standard:
condition: service_completed_successfully
ac-db-guard:
condition: service_healthy
environment:
AC_LOGIN_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_AUTH_NAME}"
AC_WORLD_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_WORLD_NAME}"
AC_CHARACTER_DATABASE_INFO: "${CONTAINER_MYSQL};${MYSQL_PORT};${MYSQL_USER};${MYSQL_ROOT_PASSWORD};${DB_CHARACTERS_NAME}"
AC_UPDATES_ENABLE_DATABASES: "0"
AC_UPDATES_ENABLE_DATABASES: "7"
AC_BIND_IP: "0.0.0.0"
AC_DATA_DIR: "/azerothcore/data"
AC_SOAP_PORT: "7878"
@@ -645,11 +754,10 @@ services:
command:
- -c
- |
apk add --no-cache curl bash git python3
(chmod +x /tmp/scripts/bash/manage-modules.sh /tmp/scripts/bash/manage-modules-sql.sh 2>/dev/null || true) && /tmp/scripts/bash/manage-modules.sh
# Fix permissions after module operations
chown -R ${CONTAINER_USER} /modules /azerothcore/env/dist/etc 2>/dev/null || true
chmod -R 755 /modules /azerothcore/env/dist/etc 2>/dev/null || true
apk add --no-cache curl bash git python3 su-exec
chmod +x /tmp/scripts/bash/manage-modules.sh /tmp/scripts/bash/manage-modules-sql.sh 2>/dev/null || true
echo "🔐 Running module manager as ${CONTAINER_USER}"
su-exec ${CONTAINER_USER} /bin/sh -c 'set -e; cd /modules && /tmp/scripts/bash/manage-modules.sh'
restart: "no"
networks:
- azerothcore
@@ -694,14 +802,10 @@ services:
- sh
- -c
- |
apk add --no-cache bash curl docker-cli
chown -R ${CONTAINER_USER} /azerothcore/config /install-markers 2>/dev/null || true
chmod -R 755 /azerothcore/config /install-markers 2>/dev/null || true
echo "📥 Running local auto-post-install script..."
(chmod +x /tmp/scripts/bash/auto-post-install.sh 2>/dev/null || true) && bash /tmp/scripts/bash/auto-post-install.sh
# Fix permissions for all files created during post-install
chown -R ${CONTAINER_USER} /azerothcore/config /install-markers 2>/dev/null || true
chmod -R 755 /azerothcore/config /install-markers 2>/dev/null || true
apk add --no-cache bash curl docker-cli su-exec
chmod +x /tmp/scripts/bash/auto-post-install.sh 2>/dev/null || true
echo "📥 Running post-install as ${CONTAINER_USER}"
su-exec ${CONTAINER_USER} bash /tmp/scripts/bash/auto-post-install.sh
restart: "no"
networks:
- azerothcore
@@ -771,6 +875,8 @@ services:
volumes:
client-data-cache:
driver: local
mysql-data:
driver: local
networks:
azerothcore: