git / moonbite-network / monitoring/MONITORING_SETUP.md · raw

# MoonBite Production Monitoring Setup## OverviewComplete monitoring system for MoonBite Production Wallet deployed on Railway.**Production URL**: https://www.moonbite.org/wallet**Railway URL**: https://helios-production-5ad6.up.railway.app**Status Page**: https://status.moonbite.org---## 1. Real-Time Monitoring### Health Check ScriptRun continuous production monitoring:```bash# One-time health checkpython monitoring/health-check.py# Continuous monitoring (every 5 minutes)bash monitoring/uptime-monitoring.sh```### Monitored Endpoints✅ Wallet PWA (https://www.moonbite.org/wallet)✅ Homepage (https://www.moonbite.org/)✅ HD Wallet API (hd/new endpoint)✅ Blockchain Status API✅ SSL Certificate validity✅ DNS Resolution (69.46.46.28)---## 2. Alert Configuration### Alert Rules (monitoring/alerts.json)**Critical Alerts** (Immediate Action Required):- Wallet endpoint down- API errors > 5%- Response time > 5 seconds- Database connection failed- Memory usage > 90%**Warning Alerts** (Investigate Within 1 Hour):- Homepage slow- Blockchain sync issues- DNS resolution problems### Alert Channels- **Email**: admin@moonbite.org- **Slack**: (configure webhook in alerts.json)- **PagerDuty**: (configure integration key for critical incidents)---## 3. Railway Platform Monitoring### Dashboard- **URL**: https://railway.app (logged in as project owner)- **Metrics**: CPU, Memory, Disk, Network- **Logs**: Real-time application logs- **Deployments**: See all service versions### Auto-Restart Policy```json{  "restartPolicyType": "ON_FAILURE",  "restartPolicyMaxRetries": 5}```- Service auto-restarts on crash- Max 5 restart attempts- Manual restart available via Railway dashboard### Health Check Configuration```json{  "endpoint": "/api/blockchain/status",  "interval": 30,  "timeout": 10}```Railway pings `/api/blockchain/status` every 30 seconds to verify health.---## 4. Performance Metrics### Track These Metrics- **CPU Usage**: Target < 70%- **Memory Usage**: Target < 80%- **Response Time (p50)**: Target < 500ms- **Response Time (p95)**: Target < 2s- **Error Rate**: Target < 1%- **Request Rate**: Track spikes- **Database Connections**: Max 10 active### SLA Targets- **Uptime**: 99.9% (8.7 hours downtime/month)- **Response Time**: < 5s (p95)- **Error Rate**: < 5%---## 5. Log Aggregation### Log Locations```Local Logs: ./monitoring/logs/Railway Logs: https://railway.app > Project > Logs```### Log Retention- Monitoring logs: 30 days- Railway logs: 7 days (Railway default)- Archive old logs monthly### Key Log Patterns to Monitor```ERROR: Flask app crashedERROR: Database connection failedERROR: Out of memoryWARNING: Slow response (> 5s)INFO: Deployment startedINFO: Service restarted```---## 6. Incident Response### If Wallet Is Down1. **Check Status**:   ```bash   python monitoring/health-check.py   ```2. **Check Railway Dashboard**:   - Is service running?   - Is there an error in logs?   - CPU/Memory at limits?3. **Check DNS**:   ```bash   nslookup www.moonbite.org   # Should return 69.46.46.28   ```4. **Restart Service** (if needed):   - Via Railway Dashboard: Click "Restart Service"   - Service auto-restarts on crash5. **Check Recent Deployments**:   - Did a new version break something?   - Rollback to previous version if needed### If API Is Slow1. Check memory usage (< 80%)2. Check active request count3. Check database connections4. Check for recent code changes5. Scale up service if needed (Railway Dashboard)### If SSL Certificate Expires- **Expiry Check**:  ```bash  openssl s_client -connect www.moonbite.org:443 -servername www.moonbite.org < /dev/null 2>/dev/null | openssl x509 -noout -dates  ```- **Renewal**: Let's Encrypt auto-renews (Certbot configured in Dockerfile)- **Manual Renewal**:  ```bash  certbot renew --force-renewal  ```---## 7. Automated Monitoring Setup### Option A: Linux/Mac Cron Job```bash# Edit crontabcrontab -e# Add this line to run health check every 5 minutes*/5 * * * * cd /opt/moonbite && python monitoring/health-check.py >> monitoring/logs/health.log 2>&1```### Option B: Railway Scheduled JobsComing soon: Railway supports scheduled jobs for monitoring.### Option C: External ServicePopular options:- **UptimeRobot** (free tier): https://uptimerobot.com- **Pingdom** (paid): https://www.pingdom.com- **StatusPage.io**: Status page + monitoring---## 8. Status Page### Create Public Status Page```bash# Option 1: Use StatusPage.io# Create account, add components, set up monitoring# Option 2: Self-hosted status page# Use Cachet or similar open-source solution```### Publish Status- https://status.moonbite.org (public)- Include: Uptime history, incident log, maintenance windows---## 9. Monthly Review### Every Month:1. Review uptime metrics2. Analyze error logs for patterns3. Check response time trends4. Update alert thresholds if needed5. Review incident log6. Test disaster recovery plan7. Update monitoring documentation### Quarterly Deep Dive:1. Capacity planning (do we need to scale?)2. Security review (are we logging enough?)3. Database performance analysis4. Review failed deployments5. Update runbooks---## 10. Quick Reference### Emergency Commands```bash# Health checkpython monitoring/health-check.py# Test connectivitycurl -I https://www.moonbite.org/wallet# Check DNSnslookup www.moonbite.org 8.8.8.8# View logstail -f monitoring/logs/*.log# Start continuous monitoringbash monitoring/uptime-monitoring.sh```### Important URLs- **Production**: https://www.moonbite.org/wallet- **Railway Dashboard**: https://railway.app- **Namecheap DNS**: https://www.namecheap.com/cp/- **Status Page**: https://status.moonbite.org- **GitHub Repo**: https://moonbite.org/git/moonbite-network### Contacts- **Admin Email**: admin@moonbite.org- **Support**: support@moonbite.org- **Emergency**: (configure PagerDuty)---## 11. Next Steps✅ **Completed**:- Health check script- Alert configuration- Railway integration- Log setup📋 **Todo**:- [ ] Configure email alerts (alerts.json)- [ ] Set up Slack webhook- [ ] Create public status page- [ ] Test incident response procedures- [ ] Schedule cron jobs for monitoring- [ ] Configure PagerDuty escalation- [ ] Document runbooks- [ ] Train ops team---**Last Updated**: 2026-08-08**Status**: ✅ Production Monitoring Live