Troubleshooting Tomcat Web Applications on AlmaLinux/RHEL

Overview

Apache Tomcat is a widely used Java servlet container that can encounter various issues during deployment and operation. This comprehensive guide covers systematic troubleshooting approaches for Tomcat web applications running on AlmaLinux and Red Hat Enterprise Linux (RHEL) systems.

Prerequisites

Before beginning troubleshooting, ensure you have:

• Root or sudo access to the server

• Basic understanding of Linux command line

• Knowledge of Tomcat directory structure

• Access to log files and configuration files

Common Tomcat Directory Structure

Understanding the standard Tomcat directory layout is crucial for effective troubleshooting:

/opt/tomcat/ (or /usr/share/tomcat/)
├── bin/           # Executable scripts
├── conf/          # Configuration files
├── lib/           # JAR libraries
├── logs/          # Log files
├── temp/          # Temporary files
├── webapps/       # Web applications
└── work/          # Compiled JSPs and temporary files

Initial Diagnostic Steps

1. Check Tomcat Service Status

First, verify if Tomcat is running:

bash

# Check service status
sudo systemctl status tomcat

# Check if Tomcat process is running
sudo ps aux | grep tomcat

# Check listening ports
sudo netstat -tlnp | grep :8080
# or using ss
sudo ss -tlnp | grep :8080

2. Verify Java Installation

Tomcat requires Java to run. Check your Java environment:

bash

# Check Java version
java -version

# Check JAVA_HOME environment variable
echo $JAVA_HOME

# Verify Java path in Tomcat configuration
cat /opt/tomcat/conf/tomcat.conf | grep JAVA_HOME

3. Review System Resources

Check if the system has sufficient resources:

bash

# Check memory usage
free -h

# Check disk space
df -h

# Check CPU usage
top -p $(pgrep -f tomcat)

# Check system load
uptime

Log File Analysis

Log files are your primary diagnostic tool. Tomcat generates several types of logs:

Key Log Files

1. catalina.out - Main Tomcat output log

2. catalina.YYYY-MM-DD.log - Daily Tomcat logs

3. localhost.YYYY-MM-DD.log - Localhost-specific logs

4. manager.YYYY-MM-DD.log - Manager application logs

5. access.YYYY-MM-DD.log - Access logs (if enabled)

Log Analysis Commands

bash

# Monitor real-time logs
sudo tail -f /opt/tomcat/logs/catalina.out

# Search for errors
sudo grep -i error /opt/tomcat/logs/catalina.out

# Check for OutOfMemoryError
sudo grep -i "outofmemory" /opt/tomcat/logs/*.log

# Look for application-specific errors
sudo grep -i "your-app-name" /opt/tomcat/logs/*.log

# Check for permission issues
sudo grep -i "permission" /opt/tomcat/logs/*.log

Common Issues and Solutions

1. Tomcat Won't Start

Symptoms:

• Service fails to start

• Process not running

• Port not listening

Troubleshooting Steps:

bash

# Check detailed service status
sudo systemctl status tomcat -l

# View service logs
sudo journalctl -u tomcat -f

# Check for port conflicts
sudo netstat -tlnp | grep 8080

# Verify file permissions
sudo ls -la /opt/tomcat/bin/catalina.sh
sudo ls -la /opt/tomcat/conf/

# Check if another process is using the port
sudo lsof -i :8080

Common Solutions:

• Kill conflicting processes using the port

• Fix file permissions: sudo chown -R tomcat:tomcat /opt/tomcat/

• Verify JAVA_HOME is correctly set

• Check for syntax errors in configuration files

2. Application Deployment Issues

Symptoms:

• WAR files not deploying

• Applications showing as "Failed" in manager

• 404 errors when accessing applications

Troubleshooting Steps:

bash

# Check webapps directory permissions
sudo ls -la /opt/tomcat/webapps/

# Verify WAR file integrity
file /opt/tomcat/webapps/yourapp.war

# Check deployment logs
sudo grep -i "deploy" /opt/tomcat/logs/catalina.out

# Look for application-specific errors
sudo grep -A 10 -B 10 "yourapp" /opt/tomcat/logs/localhost.*.log

Common Solutions:

• Ensure WAR files have correct permissions

• Check for missing dependencies in WEB-INF/lib

• Verify context.xml configuration

• Clear work directory: sudo rm -rf /opt/tomcat/work/Catalina/localhost/yourapp/

3. Memory Issues

Symptoms:

• OutOfMemoryError in logs

• Slow application performance

• Tomcat becomes unresponsive

Troubleshooting Steps:

bash

# Check current JVM memory settings
ps aux | grep tomcat | grep -o '\-Xm[sx][0-9]*[gm]'

# Monitor memory usage
sudo -u tomcat jstat -gc $(pgrep -f tomcat) 5s

# Generate heap dump for analysis
sudo -u tomcat jmap -dump:format=b,file=/tmp/heapdump.hprof $(pgrep -f tomcat)

Solutions: Edit /opt/tomcat/bin/setenv.sh (create if doesn't exist):

bash

export CATALINA_OPTS="-Xms512m -Xmx2048m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"

4. Connection and Timeout Issues

Symptoms:

• Connection timeouts

• Slow response times

• Database connection errors

Troubleshooting Steps:

bash

# Check active connections
sudo ss -an | grep :8080 | wc -l

# Monitor connection pools (if using)
sudo grep -i "pool" /opt/tomcat/logs/catalina.out

# Check database connectivity
sudo -u tomcat telnet database-host 3306

Solutions: Adjust connector configuration in /opt/tomcat/conf/server.xml:

xml

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           maxThreads="200"
           acceptCount="100"
           redirectPort="8443" />

5. SSL/HTTPS Issues

Symptoms:

• SSL handshake failures

• Certificate errors

• HTTPS connections failing

Troubleshooting Steps:

bash

# Test SSL connection
openssl s_client -connect localhost:8443 -servername yourdomain.com

# Check certificate validity
openssl x509 -in /path/to/certificate.crt -text -noout

# Verify keystore
keytool -list -v -keystore /opt/tomcat/conf/keystore.jks

Performance Troubleshooting

1. Thread Analysis

bash

# Generate thread dump
sudo -u tomcat jstack $(pgrep -f tomcat) > /tmp/threaddump.txt

# Monitor thread usage
sudo -u tomcat jstat -gc $(pgrep -f tomcat) 1s 10

2. Database Connection Monitoring

If your application uses databases:

bash

# Check active database connections
sudo netstat -an | grep :3306 | grep ESTABLISHED | wc -l

# Monitor connection pool in logs
sudo grep -i "connection" /opt/tomcat/logs/catalina.out | tail -20

Configuration File Troubleshooting

server.xml Issues

Common problems in /opt/tomcat/conf/server.xml:

bash

# Validate XML syntax
xmllint --noout /opt/tomcat/conf/server.xml

# Check for common misconfigurations
sudo grep -n "maxThreads\|connectionTimeout\|maxConnections" /opt/tomcat/conf/server.xml

web.xml Problems

bash

# Validate application web.xml
xmllint --noout /opt/tomcat/webapps/yourapp/WEB-INF/web.xml

# Check for servlet mapping issues
sudo grep -A 5 -B 5 "servlet-mapping" /opt/tomcat/webapps/yourapp/WEB-INF/web.xml

Security-Related Troubleshooting

SELinux Issues (Common on RHEL/AlmaLinux)

bash

# Check SELinux status
getenforce

# Look for SELinux denials
sudo ausearch -m avc -ts recent | grep tomcat

# Check SELinux context
ls -Z /opt/tomcat/webapps/

# Temporarily disable SELinux for testing (not recommended for production)
sudo setenforce 0

Firewall Issues

bash

# Check firewall status
sudo firewall-cmd --state

# List allowed ports
sudo firewall-cmd --list-ports

# Allow Tomcat port
sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload

Advanced Debugging Techniques

1. Enable Debug Logging

Edit /opt/tomcat/conf/logging.properties:

properties

# Enable debug logging for specific packages
org.apache.catalina.core.ContainerBase.[Catalina].level = FINE
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

2. JMX Monitoring

Add JMX options to /opt/tomcat/bin/setenv.sh:

bash

export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

3. Remote Debugging

For application debugging:

bash

export CATALINA_OPTS="$CATALINA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000"

Preventive Measures

1. Regular Monitoring

Create monitoring scripts:

bash

#!/bin/bash
# tomcat-health-check.sh

# Check if Tomcat is running
if ! pgrep -f tomcat > /dev/null; then
    echo "CRITICAL: Tomcat is not running"
    exit 2
fi

# Check memory usage
MEMORY_USAGE=$(ps -o pid,ppid,cmd,%mem --pid $(pgrep -f tomcat) | tail -1 | awk '{print $4}')
if (( $(echo "$MEMORY_USAGE > 80" | bc -l) )); then
    echo "WARNING: High memory usage: ${MEMORY_USAGE}%"
fi

# Check response time
RESPONSE_TIME=$(curl -o /dev/null -s -w '%{time_total}' http://localhost:8080/)
if (( $(echo "$RESPONSE_TIME > 5" | bc -l) )); then
    echo "WARNING: Slow response time: ${RESPONSE_TIME}s"
fi

echo "OK: Tomcat is healthy"

2. Log Rotation

Configure logrotate for Tomcat logs:

bash

# /etc/logrotate.d/tomcat
/opt/tomcat/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 tomcat tomcat
    postrotate
        systemctl reload tomcat
    endscript
}

3. Backup Strategy

bash

#!/bin/bash
# backup-tomcat-config.sh

BACKUP_DIR="/backup/tomcat/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Backup configuration
cp -r /opt/tomcat/conf "$BACKUP_DIR/"

# Backup webapps
cp -r /opt/tomcat/webapps "$BACKUP_DIR/"

# Create archive
tar -czf "$BACKUP_DIR.tar.gz" -C /backup/tomcat "$(basename $BACKUP_DIR)"

Emergency Recovery Procedures

1. Quick Restart Procedure

bash

# Stop Tomcat gracefully
sudo systemctl stop tomcat

# Kill if necessary
sudo pkill -f tomcat

# Clear temp and work directories
sudo rm -rf /opt/tomcat/temp/*
sudo rm -rf /opt/tomcat/work/*

# Start Tomcat
sudo systemctl start tomcat

# Monitor startup
sudo tail -f /opt/tomcat/logs/catalina.out

2. Rollback Application

bash

# Remove problematic application
sudo rm -rf /opt/tomcat/webapps/problematic-app*

# Deploy previous version
sudo cp /backup/previous-version.war /opt/tomcat/webapps/

# Restart Tomcat
sudo systemctl restart tomcat

Conclusion

Effective Tomcat troubleshooting requires a systematic approach combining log analysis, system monitoring, and understanding of common failure patterns. Regular monitoring and preventive measures can help avoid many issues before they impact production systems.

Key takeaways for successful troubleshooting:

• Always start with log file analysis

• Check system resources and dependencies

• Use systematic elimination of potential causes

• Document solutions for future reference

• Implement monitoring and alerting

• Maintain regular backups of configurations

Remember to test any changes in a development environment before applying them to production systems, and always have a rollback plan ready.