Notes on Apache Tomcat

Russell Bateman
January 2022

This is a effort to become a little more DevOps-ish for Tomcat.

I don't teach how to do any of this on Windows. Teaching others to work on Windows is criminal and irresponsible.


How to bump memory for Tomcat...

...when Tomcat is set up as a service. This change is for Tomcat itself and it's probably unnecessary to do it because Tomcat is only launching containers (servlets) underneath itself and comes with enough memory to accomplish this—depending on the number of applications it's asked to serve up.

  1. Go to /etc/systemd/system.
  2. Edit tomcat.service.
  3. Change environment for CATALINA_OPTS (these are the shipping defaults for Tomcat 9):
    Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -XX:+UseParallelGC'
    
    Note that
    1. -Xms is the initial (start-up) size of the stack
    2. -Xmx is the maximum size heap memory grow to
  4. Bounce the Tomcat service:
    # systemctl restart tomcat
    

How to bump memory for an application running in Tomcat...

...whether or not Tomcat is set up as a service. This changes is for the applications Tomcat runs and not Tomcat itself.

  1. Go to /opt/tomcat/bin.
  2. Create/edit setenv.sh.
  3. Create/edit environment for JAVA_OPTS:
    Environment='JAVA_OPTS=-Xms1024M -Xmx1024M'
    
    Note that
    1. -Xms is the initial (start-up) size of the memory pool
    2. -Xmx is the maximum size the memory pool can grow to
    3. -Xss is the size of each thread's stack (not being set here)
  4. Bounce the Tomcat service which will reaload the application(s) or servlet(s) Tomcat is serving as a service (if you installed it as a service)...
    # systemctl restart tomcat
    
  5. ...or, relaunch Tomcat from the command line:
    # /opt/tomcat/bin/startup.sh
    

How does this work?

When Tomcat is launched, /opt/tomcat/bin/catalina.sh is executed. This script looks for $CATALINA_HOME/bin/setenv.sh and, if it exists, it will execute it.

What if Tomcat is running in Docker?

Read later on this page about this.


How port numbers work in Tomcat

In /opt/tomcat/conf/server.xml is configuration for port numbers in Tomcat. Typically, this configuration is simple (and this is the established default):

<Service name="Catalina">
  ...
  <Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
                redirectPort="8443" />
  ...

The URL http://localhost:8080 will be responded to in the browser by Tomcat's splash page.

If the containers (servlets) that Tomcat is hosting don't respond to unsecured requests, Tomcat can be told this:

<Service name="Catalina">
  ...
  <!-- <Connector port="8080"
              protocol="HTTP/1.1"
              connectionTimeout="20000"
              redirectPort="8443" /> -->
  <Connector port="8443"
             protocol="org.apache.coyote.http11.Http11NioProtocol"
             maxThreads="150"
             SSLEnabled="true">
    <SSLHostConfig>
      <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
                   type="RSA" />
    </SSLHostConfig>
  </Connector>
  ...

In this latter case, the URL http://localhost:8080 will not even respond.


Tomcat shutdown port 8005

By default, Tomcat's /opt/tomcat/conf/server.xml specifies port 8005 as its shutdown port. This is important for firewall configuration and for starting and stopping.

<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  ...
</Server>

You can use telnet to reach any Tomcat server as long as you know the host address and port 8005 is configured.

Once connected to the (local or remote) Tomcat server over port 8005, you have 10 seconds to enter the SHUTDOWN (or other, case=sensitive shutdown command as configured in server.xml).

Restarting Tomcat can only be done at the console command line or by sshing into the remote host to reach the command line.


The Tomcat Manager...

...is an application packaged with Tomcat that provides a graphical way of administering the web application containers Tomcat serves. It's almost never necessary and more of a curiosity, but it can be useful. From the Tomcat Manager you can start, stop, reload or undeploy any container (servlet) Tomcat is managing.

From the Server Status link you can see how much memory Tomcat is using (including pool names, type, initial, current, maximum, etc.).

It is deployed by default but no one is enabled to reach it by default. To give access, you can just add, for instance,

<tomcat-users xmlns="http://tomcat.apache.org/xml"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
              version="1.0">
  ...
  <user username="russ"  password="password" roles="manager-gui,admin-gui" />
</tomcat-users>

...to /opt/tomcat/conf/tomcat-users.xml.

Thereupon, if you reach in a browser for URL http://localhost:8080/manager, you will get a sign-in dialog allowing you to type in (in this case) "russ" and "password".

It's possible to restrict access to the Tomcat Manager from remote hosts (hosts other than 127.0.0.1) by adding the following to /opt/tomcat/conf/context.xml:

<Context path="/manager"
         privileged="true"
         docBase="/usr/local/tomcat6/webapps/manager">
  <Valve className="org.apache.catalina.valves.RemoteAddrValve"
         allow="127\.0\.0\.1" />
</Context>

Demystifying Tomcat logging...

I experimented extensively one day running Tomcat on a remote server and hitting it with POSTs forcing considerable output to catalina.out. Playing around, I reached the following conclusions:

  1. It's not useful to modify /opt/tomcat/conf/logging.properties except for very global changes affecting everything running under Tomcat. This is not ordinarily what you want to do.

  2. The addition in parallel of the CONSOLE appender to the existing FILE appender will cause "double logging"—meaning that everything will come out twice in catalina.out.

    An appender in terms of logging is synonymous with "printer." You may specify a console printer (as discussed here) and/or one or more "file-system printer(s)" that will end up in different files and different places in the file system. Here, our file appender is specifically following the Tomcat/Catalina standard of catalina.out.

    Examples of appenders will appear in logback configuration below.

  3. The logging properties in play for any web application's production logging come from (Java source-code development path):

    application/src/main/resources/logback.xml

    These are what you want in that file (notice there's only a FILE appender). In a production, Tomcat-deployed environment, the logback configuration file ends up on the path:

    /opt/tomcat/webapps/application/WEB-INF/classes/logback.xml

    To get more detail, you would replace INFO with DEBUG (verbose) or TRACE (even more verbose). If you wanted to tweak this file—live—you can edit it directly on the server, however, you will have to bounce Tomcat before you'll observe any effect of your tweaking.

    <configuration>
      <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>${catalina.base}/logs/catalina.out</file>
        <append>true</append>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
          <pattern>%-4relative [%thread] %-5level %logger{5}.%method:%line - %message%n</pattern>
        </encoder>
      </appender>
    
      <!-- Get the details on generating CCD. -->
      <logger name="com.acme.mdht" level="INFO" additivity="false">
        <appender-ref ref="FILE" />
      </logger>
    
      <root level="INFO">
        <appender-ref ref="FILE" />
      </root>
    </configuration>
    
  4. Now I see this (reduced from double output) for each document processed. This example also demonstrates the encoder pattern from the logback configuration above:
    51809 [http-nio-8080-exec-9] INFO  c.a.m.Generate.generateDocument:210 -  Document (CCDA) generated for MPID 1694505
    51809 [http-nio-8080-exec-9] INFO  c.a.m.Generate.generateDocument:215 -  Vendor requesting document is Acme, Inc.
    51810 [http-nio-8080-exec-9] INFO  c.a.m.c.GenerateCcda.generate:114 -    Generating CCDA document (at 20230705004825+0000)
    51810 [http-nio-8080-exec-9] WARN  c.a.m.c.PopulateDocumentationOf.populatePerformerFromAuthor:149 - No NPI code for performer from author.person, line:319
    51827 [http-nio-8080-exec-9] INFO  c.a.m.Generate.generateDocument:265 -  Document generated; elapsed time: 00:00:00:00:017
    
    This pattern is my own, highly informative, yet super abbreviated log-file format.

  5. The logging properties in play when running JUnit tests from the IDE for application logging come from:

    application/src/test/resources/logback.xml

    Something like what's below is what you want in that file (notice that there is a CONSOLE appender, highlighted here in green , which is what comes out in the JUnit console see way below).

    <configuration>
      <property name="base.folder" value="${catalina.home:-./target}" />
    
      <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>${base.folder}/logs/catalina.out</file>
        <append>true</append>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
          <pattern>%-4relative [%thread] %-5level %logger{5}.%method:%line - %message%n</pattern>
        </encoder>
      </appender>
    
      <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
          <pattern>%-4relative [%thread] %-5level %logger{5}.%method:%line - %message%n</pattern>
        </encoder>
      </appender>
    
      <!-- Get the details on generating CCD. -->
      <logger name="com.acme.mdht" level="INFO" additivity="false">
        <appender-ref ref="FILE" />
        <appender-ref ref="CONSOLE" />
      </logger>
    
      <!-- Very verbose: "TRACE" adds one or more lines per POJO to the log. -->
      <logger name="com.acme.sax" level="INFO" additivity="false">
        <appender-ref ref="FILE" />
        <appender-ref ref="CONSOLE" />
      </logger>
    
      <root level="INFO">
        <appender-ref ref="FILE" />
        <appender-ref ref="CONSOLE" />
      </root>
    </configuration>
    
  6. This configuration, using DEBUG instead of INFO, results in great verbosity (that's often useful when debugging):
    88560 [http-nio-8080-exec-55] WARN  c.a.s.ParseIncoming.parse:115 - Warnings of IXML and other problems arising during SAX parsing and POJO construction:
    88562 [http-nio-8080-exec-55] INFO  c.a.s.ParseIncoming.parse:120 - Back from SAX parser...
    88562 [http-nio-8080-exec-55] INFO  c.a.m.Generate.generateDocument:210 -  Document (CCDA) generated for MPID 1694505
    88562 [http-nio-8080-exec-55] INFO  c.a.m.Generate.generateDocument:215 -  Vendor requesting document is Acme, Inc.
    88562 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:114 -    Generating CCDA document (at 20230704234940+0000)
    88563 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:121 -    Expected POJOs in play:
                                                            ...Next of kin        4
                                                            ...Custodian info     1
                                                            ...Allergies          3
                                                            ...Encounters         1
                                                            ...Functional Status  1
                                                            ...Immunizations      1
                                                            ...Medications        1
                                                            ...Problems           1
                                                            ...Procedures         1
                                                            ...Result Panels      1
                                                            ...Results            1
                                                            ...Social Histories   1
                                                            ...Vital Sign groups  3
                                                            ...Vital Signs        4
    88563 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:125 -      ...for 1 HIE info
    88563 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:126 -      ...for 1 Custodian info
    88563 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:127 -      ...for 4 Next of kin
    88563 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:131 -      ...for 8 Authors (and potential Participants)
    88563 [http-nio-8080-exec-55] WARN  c.a.m.c.PopulateDocumentationOf.populatePerformerFromAuthor:149 - No NPI code for performer from author.person, line:319
    88564 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:151 -      ...for 3 Allergies and Intolerances
    88565 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:159 -      ...for 1 Encounter
    88566 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:166 -      ...for 1 Functional Status
    88567 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:173 -      ...for 1 Immunization
    88568 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:180 -      ...for 1 Medication
    88569 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:194 -      ...for 1 Problem
    88571 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:201 -      ...for 1 Procedure
    88571 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:208 -      ...for 1 Result panel/Lab (1 result)
    88573 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:215 -      ...for 1 Social Histories
    88574 [http-nio-8080-exec-55] INFO  c.a.m.c.GenerateCcda.generate:224 -      ...for 3 Vital Sign groups (4 vital signs)
    88581 [http-nio-8080-exec-55] INFO  c.a.m.Generate.generateDocument:265 -  Document generated; elapsed time: 00:00:00:00:019
    

Tomcat and Docker notes on accessibility...

Accessibility problems we wish to solve in Tomcat running in a Docker container:

  1. deployment of our application to /opt/tomcat/webapps.

  2. logging to /opt/tomcat/logs/catalina.out, in particular, roll-over and the absence of logrotate in the container.

Solutions:

  1. Don't build a special Docker image for deploying your WAR file.

  2. Don't attempt to rig a container with logrotate, cron and everything else necessary inside.

  3. Simply run the simple Tomcat container of your choice with arguments that "move" the relevant subdirectories to a place on the executing host:
  4. $ docker run --publish 8080:8080                              \
                 --volume /opt/tomcat/webapps:/var/tomcat/webapps \
                 --volume /opt/tomcat/logs:/var/tomcat/logs       \
             { image-name | image-id }
    
  5. Deploy new versions of your web application to the executing host's filesystem in /var/tomcat/webapps. This is easier than messing frequently with Docker images.

  6. Allow logrotate on the executing host to manage /var/tomcat/logs/catalina.out in its filesystem. The first advantage is persistence of logs between bouncing Tomcat instances as well as application upgrades. Second is long-term accessibility to the logs despite frequent bouncing and upgrading.

If you think about it, there are no official/elevated, first-rate Docker containers for Tomcat that include logrotate. Why? Because it's wrong-headed to do this. The better way is what's described above.

This said, to understand Tomcat logging using logrotate, which you'll need to know how to do for the executing host,
see "Using logrotate to roll catalina.out."


Tomcat and Docker notes on memory...

Problem we wish to solve:

This is a little more elaborate. First thing to know is that Tomcat will take up a default of only 25% of the total memory size of your running container. There is a complicated way to intervene (see below), but simpler solutions first.



Running Tomcat as a systemd service...

Most of us run Tomcat as a service on our development host and know to modify /etc/systemd/system/tomcat.service to increase or decrease the stack and/or heap.



Running in a Docker container and using setenv.sh...

However, when Tomcat is run in a Docker container, running as a service is not irrelevant. What's going on in there is Tomcat running in the same way it would if you downloaded it to your filesystem, then executed tomcat/bin/catalina.sh.

In bin/setenv.sh, if you have added this file (it doesn't not come with the Tomcat distribution), you probably have the following two lines that everyone puts into this file (and which Tomcat uses if it's there):

bin/setenv.sh:
set JAVA_HOME="path to JRE/JDK"
set JAVA_OPTS="-Xmsstack-size -Xmxheap-size

Consequently, the container as built must grant access to bin/setenv.sh in order to be able to modify it. This is probably not already done. Here's an example of how to do that (highlighted lines) using a setenv.sh sitting right next to Dockerfile in the build directory:

FROM   registry.gitlab.com/perfectsearch/registry/tomcat:9.0.58-jdk8-openjdk-slim
COPY   setenv.sh $CATALINA_HOME/bin/setenv.sh
RUN    chmod +x $CATALINA_HOME/bin/setenv.sh
EXPOSE 8080
CMD    [ "catalina.sh", "run" ]


Best solution?

You can edit setenv.sh in the container to make changes as above (ugly, cumbersome) or you can leave off doing that and simply modify the docker run command you use to launch Tomcat:

$ docker run --env JAVA_OPTS="-Xmsnew-stack-size -Xmxnew-heap-size" ...

This has the added advantage of allowing you to deploy most any garden-variety Tomcat container image you'll find laying around (on DockerHub, etc.)—no need to roll your own.



That sophisticated, but complicated way of intervening...

...in how Docker container memory allocation is consumed when running a JVM-based application like Tomcat. What does this buy you? Better diagnostics and maybe a little more control

$ docker run --rm tomcat:9.0.44-jdk11-openjdk java -XX:+PrintFlagsFinal -version | grep UseContainerSupport

This is to validate that the image, the JVM and Tomcat support the ability (any later version of these should suffice since this was back-ported in the Java 10 era to Java 8) to make changes to memory settings.

However, it appears that the memory settings are a percentage of the memory established for the container. This is done (minimally—for there are more settings) thus:

$ docker run --memory=size

Thereafter, you can augment the docker run command with the following (example):

$ docker run --rm                                         \
             --name { image-name | image-id }             \
             --memory 24GB                                \
             --env JAVA_OPTS="-XX:InitialRAMPercentage=10 \
                              -XX:MinRAMPercentage=50     \
                              -XX:MaxRAMPercentage=75"

This will give you a container hogging 24Gb of the executing host's memory.

Thereupon, to observe the details, issue the following command:

$ docker run --rm --name { image-name | image-id } \
                         java -XX:+PrintFlagsFinal \
                              -version | grep -E "UseContainerSupport | InitialRAMPercentage | MaxRAMPercentage | MinRAMPercentage"
>> InitialRAMPercentage = some figure
>> MaxRAMPercentage     = some figure
>> MinRAMPercentage     = some figure
>> UseContainerSupport  = true

It is important to know that...

InitialRAMPercentage sets the initial heap size to a percentage of our overall Tomcat container's memory.
MinRAMPercentage sets the minimum heap size to a percentage of our overall Tomcat container's memory.
MaxRAMPercentage sets the maximum heap size to a percentage of our overall Tomcat container's memory.