Apache Tomcat

Russell Bateman
May 2021

Apache Tomcat installation on Linux as a service

There's a bit more clever and adaptable way of installing Tomcat described here:
A Step by Step Guide to Installing Apache Tomcat 9 Linux. It's what I will follow next time.

What I did and notes on same. This was my recently built—Tomcat not yet installed—personal development host, Linux Mint 20.1 (Ulyssa). This more or less follows How to Install Apache Tomcat 9 on Ubuntu 18.04 enforcing my own agenda.

Sadly, I don't have a Red Hat (Fedora, CentOS, etc.) platform so you'll have to use your imagination—yum, etc. to follow these instructions. I may come back and make a few notes to help with that. See Red Hat platform cribs.

As of this writing, IntelliJ IDEA doesn't support Tomcat 10 very well. See Simple Web Application.

    Install and set up Java
  1. Install Java; Apache Tomcat is written in Java and depends upon it also to execute Java applications (and servlets). First, update the package manager, then install what your distribution considers as its default JDK:
    russ@tirion: ~$ sudo apt-get update
    Hit:1 http://archive.canonical.com/ubuntu focal InRelease
    Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
    Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
    Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
    ...
    russ@tirion: ~$ sudo apt-get install default-jdk
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following additional packages will be installed:
      default-jdk-headless libice-dev libsm-dev libxt-dev openjdk-11-jdk openjdk-11-jdk-headless
    The following NEW packages will be installed:
      default-jdk default-jdk-headless libice-dev libsm-dev libxt-dev openjdk-11-jdk openjdk-11-jdk-headless
    0 upgraded, 7 newly installed, 0 to remove and 2 not upgraded.
    Need to get 225 MB of archives.
    After this operation, 236 MB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    ...
    
  2. Create group tomcat and user tomcat
  3. Create group and user.
    russ@tirion /opt $ sudo groupadd tomcat
    russ@tirion /opt $ sudo useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
    
  4. Download and install Apache Tomcat
  5. Download Apache Tomcat 9 from here. The download you want is a tarball named, tar.gz.
    russ@tirion /opt $ sudo mv ~/Downloads/apache-tomcat-9.0.45.tar.gz .
    
  6. Prepare /opt and explode Tomcat there.
    russ@tirion /opt $ sudo bash
    root@tirion:/opt# mkdir tomcat
    russ@tirion /opt# mv /home/russ/Downloads/apache-tomcat-9.0.45.tar.gz .
    root@tirion:/opt# tar -zxf apache-tomcat-9.0.45.tar.gz -C ./tomcat --strip-components=1
    
  7. Set up Apache Tomcat
  8. Fix up permissions.
    root@tirion:/opt# chgrp -R tomcat ./tomcat
    root@tirion:/opt# ll tomcat
    total 156
    drwxr-xr-x 9 root tomcat  4096 May 11 10:27 ./
    drwxr-xr-x 7 root root    4096 May 11 10:27 ../
    drwxr-x--- 2 root tomcat  4096 May 11 10:27 bin/
    -rw-r----- 1 root tomcat 18984 Mar 30 04:29 BUILDING.txt
    drwx------ 2 root tomcat  4096 Mar 30 04:29 conf/
    -rw-r----- 1 root tomcat  5587 Mar 30 04:29 CONTRIBUTING.md
    drwxr-x--- 2 root tomcat  4096 May 11 10:27 lib/
    -rw-r----- 1 root tomcat 57092 Mar 30 04:29 LICENSE
    drwxr-x--- 2 root tomcat  4096 Mar 30 04:29 logs/
    -rw-r----- 1 root tomcat  2333 Mar 30 04:29 NOTICE
    -rw-r----- 1 root tomcat  3257 Mar 30 04:29 README.md
    -rw-r----- 1 root tomcat  6898 Mar 30 04:29 RELEASE-NOTES
    -rw-r----- 1 root tomcat 16507 Mar 30 04:29 RUNNING.txt
    drwxr-x--- 2 root tomcat  4096 May 11 10:27 temp/
    drwxr-x--- 7 root tomcat  4096 Mar 30 04:29 webapps/
    drwxr-x--- 2 root tomcat  4096 Mar 30 04:29 work/
    
  9. Give group tomcat read access to the conf subdirectory, all of its contents, and execute access to the directory itself.
    root@tirion:/opt# cd tomcat
    root@tirion:/opt/tomcat# chmod -R g+r conf
    root@tirion:/opt/tomcat# chmod g+rxw conf
    root@tirion:/opt/tomcat# ll conf
    total 240
    drwxr-x--- 2 root tomcat   4096 Mar 30 04:29 ./
    drwxr-xr-x 9 root tomcat   4096 May 11 10:27 ../
    -rw-r----- 1 root tomcat  12873 Mar 30 04:29 catalina.policy
    -rw-r----- 1 root tomcat   7262 Mar 30 04:29 catalina.properties
    -rw-r----- 1 root tomcat   1400 Mar 30 04:29 context.xml
    -rw-r----- 1 root tomcat   1149 Mar 30 04:29 jaspic-providers.xml
    -rw-r----- 1 root tomcat   2313 Mar 30 04:29 jaspic-providers.xsd
    -rw-r----- 1 root tomcat   4144 Mar 30 04:29 logging.properties
    -rw-r----- 1 root tomcat   7588 Mar 30 04:29 server.xml
    -rw-r----- 1 root tomcat   2164 Mar 30 04:29 tomcat-users.xml
    -rw-r----- 1 root tomcat   2558 Mar 30 04:29 tomcat-users.xsd
    -rw-r----- 1 root tomcat 172359 Mar 30 04:29 web.xml
    
    Note: /opt/tomcat/conf is canonical, but Red Hat installations appear to do something different here by creating a tomcat.conf file to contain Tomcat configuration instead of in a subdirectory. I don't know if this is still practiced today. Your mileage interpreting this comment may vary vary.

  10. Make user tomcat the ower of the webapps, work, temp, logs and everything else:
    root@tirion:/opt/tomcat# chown -R tomcat *
    root@tirion:/opt/tomcat# ll webapps/
    total 28
    drwxr-x---  7 tomcat tomcat 4096 Mar 30 04:29 ./
    drwxr-xr-x  9 root   tomcat 4096 May 11 10:27 ../
    drwxr-x--- 15 tomcat tomcat 4096 May 11 10:27 docs/
    drwxr-x---  7 tomcat tomcat 4096 May 11 10:27 examples/
    drwxr-x---  6 tomcat tomcat 4096 May 11 10:27 host-manager/
    drwxr-x---  6 tomcat tomcat 4096 May 11 10:27 manager/
    drwxr-x---  3 tomcat tomcat 4096 May 11 10:27 ROOT/
    (etc.)
    
  11. Create the systemd service
  12. You will need to know where your installed Java is in order to create a valid JAVA_HOME environment variable for Tomcat:
    root@tirion:/opt/tomcat# sudo update-java-alternatives -l
    java-1.11.0-openjdk-amd64      1111       /usr/lib/jvm/java-1.11.0-openjdk-amd64
    
  13. Using this, we'll edit the systemd service file, /etc/systemd/system/tomcat.service, which doesn't exist yet. Make it look like this:
    [Unit]
    Description=Apache Tomcat Web Application Container
    After=network.target
    
    [Service]
    Type=forking
    
    Environment=JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64
    Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
    Environment=CATALINA_HOME=/opt/tomcat
    Environment=CATALINA_BASE=/opt/tomcat
    Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
    Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'
    
    ExecStart=/opt/tomcat/bin/startup.sh
    ExecStop=/opt/tomcat/bin/shutdown.sh
    
    User=tomcat
    Group=tomcat
    UMask=0007
    RestartSec=10
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    
    Use your favorite editor:
    root@tirion:/opt/tomcat# gvim /etc/systemd/system/tomcat.service
    
  14. Load the systemd service and launch Tomcat
  15. Reload the systemd dæmon so it discovers our new service, and start Tomcat. Then, double-check that it launched without errors:
    root@tirion:/opt/tomcat# systemctl daemon-reload
    root@tirion:/opt/tomcat# systemctl start tomcat
    root@tirion:/opt/tomcat# systemctl status tomcat
     tomcat.service - Apache Tomcat Web Application Container
         Loaded: loaded (/etc/systemd/system/tomcat.service; disabled; vendor preset: enabled)
         Active: active (running) since Tue 2021-05-11 11:31:50 MDT; 9s ago
        Process: 1189015 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
       Main PID: 1189036 (java)
          Tasks: 41 (limit: 38230)
         Memory: 187.5M
         CGroup: /system.slice/tomcat.service
                 └─1189036 /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf>
    
    May 11 11:31:50 tirion systemd[1]: Starting Apache Tomcat Web Application Container...
    May 11 11:31:50 tirion startup.sh[1189015]: Tomcat started.
    May 11 11:31:50 tirion systemd[1]: Started Apache Tomcat Web Application Container.
    
  16. As a second verification, we can point a browser to http://localhost:8080.

  17. Firewall and web-management configuration...
  18. You can configure your firewall if you plan on using Apache Tomcat "for real" allowing clients outside your local computer to access it. That's not my purpose, I'm no DevOp, so I will not illustrate this since I don't need it.
  19. Also, you can configure Tomcat's Web Management Interface with manager and administrator log-ins. This is also not an interest of mine.

Install Ubuntu 22.04 Server

root@russ-microservices:/home/russ# apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Hit:5 https://download.docker.com/linux/ubuntu jammy InRelease
Fetched 210 kB in 10s (20.5 kB/s)
Reading package lists... Done
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
root@russ-microservices:/home/russ# apt-get install default-jdk
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  alsa-topology-conf alsa-ucm-conf at-spi2-core ca-certificates-java dconf-gsettings-backend dconf-service
  default-jdk-headless default-jre default-jre-headless fontconfig-config fonts-dejavu-core fonts-dejavu-extra
  gsettings-desktop-schemas java-common libasound2 libasound2-data libatk-bridge2.0-0 libatk-wrapper-java
  libatk-wrapper-java-jni libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data
  libavahi-common3 libcups2 libdconf1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontconfig1
  libfontenc1 libgif7 libgl1 libgl1-amber-dri libgl1-mesa-dri libglapi-mesa libglvnd0 libglx-mesa0 libglx0
  libgraphite2-3 libharfbuzz0b libice-dev libice6 libjpeg-turbo8 libjpeg8 liblcms2-2 libllvm13 libpciaccess0
  libpcsclite1 libpthread-stubs0-dev libsensors-config libsensors5 libsm-dev libsm6 libvulkan1 libwayland-client0
  libx11-dev libx11-xcb1 libxau-dev libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0
  libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcb1-dev libxcomposite1 libxdmcp-dev libxfixes3 libxft2
  libxi6 libxinerama1 libxkbfile1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6
  libxv1 libxxf86dga1 libxxf86vm1 mesa-vulkan-drivers openjdk-11-jdk openjdk-11-jdk-headless openjdk-11-jre
  openjdk-11-jre-headless session-migration x11-common x11-utils x11proto-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
  libasound2-plugins alsa-utils cups-common libice-doc liblcms2-utils pcscd lm-sensors libsm-doc libx11-doc libxcb-doc
  libxt-doc openjdk-11-demo openjdk-11-source visualvm libnss-mdns fonts-ipafont-gothic fonts-ipafont-mincho
  fonts-wqy-microhei | fonts-wqy-zenhei fonts-indic mesa-utils
The following NEW packages will be installed:
  alsa-topology-conf alsa-ucm-conf at-spi2-core ca-certificates-java dconf-gsettings-backend dconf-service default-jdk
  default-jdk-headless default-jre default-jre-headless fontconfig-config fonts-dejavu-core fonts-dejavu-extra
  gsettings-desktop-schemas java-common libasound2 libasound2-data libatk-bridge2.0-0 libatk-wrapper-java
  libatk-wrapper-java-jni libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data
  libavahi-common3 libcups2 libdconf1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontconfig1
  libfontenc1 libgif7 libgl1 libgl1-amber-dri libgl1-mesa-dri libglapi-mesa libglvnd0 libglx-mesa0 libglx0
  libgraphite2-3 libharfbuzz0b libice-dev libice6 libjpeg-turbo8 libjpeg8 liblcms2-2 libllvm13 libpciaccess0
  libpcsclite1 libpthread-stubs0-dev libsensors-config libsensors5 libsm-dev libsm6 libvulkan1 libwayland-client0
  libx11-dev libx11-xcb1 libxau-dev libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0
  libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcb1-dev libxcomposite1 libxdmcp-dev libxfixes3 libxft2
  libxi6 libxinerama1 libxkbfile1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6 libxtst6
  libxv1 libxxf86dga1 libxxf86vm1 mesa-vulkan-drivers openjdk-11-jdk openjdk-11-jdk-headless openjdk-11-jre
  openjdk-11-jre-headless session-migration x11-common x11-utils x11proto-dev xorg-sgml-doctools xtrans-dev
0 upgraded, 102 newly installed, 0 to remove and 17 not upgraded.
Need to get 306 MB of archives.
After this operation, 597 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 alsa-topology-conf all 1.2.5.1-2 [15.5 kB]
.
.
.
Setting up default-jdk (2:1.11-72build2) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...
Running kernel seems to be up-to-date.
The processor microcode seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@russ-microservices:/opt# groupadd tomcat
root@russ-microservices:/opt# useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
root@russ-microservices:/opt# mv /home/russ/Downloads/apache-tomcat-9.0.64.tar.gz .
root@russ-microservices:/opt# mkdir tomcat
root@russ-microservices:/opt# tar -zxf apache-tomcat-9.0.64.tar.gz -C ./tomcat --strip-components=1
root@russ-microservices:/opt# ll tomcat/
total 156
drwxr-xr-x 9 root root  4096 Jun 21 21:40 ./
drwxr-xr-x 4 root root  4096 Jun 21 21:39 ../
drwxr-x--- 2 root root  4096 Jun 21 21:40 bin/
-rw-r----- 1 root root 18986 Jun  2 19:08 BUILDING.txt
drwx------ 2 root root  4096 Jun  2 19:08 conf/
-rw-r----- 1 root root  6210 Jun  2 19:08 CONTRIBUTING.md
drwxr-x--- 2 root root  4096 Jun 21 21:40 lib/
-rw-r----- 1 root root 57092 Jun  2 19:08 LICENSE
drwxr-x--- 2 root root  4096 Jun  2 19:08 logs/
-rw-r----- 1 root root  2333 Jun  2 19:08 NOTICE
-rw-r----- 1 root root  3398 Jun  2 19:08 README.md
-rw-r----- 1 root root  6901 Jun  2 19:08 RELEASE-NOTES
-rw-r----- 1 root root 16505 Jun  2 19:08 RUNNING.txt
drwxr-x--- 2 root root  4096 Jun 21 21:40 temp/
drwxr-x--- 7 root root  4096 Jun  2 19:08 webapps/
drwxr-x--- 2 root root  4096 Jun  2 19:08 work/
root@russ-microservices:/opt# rm *.gz
root@russ-microservices:/opt# chgrp -R tomcat ./tomcat
root@russ-microservices:/opt# cd tomcat
root@russ-microservices:/opt/tomcat# chown -R tomcat ./webapps/ ./work/ ./temp/ ./logs/
root@russ-microservices:/opt/tomcat# ll webapps
total 28
drwxr-x---  7 tomcat tomcat 4096 Jun  2 19:08 ./
drwxr-xr-x  9 root   tomcat 4096 Jun 21 21:40 ../
drwxr-x--- 15 tomcat tomcat 4096 Jun 21 21:40 docs/
drwxr-x---  7 tomcat tomcat 4096 Jun 21 21:40 examples/
drwxr-x---  6 tomcat tomcat 4096 Jun 21 21:40 host-manager/
drwxr-x---  6 tomcat tomcat 4096 Jun 21 21:40 manager/
drwxr-x---  3 tomcat tomcat 4096 Jun 21 21:40 ROOT/
root@russ-microservices:/opt/tomcat# update-java-alternatives -l
java-1.11.0-openjdk-amd64      1111       /usr/lib/jvm/java-1.11.0-openjdk-amd64
root@russ-microservices:/opt/tomcat# vim /etc/systemd/system/tomcat.service (You're adding the following lines:)
[Unit] Description=Apache Tomcat Web Application Container After=network.target [Service] Type=forking Environment=JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64 Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid Environment=CATALINA_HOME=/opt/tomcat Environment=CATALINA_BASE=/opt/tomcat Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC' Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom' ExecStart=/opt/tomcat/bin/startup.sh ExecStop=/opt/tomcat/bin/shutdown.sh User=tomcat Group=tomcat UMask=0007 RestartSec=10 Restart=always [Install] WantedBy=multi-user.target
root@russ-microservices:/opt/tomcat# systemctl daemon-reload root@russ-microservices:/opt/tomcat# systemctl start tomcat root@russ-microservices:/opt/tomcat# systemctl status tomcat tomcat.service - Apache Tomcat Web Application Container Loaded: loaded (/etc/systemd/system/tomcat.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2022-06-21 21:45:55 UTC; 959ms ago Process: 7148 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS) Main PID: 7158 (code=exited, status=1/FAILURE) CPU: 387ms

Installing on Red Hat (CentOS 8)...

I'll drop these here mostly as cribs. dnf, by the way, is the new yum. Some people seem to mix them. As a Debian guy, I don't care. We have our own idiosyncrasies.

Important notes for developers...



Tomcat Notes

Started back in 2012 or so, this is a little lame. It was really a place to put notes until I got better organized.

A good link for installing Tomcat (and its issues) on Ubuntu Server is https://help.ubuntu.com/10.04/serverguide/tomcat.html


Undeploying from Tomcat

Don't remove the subdirectory resulting from the deployment. Just remove the WAR file and let Tomcat undeploy it. Doing this wrong leads confusingly to errors in catalina.out.


servlet-api.jar
    [Tomcat] validateJarFile(servlet-api.jar) - jar not loaded. Offending class: javax/servlet/Servlet.class

    org.apache.catalina.loader.WebappClassLoader validateJarFile
    INFO: validateJarFile(\WEB-INF\lib\servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, \
        section 9.7.2. Offending class: javax/servlet/Servlet.class.

Note that servlet-api.jar already exists in Tomcat. This message isn't anything more than a warning, but it's annoying. It means simply that you've included another instance of servlet-api.jar in your build path (see Eclipse Build Path or the .classpath file). Simply remove this JAR from your build letting Tomcat supply it.


APR Apache Tomcat Native library

You see this starting Tomcat. It's no more than a warning. In order to overcome it, you'd have to go to some trouble. Google for it.

    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production \
        environments was not found on the java.library.path: \
        /home/russ/dev/jdk1.6.0_31/jre/lib/amd64/server:/home/russ/dev/jdk1.6.0_31/jre/lib/amd64:\
        /home/russ/dev/jdk1.6.0_31/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

See http://stackoverflow.com/questions/8716259/what-does-the-apr-based-apache-tomcat-native-library-was-not-found-mean.

What this means is:

  1. You shouldn't worry about this during development; it's just a warning.
  2. If this is a concern in production, and perhaps it should be, the only thing that can be done about it is to download Apache code and build a libnative.so for the platform that's going to execute your production application. See http://tomcat.apache.org/tomcat-6.0-doc/apr.html#Installation.

SSL and Tomcat's server.xml

Tomcat's server.xml file is amended thus in the face of implementing HTTPS. The part in bold type is what's added to an otherwise stock server.xml:

    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443" />

    <!-- We added this next paragraph in recognition that Big IP will send
         secure (formerly encrypted) requests through port 8081 and calling
         ContainerRequest.isSecure() will return true.
    -->
    <Connector port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               scheme="https"
               secure="true"
    />

If Tomcat's handling this by itself, in the situation where there is no Big IP or load-balancer, the traffic comes in encrypted over 443 (instead of 8081) and Tomcat decrypts it.


SSL and Tomcat's server.xml (part 2)

There is a second part to this equation. The following section(s), if present, must be removed (commented out) from server.xml:

     <!-- Define a SSL HTTP/1.1 Connector on port 8443
          This connector uses the JSSE configuration, when using APR, the
          connector should be using the OpenSSL style configuration
          described in the APR documentation -->
     <!-- Comment this out as we don't use SSL; instead we get secure traffic over 8081
     <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
                maxThreads="150" scheme="https" secure="true"
                clientAuth="false" sslProtocol="TLS" />
     -->

The lines being commented out come by default in the stock Tomcat server.xml. The traffic is unencrypted by the load balancer and passed over port 8081 instead of 8443 and we want to stop listening on port 8443 which will otherwise give us keystore errors like:

    Aug 09, 2013 4:55:12 PM org.apache.tomcat.util.net.jsse.JSSESocketFactory getStore
    SEVERE: Failed to load keystore type JKS with path /usr/share/tomcat6/.keystore due to /usr/share/tomcat6/.keystore (No such file or directory)
    java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at java.io.FileInputStream.open(Native Method)
            at java.io.FileInputStream.(FileInputStream.java:138)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:405)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:296)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:544)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:481)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:156)
            at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
            at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:176)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1049)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

    Aug 09, 2013 4:55:12 PM org.apache.coyote.http11.Http11Protocol init
    SEVERE: Error initializing endpoint
    java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at java.io.FileInputStream.open(Native Method)
            at java.io.FileInputStream.(FileInputStream.java:138)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:405)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:296)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:544)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:481)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:156)
            at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
            at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:176)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1049)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

    Aug 09, 2013 4:55:12 PM org.apache.catalina.core.StandardService initialize
    SEVERE: Failed to initialize connector [Connector[HTTP/1.1-8443]]
    LifecycleException:  Protocol handler initialization failed: java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1051)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
            ...

How to use a Sun JRE with Tomcat...

This assumes that on an Ubuntu Precise Server installation, only the OpenJDK is installed, which it is by default. This example assumes Java 1.6, call it jdk1.6.0_38 for the sake of our illustration.

There are reasons you want to do this. One is the ability to run more competent encryption/hashing.

First, you can see which JVM is being used by Tomcat via this command:

    $ ps -ef | grep [t]omcat

Likely, it will be /usr/lib/jvm/default-java/bin/java. If you list this path, you'll see that it's the non-Sun OpenJDK.

    $ ll /usr/lib/jvm/default-java
    lrwxrwxrwx 1 root root 24 Jan 29 06:30 /usr/lib/jvm/default-java -> java-1.6.0-openjdk-amd64/
  1. Download the Sun JRE or JDK from http://java.sun.com; download the .bin file. At time of writing, once you've reached this page, you'd hover over DOWNLOADS and choose Java for Developers under "Popular Downloads".
  2. Copy the .bin over to your server.
  3. Copy the .bin file to /usr/local, make it executable and explode it.
  4. Go to /usr/local.
  5. Create a new link, jdk1.6.0_38 to point at the new JDK/JRE or just move the whole JDK/JRE to this path.
  6. Own the exploded download (chown -R root:root jdk1.6.0_38).
  7. Go to /usr/lib/jvm.
  8. Create a link named java-6-oracle to /usr/local/jdk1.6.0_38.
  9. Create a new link named default-java (or, replace the one that's there) to point at ./java-6-oracle. This is the key to getting Tomcat to use it (examine /etc/init.d/tomcat6, search for "default-java" and you'll understand).

At this point, bounce Tomcat and you'll see that it's running atop the real, Sun JDK/JRE.

    $ /etc/init.d/tomcat6 restart
    $ ps -ef | grep [t]omcat
    (stuff similar to what you saw before except...)
    $ ll /usr/lib/jvm/default-java
    lrwxrwxrwx 1 root root 24 Jan 29 06:30 /usr/lib/jvm/default-java -> java-6-oracle/

Tomcat-deployed application behaves peculiarly (load-balancing)...

If one leg of your load-balanced application is not up, you could experience peculiarities like missing CSS, images, etc.


See if Tomcat is holding port 8080

To see whether Tomcat is running on 8080, do this:

    root@app-1:# lsof -i :8080
    COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
    java    916 tomcat6   33u  IPv6   7764      0t0  TCP *:http-alt (LISTEN)

Log files too big, flooding disk, etc....

If you're out of space or even catalina.out gets much over 500Mb, you could fail to deploy new or redeploy existing applications because of it.

Good rules of thumb for this configuration are:


JAVA_HOME for Tomcat

There's lots of idiocy out there when you look for this. Most of it comes from a Windoz orientation. However, others seem to answer not knowing that the user tomcat6 isn't something you can "become" (via su), have a "home" directory, etc.

At least for Ubuntu and Mint, the place to look is /etc/default/tomcat6, a file that contains a lot of settings or commented-out potential settings.

If you want to switch what version of Java your Tomcat installation is running on, there are two, good ways. Let's imagine you want to begin running Java 7 instead of 6.

  1. As just alluded, tweak the value for JAVA_HOME in /etc/default/tomcat6 to point at the Java 7 you installed.
  2.  
  3. Determine what version of Java is available and where it's running. You should see something like this:
        # which java
        /usr/bin/java
        # java -version
        java version "1.7.0_21"
        OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-0ubuntu0.12.04.1)
        OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
    
     
  4. Install, e.g.: openjdk7, then uninstall openjdk6. Don't touch any definitions of JAVA_HOME anywhere. This is the approach to take if you're dealing with a VM running nothing but Tomcat (and your web application suite).
        # java -version
        java version "1.6.0_27"
        OpenJDK Runtime Environment (IcedTea6 1.12.5) (6b27-1.12.5-0ubuntu0.12.04.1)
        OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
        # apt-get update
        # apt-get install openjdk-7-jre
        (installs latest, known Java 7)
        # apt-get purge openjdk-6-jre-headless
        (removes Java 6 and dependencies; lose system notion of Java)
    

    Ordinarily, however, on Ubuntu server systems I've installed using mostly defaults I have not had anything to do past this point.

  5. You should examine /etc/alternatives to see if everything is good.
        # cd /etc/alternatives
        /etc/alternatives # ln -s /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/javaws*
        (updates system notion of Java which was lost)
        # java -version
        java version "1.7.0_21"
        OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-0ubuntu0.12.04.1)
        OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
    

    If everything is okay, you should see only Java 7 here:

        root@taliesin:/etc/alternatives# ll java*
        lrwxrwxrwx 1 root root 46 Nov 12 21:01 java -> /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java*
        lrwxrwxrwx 1 root root 56 Nov 12 21:01 java.1.gz -> /usr/lib/jvm/java-7-openjdk-amd64/jre/man/man1/java.1.gz
    

    If this wasn't enough, note that replacing Java 6 with 7 is not for the faint-hearted. You need to do more than simply install openjdk/jre 7 or Sun Java 7. You also have to update the Java "alternatives" to make it look more or less like the above. (Or see below for a more twisted result.)

* Despite complaints and claims to the contrary out there, there is no way update-java-alternatives will update the javaws links; you have to do it by hand. However, you need not do it if this link was not there in the first place.

More notes

On occasion, the above is insufficient and you can still see this in the Tomcat Web Application Manager, the wrong JVM:

(To get into the manager go to http://hostname:port/manager/html. If you don't know how to respond to the authentication dialog, see below.)

I thought I had laid to rest the question of which JVM would hold sway on the server host. Following is a picture I hope will be a thousand words. Please note especially the lines in bold which describe the state of things after I fixed the remaining trouble.

root@acme:/usr/lib/jvm# ll
total 24
drwxr-xr-x  5 root root 4096 Nov  6 16:06 ./
drwxr-xr-x 64 root root 4096 Nov  6 14:18 ../
lrwxrwxrwx  1 root root   21 Nov  6 16:06 default-java -> java-7-openjdk-amd64//
lrwxrwxrwx  1 root root   20 Jul 15 19:46 java-1.7.0-openjdk-amd64 -> java-7-openjdk-amd64/
-rw-r--r--  1 root root 2439 Jul 15 19:46 .java-1.7.0-openjdk-amd64.jinfo
lrwxrwxrwx  1 root root   15 Jan 25  2013 java-6-sun -> /usr/local/java/
drwxr-xr-x  5 root root 4096 Nov  6 14:17 java-7-openjdk-amd64/
drwxr-xr-x  3 root root 4096 Nov  6 14:17 java-7-openjdk-common/
drwxr-xr-x  2 root root 4096 Feb 26  2012 java-7-oracle/
lrwxrwxrwx  1 root root   10 Jan 25  2013 old-default-java -> java-6-sun/

Links to peruse:


Application deployed, but Tomcat gives back HTTP Status 404 Not Found

There are many reasons this can happen. One that might escape is that the application was deployed (i.e.: it shows up in /var/lib/tomcat6/webapps), but it failed to start up. Examine the log, /var/log/tomcat6/catalina.out to see why. Perhaps you were trying to connect to your database node(s), but had forgotten to update the application server's /etc/hosts file with the mappings to those hosts and they aren't know to DNS. Etc.


The SEVERE: Error filterStart mess...

In catalina.out, you might find mysteriously, one of...

    SEVERE: Error filterStart
    SEVERE: Error listenerStart

...when you try to start your application. By default, Tomcat won't tell you intelligently what's happening. It won't even tell you which filter or listener is failing. The solution is very simple:

  1. Under Tomcat's webapps folder, under your deployed application, create WEB_INF/class/logging.properties—even if you've already got log4j.properties, etc. there.
  2. Add the following content to this file:
        org.apache.catalina.core.ContainerBase.[Catalina].level = ALL
        org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler
    
  3. Clear out catalina.out if you like to squint through less output.
  4. Bounce Tomcat:
        $ service tomcat6 restart
    

You should see a proper stacktrace with all the verbosity you need to sort this problem out.


Useless ${catalina.home} property...

...after installing Tomcat 6. Old Apache log4j like Apache Tomcat 5.5 Servlet/JSP Container: Logging in Tomcat says to use statements like

    log4j.appender.R.File=${catalina.home}/logs/myapp.log

...where myapp.log is the name you invent for the logfile different from catalina.out.

This won't work, at very least, on Debian platforms where ${catalina.home} points at /usr/share/tomcat6, which has no logs subdirectory or link to /var/log/tomcat6 where catalina.out lives. Use ${catalina.base} instead; it points at /var/lib/tomcat6 which has logs as a symbolic link to /var/log/tomcat6.

When you see something like...

    log4j:ERROR setFile(null,true) call failed.
    java.io.FileNotFoundException: /usr/share/tomcat6/logs/myapp.log (No such file or directory)
            at java.io.FileOutputStream.open(Native Method)
            at java.io.FileOutputStream.(FileOutputStream.java:212)
            at java.io.FileOutputStream.(FileOutputStream.java:136)
            at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
            at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
            at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
            at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
            at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
            at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
            at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
            at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
            at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
            at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
            at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
            at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
            at org.apache.log4j.LogManager.(LogManager.java:127)
            at org.apache.log4j.Logger.getLogger(Logger.java:117)
            at com.hp.web.util.ApplicationProperties.(ApplicationProperties.java:17)
            at com.hp.web.controller.AppInitializer.contextInitialized(AppInitializer.java:20)
            at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
            at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
            at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
            at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
            at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
            at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
            at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
            at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
            at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)
            at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:306)
            at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
            at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1389)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1653)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1662)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1642)
            at java.lang.Thread.run(Thread.java:722)

...it's because Tomcat is unable to create /usr/share/tomcat/logs/myapp.log. This is because the path to it doesn't exist.


Getting into the Tomcat Web Application Manager

To get into the manager go to http://hostname:port/manager/html. If you don't know what to respond to the authentication dialog, it's because /etc/tomcat6/tomcat-users.xml has nothing in it. You'll want to add...

        .
        .
        .
        <role rolename="manager"/>
        <role rolename="manager-gui"/>
        <role rolename="admin"/>
        <user username="user" password="password" roles="admin,manager,manager-gui"/>
    </tomcat-users>

...inside the <tomcat-users> tag. Replace user and password with something a little more secure.


How to make a web application Tomcat's root application

Add a file called ROOT.xml in $CATALINA_HOME/conf/Catalina/localhost/

ROOT.xml will override the default settings for the root context of the Tomcat installation for that engine and host (Catalina and localhost).

Enter the following to the ROOT.xml file:

<Context docBase="yourApp" path="" reloadable="true" />

Here, yourApp is the name of your application.

And there you go, your application is now the default application and will show up on http://localhost:8080.

However, there is one side effect; your application will be loaded twice. Once for localhost:8080 and once for localhost:8080/yourApp. To fix this you can put your application OUTSIDE $CATALINA_HOME/webapps and use a relative or absolute path in the ROOT.xml's docBase tag. Something like this:

<Context docBase="/opt/mywebapps/yourApp" path="" reloadable="true" />

Tomcat context

Individual Context elements may be explicitly defined:


Tomcat 6 access log example

In Tomcat's server.xml file, at the bottom, you can awaken ('cause it ships commented out) the access log by uncommenting and configuring it. Here's something I did one day, with consideration for Tomcat 7 and solving a problem explained in the additional comment:

  <!-- Access log processes all example.
       Documentation at: /docs/config/valve.html -->
  <!--
       Note: this is uncommented giving us 1 second granularity in the
       log entries, see
       http://tomcat.apache.org/tomcat-6.0-doc/config/valve.html

       Millisecond granularity will (only) come after moving to Tomcat 7.
       The "common" pattern, even in Tomcat 7, does not include this, so
       it will explicitly become "%h %l %u %{msec}t "%r" %s %b". See
       https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html
    -->
  <Valve className="org.apache.catalina.valves.AccessLogValve"
         directory="/home/fs/log"
         prefix="tomcat-access."
         suffix="log"
         pattern="%h %{x-forwarded-for}i; %H %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %Dms"
         resolveHosts="false"
         rotatable="false"
         checkExists="true"
              />

Debugging the dreaded "SEVERE: ERROR LISTENERSTART"
"SEVERE: ERROR LISTENERSTART" AND "SEVERE: ERROR FILTERSTART" TOMCAT ERROR MESSAGES

The errors being sparse, I add logging.properties to src with contents:

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

Forcing Tomcat to use NIO...

From a certain Faheem Sohail...

In HTTP 1.1, all connections between the browser and the server are considered persistent unless declared otherwise. Persistence, in this context, means to use a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair.

In Tomcat, the default HTTP connector is blocking and follows a one thread per connection model. This means that in order to serve 100 concurrent users, it requires 100 active threads. We end up wasting resources (the thread) because connections may not be used heavily, but just enough to avoid a timeout.

Opposed to this is the relatively new NIO or non blocking connector. This connector has a couple of poller threads used to keep the connection alive for all connected users while worker threads are called whenever data (a new HTTP request) is available. This model leads to a much better sharing of resources (threads) and a larger number of concurrent users can be served from the same server.

In order to configure tomcat to use the non-blocking NIO connector instead of the default blocking BIO one simply change the value of the protocol attribute of the connector tag in the server.xml from HTTP/1.1 to org.apache.coyote.http11.Http11NioProtocol.


To verify that you indeed are using the NIO connector, take a look at the start-up logs. You should see lines similar to this.

Mar 28, 2014 3:59:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
Mar 28, 2014 3:59:04 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector

Use VisualVM to look at the threads being created in both cases. You’ll find NIO to use threads much more efficiently.


servlet-api.jar needed for JAX-RS use

This is also a good example of how to find the JAR containing a symbol that will satisfy what's missing in your Java code. Note that the convoluted nature of this search, which has the added complication that this symbol is only provided and not linked in (it's Tomcat that will load my servlet and provide the JAR containing the symbol), demonstrates that you must be careful where you get the symbol. www.findjar.com is a very useful site, but you must be certain which JAR is the right source from among the many it often suggests, i.e.: there's a sort of Zen to it that comes with experience in the sort of Java programming that you're doing.

To code to get the incoming request for a POST operation in my restlet, ...

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import javax.servlet.http.HttpServletRequest;

public class Restlet
{
  @POST
  @Consumes( MediaType.APPLICATION_FORM_URLENCODED )
  @Produces( MediaType.TEXT_XML )
  public Response postPatientData( @Context HttpServletRequest request, @Context HttpHeaders header )
  {
    ...
  }

...I needed to consume this symbol from Tomcat. Here's how I determined that:

  1. Google "what jar HttpServletRequest".
  2. http://www.findjar.com/class/javax/servlet/http/HttpServletRequest.html suggested a number of JARs.
  3. I looked in my Tomcat download, ~/dev/apache-tomcat-9.0.7/lib and found servlet-api.jar that matches one of the JARs the find-jar site suggested. This happesn to be the server container I'm using in my restlet, so, I conclude that I'll get HttpServletRequest from there.
  4. I use the (Linux) Nemo filesystem browser to navigate to and open (double-click) ~/dev/apache-tomcat-9.0.7/lib/servlet-api.jar. This launches Archive Manager which shows me inside this JAR without the need to explode it manually. Where it not for this, I'd have to resort to doing what I describe in Viewing the contents of a JAR.
  5. I verify that the symbol, HttpServletRequest, can be found on the package path, javax.servlet.http, by clicking down through these three subdirectories to see HttpServletRequest.class.
  6. I look inside META-INF/MANIFEST.MF for the version of this JAR, 4.0:
    Name: javax/servlet/
    Specification-Title: Java API for Servlets
    Specification-Version: 4.0
    Specification-Vendor: Sun Microsystems, Inc.
    Implementation-Title: javax.servlet
    Implementation-Version: 4.0.FR
    Implementation-Vendor: Apache Software Foundation
    
  7. I Google "maven servlet-api" and see "Maven Repository: javax.servlet >> servlet-api"; I click below on 4.0.0. This gives me the Maven-dependency paragraph:
    <dependency>
      <groupId>javax.servlet</groupId>
      <artifactId>javax.servlet-api</artifactId>
      <version>4.0.0</version>
      <scope>provided</scope>
    </dependency>
    
  8. I paste this into my project's pom.xml and refresh Maven (a function of IntelliJ IDEA) whereupon the red for this symbol disappears.

Apache Tomcat on Linux

Let's look at a rather stand-alone way (not as a service) to set up and run Tomcat under Linux. I'm doing this on Linux Mint (Ubuntu).

  1. For much of this, you'll need to have root access:
    $ sudo bash
    # whoami
    root
    
  2. Add user tomcat to system; I chose tomcat as the password too since this is only a tutorial:
    # adduser tomcat
    Adding user `tomcat' ...
    Adding new group `tomcat' (1003) ...
    Adding new user `tomcat' (1003) with group `tomcat' ...
    Creating home directory `/home/tomcat' ...
    Copying files from `/etc/skel' ...
    Enter new UNIX password: tomcat
    Retype new UNIX password: tomcat
    passwd: password updated successfully
    Changing the user information for tomcat
    Enter the new value, or press ENTER for the default
    	Full Name []: Tomcat
    	Room Number []:
    	Work Phone []:
    	Home Phone []:
    	Other []:
    Is the information correct? [Y/n] Y
    nargothrond ~ # su - tomcat
    tomcat@nargothrond ~ $ pwd
    /home/tomcat
    
  3. Download Tomcat 9 from here. Under Core, click on the tarball.

  4. Explode the tarball:
    tomcat@nargothrond ~ $ tar -zxf apache-tomcat-9.0.8.tar.gz
    tomcat@nargothrond ~ $ ll -d apache*
    drwxrwxr-x 9 tomcat tomcat    4096 Jun 12 14:18 apache-tomcat-9.0.8/
    -rw-rw-r-- 1 tomcat tomcat 9818695 Jun 12 14:08 apache-tomcat-9.0.8.tar.gz
    
  5. Modify ~/.bash_profile to define the Catalina root:
    tomcat@nargothrond ~ $ vi ~/.bash_profile
    
    export CATALINA_HOME=/home/tomcat/apache-tomcat-9.0.8
    
  6. Verify that all this is set up:
    tomcat@nargothrond ~ $ exit
    nargothrond ~ # su - tomcat
    tomcat@nargothrond ~ $ set | grep CATALINA
    CATALINA_HOME=/home/tomcat/apache-tomcat-9.0.8
    
  7. Start Tomcat and verify that it's running:
    tomcat@nargothrond ~ $ $CATALINA_HOME/bin/catalina.sh start
    Using CATALINA_BASE:   /home/tomcat/apache-tomcat-9.0.8
    Using CATALINA_HOME:   /home/tomcat/apache-tomcat-9.0.8
    Using CATALINA_TMPDIR: /home/tomcat/apache-tomcat-9.0.8/temp
    Using JRE_HOME:        /usr
    Using CLASSPATH:       /home/tomcat/apache-tomcat-9.0.8/bin/bootstrap.jar:/home/tomcat/apache-tomcat-9.0.8/bin/tomcat-juli.jar
    Tomcat started.
    tomcat@nargothrond ~ $ ps -ef | grep -i [t]omcat
    root     26366 24746  0 14:28 pts/3    00:00:00 su - tomcat
    tomcat   26374 26366  0 14:28 pts/3    00:00:00 -su
    tomcat   26459     1  2 14:31 pts/3    00:00:03 /usr/bin/java -Djava.util.logging.config.file=/home/tomcat/apache-tomcat-9.0.8/co...
    tomcat   26607 26374  0 14:33 pts/3    00:00:00 ps -ef
    tomcat   26608 26374  0 14:33 pts/3    00:00:00 grep -i [t]omcat
    
  8. Examine Tomcat log files:
    tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ head --lines=13 catalina.out
    12-Jun-2018 14:31:25.718 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:         Apache Tomcat/9.0.8
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:           Apr 27 2018 19:32:00 UTC
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:          9.0.8.0
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:                Linux
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:             4.13.0-36-generic
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:           amd64
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:              /usr/lib/jvm/java-8-openjdk-amd64/jre
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:            1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:             Oracle Corporation
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:          /home/tomcat/apache-tomcat-9.0.8
    12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:          /home/tomcat/apache-tomcat-9.0.8
    12-Jun-2018 14:31:25.720 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/tomcat/apach...
    12-Jun-2018 14:31:25.720 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassL...
    
    tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ tail --lines=3 catalina.out
    12-Jun-2018 14:31:26.174 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
    12-Jun-2018 14:31:26.179 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
    12-Jun-2018 14:31:26.181 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 389 ms
    
  9. Go back up to the root:
    tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ cd ..
    tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ ../bin/catalina.sh stop
    
  10. You only need one of these set in ~/.bash_profile if you want to avoid using whatever installed version of Java you have on your host:
    • JAVA_HOME
    • JRE_HOME
    Map it to the root of your alternative JDK, etc. installation.

  11. Last, check out Apache Tomcat's default home page: http://localhost:8080


Apache Tomcat as a service on Linux

Let's look at a way to install Tomcat 9 as a service on CentOS. I nevertheless did all of this (minus the firewalld stuff) successfully on Linux Mint (Ubuntu). This is the result of some research: as of this writing, there was no packaging by which you can get Tomcat 9 from yum, apt, etc. So, this is how they do it. (I read probably six different posts about this.)

  1. For much of this, you'll need to have root access:
    $ sudo bash
    # whoami
    root
    
  2. Create a system user and group before installing Tomcat 9. This disables shell access, creates a new user tomcat and new group tomcat, then puts the user into the group. /opt/tomcat becomes the home directory:
    # groupadd tomcat
    # useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
    
  3. Download and install Tomcat 9. You can't use yum (or apt-get).
    # cd /opt
    # wget http://www.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
    --2018-06-12 15:18:34--  http://www.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
    Resolving www.apache.org (www.apache.org)... 40.79.78.1, 95.216.24.32, 2a01:4f9:2a:185f::2
    Connecting to www.apache.org (www.apache.org)|40.79.78.1|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 9818695 (9.4M) [application/x-gzip]
    Saving to: 'apache-tomcat-9.0.8.tar.gz'
    
    apache-tomcat-9.0.8.tar.gz   100%[========================================>]   9.36M  4.94MB/s    in 1.9s
    2018-06-12 15:18:36 (4.94 MB/s) - 'apache-tomcat-9.0.8.tar.gz' saved [9818695/9818695]
    # tar -zxf apache-tomcat-9.0.8.tar.gz       # explode tarball to subdirectory
    # rm apache-tomcat-9.0.8.tar.gz             # remove unneeded tarball
    # mv apache-tomcat-9.0.8 tomcat             # rename tomcat subdirectory
    # chown -R tomcat:tomcat tomcat/*           # fix up ownership
    
  4. Set up the systemd service script. This is something you must do and a configuration opportunity. The biggest example is memory—as configured in the CATALINA_OPTS line. See this discussion.
    # vim /etc/systemd/system/tomcat.service
    [Unit]
    Description=Apache Tomcat 9 Servlet Container
    After=syslog.target network.target
    
    [Service]
    Type=forking
    
    User=tomcat
    Group=tomcat
    
    Environment=CATALINA_PID=/opt/tomcat/tomcat.pid
    Environment=CATALINA_HOME=/opt/tomcat
    Environment=CATALINA_BASE=/opt/tomcat
    Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
    Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'
    
    ExecStart=/opt/tomcat/bin/startup.sh
    ExecStop=/opt/tomcat/bin/shutdown.sh
    Restart=on-failure
    
    User=tomcat
    Group=tomcat
    UMask=0007
    RestartSec=10
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    
  5. Run the following commands to cause Tomcat to start as a service upon boot:
    # systemctl daemon-reload                   # to reload the systemd dæmon
    # systemctl start tomcat                    # to start Tomcat
    # systemctl enable tomcat                   # to enable Tomcat
    # netstat -plntu                            # to check to see what port Tomcat's running on (default is 8080)
    # systemctl status tomcat                   # to check Tomcat's status
    
  6. Add the following lines to /opt/tomcat/conf/tomcat-users.xml just before the </tomcat-users> closing element:
    <role rolename="manager-gui" />
    <user username="admin"* password="password"* roles="manager-gui,admin-gui" />
    
    * I used tomcat and tomcat.

  7. Comment out the following line(s) from /opt/tomcat/webapps/manager/META-INF/context.xml (see Remote access below). This is only so that you can reach the Tomcat Web Application Manager remotely. It has nothing to do with whether you can reach the application you're hosting using Tomcat. (For that, see Hosted application access below.)
    18 <Context antiResourceLocking="false" privileged="true">
    19   <!-- <Valve className="org.apache.catalina.valves.RemoteAddrValve"
    20         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" /> -->
    21   <Manager sessionAttributeValueClassNameFilter="java\.lang\.(?:Boolean|Integer|Long|Num...
    22 </Context>
    
  8. /opt/tomcat/webapps/host-manager/META-INF/context.xml governs local and remote access to the Tomcat Virtual Host Manager application. See a note in Remote access below to understand more about this.

  9. There might be some value to adding the following line to /opt/tomcat/conf/context.xml. This might speed Tomcat up a bit by changing what it does when it loads applications dropped into the webapps subdirectory:
      <JarScanner scanClassPath="false" />
    </Context>
    

  10. Important note: I tried firewalld once; it totally destroyed my host. It's very intrusive and I couldn't unwind it from everything it invaded. I suggest something else (and since I'm not a Red Hat sort of guy, you're on your own there).

    Configure the firewall (example using firewalld and default port 8080):

    # yum install firewalld                                       # install firewalld
    # systemctl start firewalld                                   # start firewalld
    # systemctl enable firewalld                                  # enable firewalld
    # firewall-cmd --zone=public --permanent --add-port=8080/tcp  # configure firewalld
    # firewall-cmd --reload                                       # (ibid)
    # firewall-cmd --list-ports                                   # verify configuration of firewalld
    # firewall-cmd --list-services                                # ibid
    

  11. Check out web access from localhost and from a remote host:
    http://localhost:8080
    http://10.10.10.6:8080
    The latter did not work for me, but as it's not pressing (yes, it should be); I'm letting this slide for now as I'm busy. I'll come back later and say why and what I did to fix it. Maybe it's the stuff I commented out in context.xml.

  12. Troubleshooting: if the service doesn't start, try
    # systemctl status tomcat.service
    
    and if you see something like:
    Jun 12 16:09:46 nargothrond systemd[1]: Starting Apache Tomcat 9 Servlet Container...
    Jun 12 16:09:46 nargothrond startup.sh[32038]: touch: cannot touch '/opt/tomcat/logs/catalina.out': Permission denied
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Control process exited, code=exited status=1
    Jun 12 16:09:46 nargothrond systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Unit entered failed state.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Failed with result 'exit-code'.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Service hold-off time over, scheduling restart.
    Jun 12 16:09:46 nargothrond systemd[1]: Stopped Apache Tomcat 9 Servlet Container.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Start request repeated too quickly.
    Jun 12 16:09:46 nargothrond systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    
    Ensure that /opt/tomcat/logs isn't full of log files that are owned by root:root (because of early, imprecise attempts at running Tomcat). If, instead, you see this:
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Unit tomcat.service entered failed state.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service failed.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service holdoff time over, scheduling restart.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: start request repeated too quickly for tomcat.service
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Unit tomcat.service entered failed state.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service failed.
    
    I found that this cleared itself by doing systemctl restart tomcat.service. However, another time, it appeared because the download was itself broken which was discovered as soon as an attempt was made to look at logs/catalina.out, which didn't exist. The correctly exploded download must be contained on the path /opt/tomcat with various subdirectories like bin, conf, logs, etc.

HTTP 404 status on Apache Tomcat

Think your servlet is just perfect, it works in IntelliJ IDEA, all your ducks are in line, but you get HTTP status or error codes of 404, 406, 415 or 400?

In Tomcat, the application context root (that is, what comes immediately after the port number in the URL) is the name of the WAR file. This isn't true for all that's in ROOT, which is just /.

So, this is the answer. It was suggested by something my friend Scott said to me:

What is the URL used to try to access the endpoint? From the configuration you posted, it looks like it would be: http://localhost:8080/mdht-restlet/mdht-restlet

I'm going to attempt to quantify it. Why did this not mess up in IntelliJ IDEA's deployment, I don't know yet. Now some of this is arbitrary and there is flexibility, but following this will work.

  1. web.xml. Start here. Make servlet-name identical to the servlet's project name (in the highlighted lines below; this code is also not complete):
      <display-name>mdht-restlet</display-name>
    
      <servlet>
        <servlet-name>mdht-restlet</servlet-name>
    
        <init-param>
          <param-name>com.sun.jersey.config.property.packages</param-name>
          <param-value>com.acme.servlet</param-value>
        </init-param>
    
        <init-param>
          <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
          <param-value>true</param-value>
        </init-param>
      </servlet>
    
      <servlet-mapping>
        <servlet-name>mdht-restlet</servlet-name>
        <url-pattern>/*</url-pattern>
      </servlet-mapping>
    
  2. You can't name the Java source of the servlet this, but I keep it the same plus conventions: MdhtRestlet.java.

  3. Inside the servlet code, using the @Path annotation, indicate the path from the root to the actions named for each method. I don't have any deeper naming, so I just have no additional paths. But, when I had this problem (after 5+ years of not doing web applications), I was using @Path( "/mdht-restlet" )—hence Scott's comment to me (as reported above). Do this instead:
    @Path( "" )
    public class MdhtRestlet
    {
      @POST
      public Response postPatientData( ... ) { }
    
      @GET
      public String getStatusInPlainText() { }
    }
    
    This makes the URL to reach the servlet methods, whether POST or GET, always (and just) http://localhost:8080/mdht-restlet.

  4. Of course, this could be very much more complex—and would be in a complex application like Snapfish's accountmgr back when I created that:
    @Path( "/account" )
    public class AccountMgr
    {
      @POST
      @PATH( "/user" )
      public Response createAccount( ... ) { }
    
      @GET
      @PATH( "/user" )
      public String getAccount() { }
    
      @PUT
      @PATH( "/user/update" )
      public String modifyAccount() { }
    
      @DELETE
      @PATH( "/user/delete" )
      public String deleteAccount() { }
    }
    
    Thus, the URLs will be (in above order):
    1. POST http://localhost:8080/accountmgr/account/user
    2. GET http://localhost:8080/accountmgr/account/user
    3. PUT http://localhost:8080/accountmgr/account/user/update
    4. DELETE http://localhost:8080/accountmgr/account/user/delete

https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000021519-Tomcat-deployment-and-use-in-IDEA-works-but-404-when-deployed-to-server


Remote access

(${CATALINA_ROOT}/conf/context.xml is a file in your neighborhood (à la Mr. Rogers). It's not used for anything in this note.)

This setting, in ${CATALINA_ROOT}/webapps/manager/META-INF/context.xml,

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
   allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />

...will enforce access only for the clients connecting from localhost.

(Remember, we're talking about remote access here using the org.apache.catalina.valves.RemoveAddrValve class. Besides these "valve" details to consider, you must also punch a hole in your firewall for whatever port you're running Tomcat on as defined in conf/server.xml.)

This comes by default in the installation. So, to permit access from the outside, comment the <Valve... /> element out.

To permit unrestricted access for clients from localhost, but require authentication for all other (remote) clients, which must come in over port 8443, add this:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       addConnectorPort="true"
       allow="127\.\d+\.\d+\.\d+;\d*|::1;\d*|0:0:0:0:0:0:0:1;\d*|.*;8443" />
<Valve className="org.apache.catalina.authenticator.BasicAuthenticator" />

The basic authenticator will work off the details you (may have) added to ${CATALINA_ROOT}/conf/tomcat-users.xml. Look elsewhere in these notes for this file.

To enable unrestricted access via port 8080:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       addConnectorPort="true"
       allow=".*;8080" />

The configuration above really belongs to the Tomcat Web Application Manager:

Notes on accessing Tomcat "host" Manager

By default, the Host Manager, configured in ${CATALINA_ROOT}/webapps/host-manager/META-INF/context.xml, is only accessible from a browser running on the same host as Tomcat. If you can't live with this restriction, you'll need to edit this file to allow all IP addresses (or whatever ones you want). This file does not govern access to applications Tomcat is hosting, but only its manager interface.

To allow this access, change to:

<Context antiResourceLocking="false" privileged="true">
  <Valve className="org.apache.catalina.valves.RemoteAddrValve" allow=".*" />
</Context>

This configuration belongs to the Tomcat Virtual Host Manager:


Remove Tomcat Manager access

Not sure what I was thinking in the note above, but to accesss anything from the Tomcat Manager (http://host:8080), of all the possible "solutions" I found, only doing this to

${CATALINA_HOME}/webapps/manager/META-INF/context.xml

...allowed me to reach through any of the buttons in the manager from the browser on my development host on the remote server running my target application on Tomcat. Basically, I added the comment characters in the paragraph below to what was already there:

<Context antiResourceLocking="false" privileged="true">
  <CookieProcessor className="org.apache.tomcat.util.http.Rfc6265CookieProcessor"
                   sameSiteCookies="strict" />
  <!--
  <Valve className="org.apache.catalina.valves.RemoteAddrValve"
         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />
    -->
  <Manager sessionAttributeValueClassNameFilter="java\.lang\.(?:Boolean|Integer|Long|Number|String)|org\.apache\... (truncated)
</Context>

Caveats

Obviously, this opens that host's Tomcat Manager to any Tom, Dick or Harry that can reach it to gain access. In my case, I don't care because I'm behind a firewall and I put nothing sensitive on this (testing) server. I googled for and saw several "safer" and "more correct" ways of doing this, including adding a file, manager.xml to /opt/tomcat/conf/Catalina/localhost with a regular expression in the RemoteAddrValve's allow attribute, but I could not get that to work. Nor could I get such a modification to work in the solution above in being more clever than just commenting it out.


Hosted application access

You've developed a web application that you deploy to Tomcat and it works as you access it locally, but when you try to access it remotely you cannot.

First, this is likely because your firewall, for the port on which Tomcat is configured in conf/server.xml is not open (to the world). For example, if I have Tomcat running on port 8080:

# netstat -pln | grep 8080                            # see that it's not open
# iptables -A INPUT -p tcp --dport 8080 --jump ACCEPT # make a hole
# netstat -pln | grep 8080                            # see if it's open

This did it for me, however...

Second, by default, Tomcat disables access from outside IP addresses. You may (or not—I didn't have to) add the following new line to conf/server.xml:

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           URIEncoding="UTF-8"
           redirectPort="8443"
           useIPVHosts="true" />

...and, last, sometimes even:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
       prefix="localhost_access_log" suffix=".txt"
       pattern="%h %l %u %t "%r"e; %s %b"
       resolveHosts="true" />

Tomcat vs. Jetty

Both...

In embedded constructions, for example, Apache NiFi uses an embedded Jetty to support its UI and its ReST API, Jetty may have an edge.

Eclipse Jetty Apache Tomcat
Birth 1995 1999
Owned Mort Bay, sourceforge.net, Eclipse Sun Microsystems, Apache Software Foundation
Consumers Yahoo, Google Spring, Jenkins, JBoss (!)
Market share 8-12% > 50%, maybe 60%
Perception Performance/user focused Specification focused
Servlet 4.0 Support in 2020 Supported in 2017

Tomcat versus application memory

Any (and all) application(s) running under Tomcat use Tomcat's Java virtual machine (JVM) including memory.

The stack- and heap memory established for Tomcat are defined in environment variable, CATALINA_OPTS, as:

The above are Tomcat 9's defaults, but they can be set to anything, in particular, the heap-size option (-Xmx).

When Tomcat is run as a service, CATALINA_OPTS is defined (and, therefore, modifiable) in /etc/systemd/system/tomcat.service.


Tomcat memory heap when run as a service

First, how much memory does Tomcat have?

root@tirion:/home/russ# ps -ef | grep [t]omcat
tomcat    127078       1 71 15:12 ?        00:00:09 \
    /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java \
    -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties \
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
    -Djava.awt.headless=true \
    -Djava.security.egd=file:/dev/./urandom \
    -Djdk.tls.ephemeralDHKeySize=2048 \
    -Djava.protocol.handler.pkgs=org.apache.catalina.webresources \
    -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 \
    -Xms512M -Xmx1024M -server -XX:+UseParallelGC \
    -Dignore.endorsed.dirs= \
    -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar \
    -Dcatalina.base=/opt/tomcat \
    -Dcatalina.home=/opt/tomcat \
    -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start

How to change this amount?

root@tirion:/etc/systemd/system# vim tomcat.service
    [Unit]
    Description=Apache Tomcat Web Application Container
    After=network.target

    [Service]
    Type=forking

    Environment=JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64
    Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
    Environment=CATALINA_HOME=/opt/tomcat
    Environment=CATALINA_BASE=/opt/tomcat
    Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
    Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

    ExecStart=/opt/tomcat/bin/startup.sh
    ExecStop=/opt/tomcat/bin/shutdown.sh

    User=tomcat
    Group=tomcat
    UMask=0007
    RestartSec=10
    Restart=always

    [Install]
    WantedBy=multi-user.target

An HTTP client to talk to Tomcat (or any such thing)...

So, how would one write a simple HTTP client to talk to an application (servlet) running in a Tomcat container? Assuming there's something easy to GET, do this first:

package com.windofkeltia.client;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

import static java.util.Objects.nonNull;

/**
 * How to get back the "I'm up and good" from out Tomcat-based application.
 * @author Russell Bateman
 * @since July 2021
 */
public class GetRequest
{
  private static final int    PORT     = 8080;
  private static final String BASE_URL = "http://localhost:" + PORT;
  private static final String APP_NAME = "application-3.2.0";

  public static void main( String ... args ) throws IOException
  {
    // set up HTTP connection and request...
    URL               url        = new URL( BASE_URL + '/' + APP_NAME );
    HttpURLConnection connection = ( HttpURLConnection ) url.openConnection();
    connection.setRequestMethod( "GET" );
    connection.setRequestProperty( "Accept", "text/plain" );

    // fire off request, then read response...
    int            status = connection.getResponseCode();
    BufferedReader reader = new BufferedReader( new InputStreamReader( connection.getInputStream() ) );
    String         line;
    StringBuilder  content = new StringBuilder();

    while( nonNull( line = reader.readLine() ) )
      content.append( line ).append( '\n' );

    reader.close();

    System.out.println( content );
  }
}

...and assuming there's some reason to POST to it, do this too:

package com.windofkeltia.client;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;

import static java.util.Objects.nonNull;

/**
 * How to post an XML document to our application, get back a response and read it.
 * @author Russell Bateman
 * @since July 2021
 */
public class PostRequest
{
  private static final int    PORT      = 8080;
  private static final String BASE_URL  = "http://localhost:" + PORT;
  private static final String APP_NAME  = "application-3.2.0";
  private static final String TESTING   = "src/test/resources/fodder/";
  private static final String FILENAME  = "test-file.xml";
  private static final String IXML_PATH = TESTING + "client/" + FILENAME;

  public static void main( String ... args ) throws IOException
  {
    // set up HTTP connection and request...
    URL               url        = new URL( BASE_URL + '/' + APP_NAME );
    HttpURLConnection connection = ( HttpURLConnection ) url.openConnection();
    connection.setRequestMethod( "POST" );
    connection.setRequestProperty( "Content-Type", "application/xml" );
    connection.setRequestProperty( "Accept",       "application/xml" );
    connection.setDoOutput( true );   // (otherwise, can't write payload to output stream)

    // build and POST payload...
    try( BufferedReader reader       = new BufferedReader( new FileReader( IXML_PATH ) );
        OutputStream   outputStream = connection.getOutputStream() )
    {
      for( String line = reader.readLine(); nonNull( line ); line = reader.readLine() )
      {
        final String PAYLOAD = line + '\n';
        outputStream.write( PAYLOAD.getBytes() );
      }
    }
    catch( Exception e )
    {
      e.printStackTrace();
    }

    // fire off request, then read response...
    int            status = connection.getResponseCode();
    BufferedReader reader = new BufferedReader( new InputStreamReader( connection.getInputStream() ) );
    String         line;
    StringBuilder  content = new StringBuilder();

    while( nonNull( line = reader.readLine() ) )
      content.append( line ).append( '\n' );

    reader.close();

    System.out.println( content );
  }
}

Besides GET and POST above, other HTTP interactions (HEAD, PUT, PATCH, DELETE, etc.) are similarly coded.


Tomcat woes...

I found these four problems inspecting catalina.out, Tomcat's log:

  1. I see this everywhere inside catalina.out. What is a "Jasper scratchDir"?
    org.apache.jasper.EmbeddedServletOptions The scratchDir you specified is unusable
    
    Missing information: I don't know what subdirectory may not permit adequate privileges (likely to Tomcat, tomcat:tomcat). Solution: check ownership and privileges under /opt/tomcat. Assessment: this is probably not a very important problem and likely has nothing to do with what I'm chasing down. The source of the problem may be that Tomcat is being started and stopped by the wrong user. What is the right user? tomcat, but isn't it managed thus?
    # systemctl start|stop|restart|status tomcat
    
  2. Looking harder at catlina.out, I think I see this just before the cron job bounces Tomcat:
    com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within \
        MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
    
    This may have to do with mdht-restlet missing a parameter name
    <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
    
    which I did add to fhir-server. It would be good to try that and test whether it solves the problem on Tomcat. It's not likely anything to do with what we're looking for today.


    No. JSON is written to catalina.out at some point, but not by any code associated with the Jersey framework. This parameter is not needed.

  3. The servlet makes calls into the MDHT library to construct the CCDA output (CCD), this is seen frequently in catalina.out:
    org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application created a \
        ThreadLocal with key of type [java.lang.ThreadLocal] and a value of type [org.eclipse.ocl.ecore.QueryImpl] \
        but failed to remove it when the web application was stopped
    
    Research demonstrates that this is something to expect when Tomcat is shut down. As it seems to be associated in time with the cron job bouncing, this isn't likely a real problem, but only resources that the thread acquires as it makes its way through the MDHT library and returns thence without freeing them. The notification comes, as pointed out, when Tomcat is being shut down.

Researching those messages above we do not understand...


Tomcat memory settings...

Recommended best practice for memory sizes are below. They are set up when Tomcat is installed as a service.

Just so you know, if you're running multiple applications (servlets) under Tomcat, ...

You can only control Tomcat's JVM settings for all applications run. The JVM settings are for Tomcat which divides its heap among everything you deploy to /opt/tomcat/webapps.

Here's how to adjust Tomcat's JVM memory. Assuming Tomcat is running as a service, you should find the following path/filename:

/etc/systemd/system/tomcat.service

Heap size

In that file (tomcat.service), which you can edit (then bounce Tomcat), you should find a line like this:

Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'

The above is the default value (for Tomcat 9). -Xms refers to the stack size (see discussion below); -Xmx to the heap size. The latter (heap size) is probably what you want to increase. The value shown above there is 1 gigabyte. I would suggest -Xmx4096M as best practice for a starting value.

While it is possible to increase the JVM heap size greater even than -Xmx4096M, it is recommended to avoid increasing it beyond one-half of the total, physical RAM available on the executing host (or VM).

To bounce Tomcat after changes:

# systemctl restart tomcat

Stack size

Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'

-Xms1024M seems a good maximum to respect, but -Xms512M is the default (Tomcat 9) and best starting value.

Stack size will inverse-proportionally limit the number of threads available for processing in the reverse sense.

Too big a stack size and each thread spun up to tackle a request by the application(s) Tomcat is serving depleats overall memory faster as each thread is allocated more stack size. Therefore: be certain you really need as much stack as you're giving. You'll know a thread has run short of stack memory when a java.lang.StackOverflowError appears in catalina.out.

To bounce Tomcat after changes:

# systemctl restart tomcat

Not running Tomcat as a service?

The above is done inside subdirectory /opt/tomcat/bin (or wherever your Tomcat files are in the filesystem).

Create a new file, setenv.sh:

export CATALINA_OPTS="$CATALINA_OPTS -Xms512M"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx1024M"

Then, bounce Tomcat, which you are running "locally" with:

$ ./catalina.sh run

You will see it launch and, confirmed somewhere in the logging that comes out to stdout, the following:

.
.
.
INFO [main] VersionLoggerListener.log Command line argument: -Xms512m
INFO [main] VersionLoggerListener.log Command line argument: -Xmx2048m
.
.
.

Safe to remove catalina.out?

If catalina.out is removed after Tomcat is stopped, a new catalina.out will simply be created once Tomcat is restarted.


Rotating catalina.out...

...including removing rotated log files.

Note that there is a default configuration Tomcat employs in the case of no system logrotate configuration for Tomcat, but, if you don't like the default behavior, do as below:

  1. (I would never touch /opt/tomcat/conf/logging.properties.)
  2. Instead, go to /etc/logrotate.d.
  3. Note that, at /etc/cron.daily, there is a logrotate file. This cron job triggers /etc/logrotate.conf which includes all the system-wide log-rotation configuration files in /etc/logrotate.d. This can be made to work for Tomcat also.
  4. Therefore, add a new file (if missing), tomcat.
  5. In this file, put the path to the Tomcat log, e.g.: /opt/tomcat/logs/catalina.out.
  6. Then add lines like the following thereafter:
    /opt/tomcat/logs/catalina.out
    {
      create                         # creates new, empty log file
      notifempty                     # creates new log file only if current one not empty
      daily|weekly|monthly           # choose one or...
      size 1m                        # rotate when log reaches 1Mb
      compress                       # compress rotated log using gzip
      missingok                      # don't error out if log missing
      rotate 7                       # keep at most 7 rotated logs, or
      maxage <days>                  # remove rotated logs after days
    }