Wednesday, January 25, 2017

Spring boot embeded tomcat stuck during start

Problem

During our last initial deployment to test environment, we faced a following problem: the embedded tomcat from spring boot got stuck during the start. There was no exception. It just kept stopping on some level. Our service was contained inside docker. The good thing is, it was working fine in development environment and work stations of developers. But not in the test environment (Centos). We've asked ourselves a question: does the docker container depend on the host operating system version? The answer is: YES.

Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in user space on the host operating system.


Implementation details

But we've run one more test. We've run the same microservice without the docker like this:

 java -jar service.jar  

and the problem still existed.

Our next test was to run some simple spring boot project. We've chosen one of my example projects from github: https://github.com/bartoszkomin/hibernate-many-to-many-demo
This project uses it as a parent:

 <parent>  
   <groupId>org.springframework.boot</groupId>  
   <artifactId>spring-boot-starter-parent</artifactId>  
   <version>1.4.3.RELEASE</version>  
 </parent>  

And it was fine! To compare, our "not working" project:

 <parent>  
   <groupId>org.springframework.cloud</groupId>  
   <artifactId>spring-cloud-starter-parent</artifactId>  
</parent>  

The Dockerfile during our development process looked like this:

 FROM java:8  
 VOLUME /tmp  
 ADD api-0.1.0.jar app.jar  
 RUN bash -c 'touch /app.jar'  
 ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]  

But the docker image after build was quite heavy, about 700 MB. So just before release we decided to use openjdk:8-jre-alpine, image size with that is about 160 MB.

 FROM openjdk:8-jre-alpine  
 ADD target/api-1.0.0.jar app.jar  
 ENTRYPOINT ["java", "-jar", "/app.jar"]  


Solution

The problem occurred because we removed this line from our Dockerfile:

ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]  

and the important parameter was the one that configures JRE to use a non-blocking entropy source:

-Djava.security.egd=file:/dev/./urandom  

Tomcat 7+ heavily relies on SecureRandom class to provide random values for its session ids. Depending on your JRE, it can cause delays during start up if the entropy source that is used to initialize SecureRandom is short of entropy.
And finally our working Dockerfile looks like this:

 FROM frolvlad/alpine-oraclejdk8:slim  
 VOLUME /tmp  
 ADD target/api-1.0.0.jar app.jar  
 RUN sh -c 'touch /app.jar'  
 ENV JAVA_OPTS=""  
 ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]  

Why a "simple" project with parent from org.springframework.boot worked fine, and the one with org.springframework.cloud did not? It's probably because there was a different version of Tomcat inside the jar file.


1 comment:

  1. Yeah, it's a very common Tomcat problem that manifests itself not only in contenerized applications but with also with regular Tomcat deployments but I'm writing to point out that the Docker image ENTRYPOINT described here may not work as intended. I'm talking about $JAVA_OPTS environment variable. At first glance everything looks OK but in fact the variable will not be populated from the environment (will be empty). To fix this an ENTRYPOINT adjustment is required:

    ENTRYPOINT [ "exec", "/usr/bin/java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

    You can find the detailed explanation here: https://c4.vc/2015/12/23/Docker-Spring-Boot-and-JAVA-OPTS/

    ReplyDelete