Spring boot embeded tomcat stuck during start

Problem

During our last initial deployment to test environment, we faced a following problem: the embedded tomcat from spring boot got stuck during the start. There was no exception. It just kept stopping on some level. Our service was contained inside docker. The good thing is, it was working fine in development environment and work stations of developers. But not in the test environment (Centos). We've asked ourselves a question: does the docker container depend on the host operating system version? The answer is: YES.

Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in user space on the host operating system.

Implementation details

But we've run one more test. We've run the same microservice without the docker like this:

 java -jar service.jar  

and the problem still existed.

Our next test was to run some simple spring boot project. We've chosen one of my example projects from github: https://github.com/bartoszkomin/hibernate-many-to-many-demo
This project uses it as a parent:

 <parent>  
   <groupId>org.springframework.boot</groupId>  
   <artifactId>spring-boot-starter-parent</artifactId>  
   <version>1.4.3.RELEASE</version>  
 </parent>  

And it was fine! To compare, our "not working" project:

 <parent>  
   <groupId>org.springframework.cloud</groupId>  
   <artifactId>spring-cloud-starter-parent</artifactId>  
</parent>  

The Dockerfile during our development process looked like this:

 FROM java:8  
 VOLUME /tmp  
 ADD api-0.1.0.jar app.jar  
 RUN bash -c 'touch /app.jar'  
 ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]  

But the docker image after build was quite heavy, about 700 MB. So just before release we decided to use openjdk:8-jre-alpine, image size with that is about 160 MB.

 FROM openjdk:8-jre-alpine  
 ADD target/api-1.0.0.jar app.jar  
 ENTRYPOINT ["java", "-jar", "/app.jar"]  

Solution

The problem occurred because we removed this line from our Dockerfile:

ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]  

and the important parameter was the one that configures JRE to use a non-blocking entropy source:

-Djava.security.egd=file:/dev/./urandom  

Tomcat 7+ heavily relies on SecureRandom class to provide random values for its session ids. Depending on your JRE, it can cause delays during start up if the entropy source that is used to initialize SecureRandom is short of entropy.

And finally our working Dockerfile looks like this:

 FROM frolvlad/alpine-oraclejdk8:slim  
 VOLUME /tmp  
 ADD target/api-1.0.0.jar app.jar  
 RUN sh -c 'touch /app.jar'  
 ENV JAVA_OPTS=""  
 ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]  

Why a "simple" project with parent from org.springframework.boot worked fine, and the one with org.springframework.cloud did not? It's probably because there was a different version of Tomcat inside the jar file.

Spring Security takes long time to login

Problem

In our project we've implemented a login process based on spring security. After that we faced one problem -  why does it take so long? Our login request took about 6-7 seconds. There was a quite long CPU peak during this operation. We asked ourselves, what might be the reason. There were a few points where the problem might have existed - the application was contained in docker, it contained hystrix, api gateway, service discovery, 2 databases - postgres and redis. We tried to run this application without all that extra stuff - no progress, still 7 seconds. After that we knew, the problem was inside the spring library, not within our code.




Implementation

You can find a simple example project on GitHub here:

https://github.com/bartoszkomin/spring-login-long-demo




Please take a look at class SecurityConfig extended from WebSecurityConfigurerAdapter:

 package com.blogspot.bartoszkomin.spring_login.config;  
 import org.springframework.beans.factory.annotation.Autowired;  
 import org.springframework.boot.autoconfigure.security.SecurityProperties;  
 import org.springframework.context.annotation.Bean;  
 import org.springframework.context.annotation.Configuration;  
 import org.springframework.core.annotation.Order;  
 import org.springframework.http.HttpMethod;  
 import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;  
 import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;  
 import org.springframework.security.config.annotation.web.builders.HttpSecurity;  
 import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;  
 import org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder;  
 import org.springframework.security.crypto.password.PasswordEncoder;  
 import com.blogspot.bartoszkomin.spring_login.service.CustomUserDetailsService;  
 /**  
  * @author Bartosz Komin  
  *  
  */  
 @Configuration  
 @EnableGlobalMethodSecurity(prePostEnabled = true)  
 public class SecurityConfig extends WebSecurityConfigurerAdapter {  
     /**  
      * Init custom user detail service  
      */  
     @Autowired  
     private CustomUserDetailsService userDetailsService;  
     /* (non-Javadoc)  
      * @see org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter#configure(org.springframework.security.config.annotation.web.builders.HttpSecurity)  
      */  
     @Override  
     protected void configure(HttpSecurity http) throws Exception {  
         http.csrf().disable();  
         http.httpBasic().and()  
             .authorizeRequests()  
                 .antMatchers(HttpMethod.OPTIONS).permitAll()  
                 .antMatchers(HttpMethod.POST,"/login").permitAll()  
                 .antMatchers(HttpMethod.GET, "/insert").permitAll()  
                 .anyRequest().authenticated();  
     }  
     /* (non-Javadoc)  
      * @see org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter#configure(org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder)  
      */  
     @Override  
     protected void configure(AuthenticationManagerBuilder auth) throws Exception {  
         auth.userDetailsService(userDetailsService).passwordEncoder(getPasswordEncoder());  
     }  
     @Bean  
     public PasswordEncoder getPasswordEncoder() {  
         BCryptPasswordEncoder bCryptPAsswordEncoder = new BCryptPasswordEncoder(16);      
         return bCryptPAsswordEncoder;  
     }  
 }  

We override the configure method to use our own user entity object to get the password from our database and define the password encoder. In our case the password is encoded with BCrypt. BCryptPasswordEncoder obtains the strength of the password as a parameter. In our case it is 16.

CustomUserDetailsService looks like this:

 package com.blogspot.bartoszkomin.spring_login.service;  
 import java.util.ArrayList;  
 import java.util.List;  
 import org.springframework.beans.factory.annotation.Autowired;  
 import org.springframework.security.core.GrantedAuthority;  
 import org.springframework.security.core.authority.SimpleGrantedAuthority;  
 import org.springframework.security.core.userdetails.User;  
 import org.springframework.security.core.userdetails.UserDetails;  
 import org.springframework.security.core.userdetails.UserDetailsService;  
 import org.springframework.security.core.userdetails.UsernameNotFoundException;  
 import org.springframework.stereotype.Service;  
 import com.blogspot.bartoszkomin.spring_login.repository.UserRepository;  
 @Service  
 public class CustomUserDetailsService implements UserDetailsService {  
     @Autowired  
     private UserRepository repository;  
     public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {  
         User user = null;  
         if (username == null || username.isEmpty()) {  
             throw new UsernameNotFoundException("User not found");  
         }  
         com.blogspot.bartoszkomin.spring_login.model.User u = repository.findByName(username);  
         if (u == null) {  
             throw new UsernameNotFoundException("User " + username + " not found");  
         }  
         boolean enabled = true;  
         boolean accountNonExpired = true;  
         boolean credentialsNonExpired = true;  
         boolean accountNonLocked = true;  
         user = new User(u.getName(), u.getPassword(), enabled, accountNonExpired, credentialsNonExpired, accountNonLocked, getAuthorities(null));  
         return user;  
     }  
     private List<SimpleGrantedAuthority> getAuthorities(String role) {  
     List<SimpleGrantedAuthority> authList = new ArrayList<SimpleGrantedAuthority>();  
     authList.add(new SimpleGrantedAuthority("ROLE_USER"));  
     return authList;  
   }  
 }  

And User entity:

 package com.blogspot.bartoszkomin.spring_login.model;  
   
 import javax.persistence.Entity;  
 import javax.persistence.GeneratedValue;  
 import javax.persistence.GenerationType;  
 import javax.persistence.Id;  
 import javax.persistence.Table;  
   
 /**  
  * @author Bartosz Komin  
  * User  
  */  
 @Entity  
 @Table(name = "users")  
 public class User {  
       
     @Id  
     @GeneratedValue(strategy = GenerationType.IDENTITY)  
     private Integer id;  
   
     private String name;  
   
     private String password;  
   
   
     public String getName() {  
         return name;  
     }  
   
     public void setName(String name) {  
         this.name = name;  
     }  
   
     public String getPassword() {  
         return password;  
     }  
   
     public void setPassword(String password) {  
         this.password = password;  
     }  
   
     public Integer getId() {  
         return id;  
     }  
   
 }  

In our example project we have 3 REST endpoints:

• POST /login - to display token.
• GET /insert - you don't need to be authenticated to use this endpoint, it inserts user login and password to database, the login is "user" and the password is "password". Data is needed to login to the system.
• GET /secret - if you are not authenticated, you will get a 401 Unauthorized HTTP response. To login you need to use Basic authentication with base64 login/pass encoded. In our case it will be: Authorization: Basic dXNlcjpwYXNzd29yZA==

Example response to GET /secret without authentication:

After executing GET/insert (where we put user and password record to database), GET /secret with basic auth response looks like this:

Let's execute the same endpoint, but with measuring the exact execution time:

On i7 processor the login execution time was 4.6 seconds. Quite long, isnt't it?


Solution

To check what was wrong, we used VisualVM.
Let's execute the project jar like this:

 java -jar spring-login-1.0.0.jar  

and then run VisualVM. Insert login/pass with GET/insert, and then try to login. This is what we see on VisualVM for this operation:

Got you! 96% of CPU time was used by org.springframework.security.crypto.bcrypt.BCrypt.key() and it was 4699 ms. And this is our answer - everything works fine, but our password encryption level is so high, it has to take time.

In this example, strength of the password for BCrypt was set to 16.
Let's check the time one more time, for password strength = 12.
The results are:

This time the encrypt operation was ready in 299 ms.

Many to many relation with hibernate and json never ending recursion problem

Problem

Most examples of many-to-many relation for Spring found on the internet had a list of objects on both sides of the relation, but one of those sides was the owner of the relation. This gives us for example a list of books for a user, but we can not display a list of users for a certain book.

We were required to get a list of opposite objects on both sides (for example to get alist of books for a user, but also a list of users for one book).

On Hibernate level it wasn't a problem. Problem appeared when Jackson parser tried to change that kind of object to Json. For a linked object we obtained a neverending recursion ("user" contains a list of books, which contains a list of users, which contains a list of books etc...).

Implementation

You can find a working example-project on GitHub here:

https://github.com/bartoszkomin/hibernate-many-to-many-demo/



An example structure of the database looks as follows:

At first we've defined the entity objects in a following manner:

1:  package com.blogspot.bartoszkomin.hibernate_many_to_many_demo.model;  
2:    
3:  import java.util.List;  
4:  import javax.persistence.CascadeType;  
5:  import javax.persistence.Column;  
6:  import javax.persistence.Entity;  
7:  import javax.persistence.GeneratedValue;  
8:  import javax.persistence.GenerationType;  
9:  import javax.persistence.Id;  
10:  import javax.persistence.JoinColumn;  
11:  import javax.persistence.JoinTable;  
12:  import javax.persistence.ManyToMany;  
13:  import javax.persistence.Table;  
14:  import javax.persistence.UniqueConstraint;  
15:  import javax.validation.constraints.NotNull;  
16:    
17:  import com.fasterxml.jackson.annotation.JsonIgnoreProperties;  
18:  import com.fasterxml.jackson.annotation.JsonManagedReference;  
19:    
20:  /**  
21:   * @author Bartosz Komin  
22:   * User entity  
23:   */  
24:  @Entity  
25:  @Table(name = "users")  
26:  public class User {  
27:    
28:        
29:      @Id  
30:      @GeneratedValue(strategy = GenerationType.IDENTITY)  
31:      private Integer id;  
32:        
33:      @NotNull  
34:      @Column(nullable = false)  
35:      private String name;  
36:    
37:      @ManyToMany(cascade = {CascadeType.MERGE})  
38:      @JoinTable(name = "UserBook",   
39:          joinColumns = @JoinColumn(name = "user_id", referencedColumnName = "id"),   
40:          inverseJoinColumns = @JoinColumn(name = "book_id", referencedColumnName = "id"),   
41:          uniqueConstraints={@UniqueConstraint(columnNames={"user_id", "book_id"})})  
42:      private List<Book> books;  
43:        
44:        
45:    
46:      public String getName() {  
47:          return name;  
48:      }  
49:    
50:      public void setName(String name) {  
51:          this.name = name;  
52:      }  
53:    
54:      public List<Book> getBooks() {  
55:          return books;  
56:      }  
57:    
58:      public void setBooks(List<Book> books) {  
59:          this.books = books;  
60:      }  
61:    
62:      public Integer getId() {  
63:          return id;  
64:      }  
65:        
66:  }  

1:  package com.blogspot.bartoszkomin.hibernate_many_to_many_demo.model;  
2:    
3:  import java.util.List;  
4:  import javax.persistence.CascadeType;  
5:  import javax.persistence.Column;  
6:  import javax.persistence.Entity;  
7:  import javax.persistence.GeneratedValue;  
8:  import javax.persistence.GenerationType;  
9:  import javax.persistence.Id;  
10:  import javax.persistence.JoinColumn;  
11:  import javax.persistence.JoinTable;  
12:  import javax.persistence.ManyToMany;  
13:  import javax.persistence.Table;  
14:  import javax.persistence.UniqueConstraint;  
15:  import javax.validation.constraints.NotNull;  
16:    
17:  import com.fasterxml.jackson.annotation.JsonBackReference;  
18:  import com.fasterxml.jackson.annotation.JsonIgnore;  
19:  import com.fasterxml.jackson.annotation.JsonIgnoreProperties;  
20:    
21:  /**  
22:   * @author Bartosz Komin  
23:   * Book entity  
24:   */  
25:  @Entity  
26:  @Table(name = "books")  
27:  public class Book {  
28:    
29:        
30:      @Id  
31:      @GeneratedValue(strategy = GenerationType.IDENTITY)  
32:      private Integer id;  
33:        
34:      @NotNull  
35:      @Column(nullable = false)  
36:      private String name;  
37:    
38:      @ManyToMany(cascade = {CascadeType.MERGE})  
39:      @JoinTable(name = "UserBook",   
40:          joinColumns = @JoinColumn(name = "book_id", referencedColumnName = "id"),   
41:          inverseJoinColumns = @JoinColumn(name = "user_id", referencedColumnName = "id"),   
42:          uniqueConstraints={@UniqueConstraint(columnNames={"book_id", "user_id"})})  
43:      private List<User> users;  
44:        
45:      public String getName() {  
46:          return name;  
47:      }  
48:    
49:      public void setName(String name) {  
50:          this.name = name;  
51:      }  
52:    
53:      public List<User> getUsers() {  
54:          return users;  
55:      }  
56:    
57:      public void setUsers(List<User> users) {  
58:          this.users = users;  
59:      }  
60:    
61:      public Integer getId() {  
62:          return id;  
63:      }  
64:        
65:  }  

In the presented GitHub project we can find a few REST endpoints:

POST /users
GET /users
GET /users/user_id
POST /books
GET /books
GET /books/book_id

With presented entities it works properly on Database/Hibernate level. But if we want to obtain some linked objects, we get a neverending recursion, which occurs on Jackson level, and it looks like this:

{"id":1,"name":"Albert Einstein","books":[{"id":1,"name":"The Meaning of Relativity", "users":[{"id":1,"name":"Albert Einstein","books":[{"id":1,"name":"The Meaning of Relativity", "users":{...}}]}]}]}

Solution
I've found some tips on the internet with an @JsonIgnore or @JsonManagedReference + @JsonBackReference annotation, but it was not working in our case (we were able to see the list only on one side).

Our solution was to use @JsonIgnoreProperties annotation as follows:

User class

1:  @ManyToMany(cascade = {CascadeType.MERGE})  
2:  @JoinTable(name = "UserBook",   
3:        joinColumns = @JoinColumn(name = "user_id", referencedColumnName = "id"),   
4:        inverseJoinColumns = @JoinColumn(name = "book_id", referencedColumnName = "id"),   
5:        uniqueConstraints={@UniqueConstraint(columnNames={"user_id", "book_id"})})  
6:  @JsonIgnoreProperties("users")  
7:  private List<Book> books;  

Book class

1:  @ManyToMany(cascade = {CascadeType.MERGE})  
2:  @JoinTable(name = "UserBook",   
3:        joinColumns = @JoinColumn(name = "book_id", referencedColumnName = "id"),   
4:        inverseJoinColumns = @JoinColumn(name = "user_id", referencedColumnName = "id"),   
5:        uniqueConstraints={@UniqueConstraint(columnNames={"book_id", "user_id"})})  
6:  @JsonIgnoreProperties("books")  
7:  private List<User> users;  

Working example result:

GET /users/1

GET /books/1

Summary

In the GitHub demo project you can find a working example in master branch.

There is also one more branch: many-to-many-recursion-problem, where you can check how application behaves, when neverending recursion appears. My IDE got laggy, and that might cause a problem with getting a response. I removed tests from this branch to not overload the computer when trying to build a project.

On GitHub "readme" there is also some information on how to run the project and how to send a request to api with curl.