Add support for Hazelcast 4

[justinnichols]

With Hazelcast 4.0, Spring Boot 2.2.6 is not properly adhering to the spring.hazelcast.config application property which allows pointing to a specific Hazelcast configuration and then instantiating the proper bean.

Spring Boot is attempting to create a HazelcastInstance of HazelcastClientConfiguration (which expects <hazelcast-client> XML root tag) instead of HazelcastServerConfiguration (which expects <hazelcast> XML root tag).

The error that occurs:

Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.

The following Github repository depicts two example Spring Boot 2.2.6 projects, one with this error, and one with a workaround:

https://github.com/justinnichols/spring-boot-2.2.6-hazelcast-4.0

Dependencies:

<dependency>
	<groupId>com.hazelcast</groupId>
	<artifactId>hazelcast</artifactId>
	<version>4.0</version>
</dependency>

Configuration in application.properties:

spring.hazelcast.config=classpath:config/hazelcast.xml

The stacktrace that occurs is:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [org/springframework/boot/autoconfigure/hazelcast/HazelcastClientConfiguration$HazelcastClientConfigFileConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1290) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1210) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	... 18 common frames omitted

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	... 32 common frames omitted

Caused by: com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at com.hazelcast.client.config.XmlClientConfigBuilder.checkRootElement(XmlClientConfigBuilder.java:183) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.parseAndBuildConfig(XmlClientConfigBuilder.java:168) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:157) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:150) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:145) ~[hazelcast-4.0.jar:4.0]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientFactory.getClientConfig(HazelcastClientFactory.java:66) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientFactory.<init>(HazelcastClientFactory.java:48) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientConfiguration$HazelcastClientConfigFileConfiguration.hazelcastInstance(HazelcastClientConfiguration.java:55) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_192]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_192]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_192]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_192]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]

Workarounds (kindly provided by @mesutcelik via Gitter)

• Place the hazelcast.xml in the root of the classpath. (This is depicted in the repo linked above)
• Specify a hazelcast.config system property.
• Define a configuration bean programmatically.

[wilkinsona]

Thanks for the report. We currently build and test against Hazelcast 3.12.6. I'm not surprised to learn that the auto-configuration does not work out of the box with Hazelcast 4.0 as it's a new major version.

[vpavic]

Note that besides a handful of binary changes made without deprecation period, Hazelcast 4.0 also made changes to packaging and started including client components into the main artifact (see hazelcast/hazelcast#7448). IMO this will make supporting both 3.6 and 4.0 quite challenging from Boot's perspective.

[justinnichols]

The following also works, but it definitely assumes much more than the generic auto-configuration. It will work in the meantime for our application:

import com.hazelcast.config.Config;
import com.hazelcast.config.XmlConfigBuilder;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;

import java.io.IOException;

@Configuration
public class HazelcastConfiguration {
    @Autowired ResourceLoader resourceLoader;

    @Value("${spring.hazelcast.config:classpath:hazelcast.xml}")
    private String hazelcastConfig;

    @Bean
    HazelcastInstance hazelcastInstance() throws IOException  {
        Resource configAsResource = resourceLoader.getResource(hazelcastConfig);

        Config config = new XmlConfigBuilder(configAsResource.getInputStream()).build();
        return Hazelcast.newHazelcastInstance(config);
    }
}

[snicoll]

This is also blocked on micrometer-metrics/micrometer#1697 and if Micrometer drops support for Hazelcast 3.x, that would put pressure on us to upgrade for 2.3 indeed.

The change Vedran mentioned (thanks @vpavic!) is indeed very annoying as it breaks our checks to determine whether the user wants an embedded server or connect to an existing one with the client. We'll probably have to introduce a property with an enum and the user will have to opt-in for the behaviour they want.

[snicoll]

I've added a comment on hazelcast/hazelcast#7448 (comment). If Hazelcast 4.1 split jars after all, then the enum I've just mentioned would be useless. I don't think we should upgrade until this is clarified.

[snicoll]

The current problem that has been described here is due to the merging that Vedran described above. Now that we have both the embedded server and the client on the classpath, the auto-configuration for the client always matches altough the configuration points to an embedded server. You'd get the same problem if you put hazelcast-client on your classpath altough you want to use an embedded server so there might be something we can do here with the enum.

I am going to give that a try and see if the health indicator and cache infrastructure still works. Metrics won't but that's currently worked on the micrometer side of things.

Given that Hazelcast has dropped support for hazelcast-client, this is also a breaking change from a dependency management perspective and you won't be able to simply override the hazelcast version with hazelcast.version unless we start tracking the client with a different property.

[snicoll]

The health indicator is also broken

Caused by: java.lang.NoSuchMethodError: com.hazelcast.core.HazelcastInstance.getLocalEndpoint()Lcom/hazelcast/core/Endpoint;
	at org.springframework.boot.actuate.hazelcast.HazelcastHealthIndicator.lambda$doHealthCheck$0(HazelcastHealthIndicator.java:47) ~[spring-boot-actuator-2.3.0.BUILD-20200407.071319-511.jar:2.3.0.BUILD-SNAPSHOT]
	at com.hazelcast.transaction.impl.TransactionManagerServiceImpl.executeTransaction(TransactionManagerServiceImpl.java:119) ~[hazelcast-4.0.jar:4.0]
	... 47 common frames omitted

We can "fix" that with an additional class check so that it backs off with Hazelcast 4.

[mesutcelik]

Given that Hazelcast has dropped support for hazelcast-client, this is also a breaking change from a dependency management perspective and you won't be able to simply override the hazelcast version with hazelcast.version unless we start tracking the client with a different property.

I think this is the best option to move forward. A new property like spring.hazelcast.client.config is very explicit and people upgrading to newer spring versions would just need to change their configuration to spring.hazelcast.client.config if they were meant to use spring.hazelcast.config for their hazelcast-client configuration.

That is also in sync with how hazelcast explicitly asks their users to configure hazelcast. It is either via hazelcast.config or hazelcast.client.config

[snicoll]

I think this is the best option to move forward.

I am not sure about that and that's not what I meant anyway. The separate property I referred to was to let the user specify if they want a client or a server. It's a bit tricky right now as if we had a way to figure that out upfront, the property would become obsolete.

That is also in sync with how hazelcast explicitly asks their users to configure hazelcast. It is either via hazelcast.config or hazelcast.client.config

I don't think that's in sync. Spring Boot auto-configures one or the other. If you have two configuration items for this, you have to support the case where both properties are set. And configuring both a client and a server if both are set looks wrong to me.

[justinnichols]

I don't think that's in sync. Spring Boot auto-configures one or the other. If you have two configuration items for this, you have to support the case where both properties are set. And configuring both a client and a server if both are set looks wrong to me.

Would it not be a valid use case that a Spring Boot application developer may wish to not only host an embedded Hazelcast server, but also connect to an external one? While it might not be a typical use case, and it certainly wouldn't be a case for us, it doesn't seem out of the realm of possibility. Of course, this would mean that the bean references couldn't be the same and it negates the abstraction layer (HazelcastInstance) provided with the auto-configuration.

I suppose it boils down to what should be supported for auto-configuration. If the intent is to only support one hazelcast configuration and bean for auto-configuration, then having a single property that toggles the mode (client versus server) would seem best. Of course, this is the opinion of a developer using Spring Boot, so please take that with the grain of salt it deserves.

[snicoll]

Would it not be a valid use case that a Spring Boot application developer may wish to not only host an embedded Hazelcast server, but also connect to an external one?

I don't know but that's out-of-scope for an auto-configuration. Doing so would lead to two HazelcastInstance beans in the context and we're not going to do that. I've done that in the past for caching and it complexifies greatly the code and makes it harder for users to resonate about expectations.

then having a single property that toggles the mode (client versus server) would seem best.

I went with that idea but if we find a way to detect the "mode" based on the configuration file, this makes the option rather dumb and unnecessary. The Hazelcast team submitted a PR and then closed it as it was incomplete. I can see they're planning a check for 4.1 which would force us to call that check via reflection as we have to keep compatibility with 3.x at this time.

[mmedenjak]

Without going over the history here again, I believe one of the issues was the inability to detect which type of instance to create - a client or a member/server since we merged the client code into the same jar. To that end, we merged a PR with an API to detect if the provided declarative configuration corresponds to a client or member configuration (hazelcast/hazelcast#17093). The approach is best-effort, since in some cases the configuration can be reused for both a client and a member. Hazelcast 4.0.2 that contains this API will probably be released today.

[snicoll]

Thanks for the feedback @mmedenjak. We’ll see what we can do with Spring Boot 2.4.

[snicoll]

Actually, Spring Session does not have support for Hazelcast 4 yet. I've asked for some clarification.

[snicoll]

I've started to work on this on a branch with the Spring Session tests for Hazelcast failing as expected, see ff880a5.

[maslano]

If anyone cannot upgrade spring-boot(or cloud) and has that problem, here is a better workaround that will allow you to have exact same functionality, but the parameter will be named spring.hazelcast-workaround.config, simply put this in your main config class:

    @Bean
    @Primary
    @ConditionalOnProperty(prefix = "spring.hazelcast-workaround", name = "config")
    public HazelcastProperties hzAutoConfProperties(@Value("${spring.hazelcast-workaround.config}") Resource configResource) {
    	HazelcastProperties properties = new HazelcastProperties();
    	properties.setConfig(configResource);
    	return properties;
    }