cmsantos
Reputation: 327
Fallback disables - Hystrix
I'm using Hystrix in my environment associated with Liferay and other external services. My circuit gets some fails and opens the circuit to a specific service. Everything is fine so far. However, the circuit never changes to closed again. My fallback is disabled with this property:
hystrix.command.default.fallback.enabled=false
What should I do to circuit checks that my service is working and change to closed again? I have verified the circuits with the Hystrix Dashboard.
com.netflix.hystrix.exception.HystrixRuntimeException: BusinessActivity-FlightAvailabilityV1 timed-out and fallback disabled.
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:843) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:59) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.5.jar!/:1.1.5]
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:953) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:970) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_111]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_111]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.util.concurrent.TimeoutException: null
... 18 common frames omitted
2017-05-11 17:11:14.979 WARN 97456 --- [io-8080-exec-14] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
com.netflix.hystrix.exception.HystrixRuntimeException: BusinessActivity-FlightAvailabilityV1 short-circuited and fallback disabled.
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:843) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:59) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:421) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:369) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.Observable.unsafeSubscribe(Observable.java:8460) ~[rxjava-1.1.5.jar!/:1.1.5]
at com.netflix.hystrix.AbstractCommand$ObservableCommand$1.call(AbstractCommand.java:1133) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.AbstractCommand$ObservableCommand$1.call(AbstractCommand.java:1129) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.Observable.subscribe(Observable.java:8553) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.Observable.subscribe(Observable.java:8520) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:57) ~[rxjava-1.1.5.jar!/:1.1.5]
at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:401) ~[rxjava-1.1.5.jar!/:1.1.5]
at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:334) ~[hystrix-core-1.5.2.jar!/:1.5.2]
at br.com.smiles.hystrix.routes.SmilesHostRoutingFilter.run(SmilesHostRoutingFilter.java:196) ~[smiles-hystrix-server.jar!/:1.0.1-SNAPSHOT]
at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:120) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.1.0.jar!/:1.1.0]
at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.1.0.jar!/:1.1.0]
at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequestInternal(ZuulController.java:43) [spring-cloud-netflix-core-1.1.0.RELEASE.jar!/:1.1.0.RELEASE]
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) [spring-webmvc-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:261) [spring-boot-actuator-1.3.5.RELEASE.jar!/:1.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:115) [spring-boot-actuator-1.3.5.RELEASE.jar!/:1.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.5.RELEASE.jar!/:1.3.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1502) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1458) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.33.jar!/:8.0.33]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:418) ~[hystrix-core-1.5.2.jar!/:1.5.2]
... 85 common frames omitted
Here are my hystrix properties:
# POOL SIZES
hystrix.threadpool.default.coreSize=70
#QUEUES
hystrix.threadpool.checkout.queueSizeRejectionThreshold=5
#TIMEOUTS
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=30000
hystrix.command.getMember.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.checkout.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.joinClub.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.getAvailability.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.myFlights.execution.isolation.thread.timeoutInMilliseconds=10000
hystrix.command.login.execution.isolation.thread.timeoutInMilliseconds=10000
#general HTTP TIMEOUT
zuul.host.socket-timeout-millis=70000
zuul.host.maxTotalConnections=600
#hystrix.command.checkout.circuitBreaker.forceOpen=true
hystrix.threadpool.default.metrics.rollingStats.timeInMilliseconds=10000
hystrix.command.default.circuitBreaker.requestVolumeThreshold=7
# FALLBACKS
hystrix.command.default.fallback.enabled=false
metrics.healthSnapshot.intervalInMilliseconds=500
Upvotes: 0
Views: 6143
Answers (2)
formerly1
Reputation: 11
The result of my test is that:
if hystrix.command.default.fallback.enabled = true, the circuit breaker can turn off after a specified time, otherwise, it cannot be closed normally
Upvotes: 0
yongsung.yoon
Reputation: 5589
As you know circuit is opened, all requests during sleepWindowInMilliseconds (default 5000ms) will be rejected. After sleepWindowInMillisec time, your circuit becomes half-opened state and it allows just SINGLE request to flow to your backend server. If it fails, your circuit will become opened again and all following requests will be rejected during next sleepWindowInMillisec time. If the request that was arrived at half-opened state succeeds, your circuit will become closed state.
If your circuit looks opened always, it means that the request at half-opened state failed again. You can check the situation with exception message.
Upvotes: 3
Related Questions
- Hystrix fallback method is not invoked
- Hystrix fallback method does not run
- Hystrix-javanica -Hystrix Timeout fallbacks are not triggering
- Hystrix: cannot trip circuit breaker
- Hystrix fallback method gives error
- hystrix first request error fallback
- Hystrix Fallback method execution
- Hystrix Calls the fallbackMethod even when backend API gets executed
- Fallback chain in hystrix
- Hystrix circuit breaker with business exceptions