CODEWITHSUNDEEP
artifactoryjfrog-clijfrog-mission-control
PostgresQuestion
PostgresQuestion

Reputation: 95

Artifactory JFrog Backup failing with error code 401

Will appreciate if someone can please guide or provide me pointers to debug the backup issues with Artifactory. Whenever backups are performed - there is always an error message of 401 on /api/v1/system/backup/export in the artifactory.log The backups exist on backup location but with an error message in the log. Not sure how can debug and implications of this error in the logs. I can see in the stack the rest call is failing, have tried setting a password to unsupported and multiple other things but the error persists. Also checked the Jira on Artifactory to no avail. Any pointers will be greatly appreciated

More details

artifactory.version=5.9.3
artifactory.timestamp=1521564024289
artifactory.revision=50903900
artifactory.buildNumber=820

The backups are failing with following the info in the log.

2018-04-24 11:59:24,620 [ajp-apr-8009-exec-9] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup org.jfrog.access.client.AccessClientHttpException: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]} at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:154) at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:113) ... Following is the full stack as displayed in artifactory.log

2018-04-24 11:59:24,620 [ajp-apr-8009-exec-9] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup org.jfrog.access.client.AccessClientHttpException: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]} at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:154) at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:113) at org.jfrog.access.client.system.SystemClientImpl.exportAccessServer(SystemClientImpl.java:21) at org.artifactory.security.access.AccessServiceImpl.exportTo(AccessServiceImpl.java:1060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) at com.sun.proxy.$Proxy144.exportTo(Unknown Source) at org.artifactory.spring.ArtifactoryApplicationContext.exportTo(ArtifactoryApplicationContext.java:662) at org.artifactory.ui.rest.service.admin.importexport.exportdata.ExportSystemService.execute(ExportSystemService.java:67) at org.artifactory.rest.common.service.ServiceExecutor.process(ServiceExecutor.java:38) at org.artifactory.rest.common.resource.BaseResource.runService(BaseResource.java:92) at org.artifactory.ui.rest.resource.admin.importexport.ExportArtifactResource.exportSystem(ExportArtifactResource.java:65) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184) at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403) at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:212) at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502) at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:877) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2527) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2516) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) 2018-04-24 11:59:24,660 [ajp-apr-8009-exec-9] [INFO ] (o.a.s.ArtifactoryApplicationContext:819) - Note: the etc exported folder has excessive permissions. Be careful with the files.

Upvotes: 1

Views: 4669

Answers (3)

PostgresQuestion
PostgresQuestion

Reputation: 95

Added bootstrap.creds

tomcat@XXXXXXXX:/home/bitnami/apps/artifactory/artifactory_home/etc$ ls -ltrh
   total 136K
  -rw-r--r-- 1 tomcat tomcat 5.5K Mar 20 16:41 mimetypes.xml
  -rw-r--r-- 1 tomcat tomcat  12K Mar 20 16:41 logback.xml
  -rw-r--r-- 1 tomcat tomcat  13K Mar 20 16:41 artifactory.system.properties
  -rw-r--r-- 1 tomcat tomcat  11K Mar 20 16:41 artifactory.config.bootstrap.xml
  -rw-rw-r-- 1 tomcat tomcat  145 Mar 21 13:36 db.properties
  drwxrwxr-x 2 tomcat tomcat 4.0K Mar 21 13:37 ui
  drwxrwxr-x 2 tomcat tomcat 4.0K Mar 21 13:37 plugins
  drwx------ 3 tomcat tomcat 4.0K Mar 21 13:37 security
  -rw-rw-r-- 1 tomcat tomcat  13K Mar 21 13:37 
   artifactory.config.latest.1521639475000.xml
  -rw-r--r-- 1 tomcat tomcat  199 Apr 24 18:50 binarystore.xml
  -rw-rw-r-- 1 tomcat tomcat  13K Apr 24 18:57 artifactory.config.latest.xml_bkp
  -rw-rw-r-- 1 tomcat tomcat  12K Apr 24 18:58 
   artifactory.config.latest.1524596291000.xml
  -rw-rw-r-- 1 tomcat tomcat  14K Apr 25 14:39 artifactory.config.latest.xml
  -rw------- 1 tomcat tomcat   24 Apr 25 15:09 bootstrap.creds
  -rw-rw-r-- 1 tomcat tomcat  864 Apr 25 15:12 artifactory.properties

Contents of bootstrap.creds are access-admin@*=password AND restarted the arttifactory BUT still seeing the same error message

2018-04-25 15:14:29,607 [art-exec-1] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}

I tried the backup using access-admin as well as user from gui with same error (both are admin users).

Following are from the access log when artifactory restarted

2018-04-25 15:10:54,396 [localhost-startStop-1] [INFO ] (o.j.a.c.AccessApplicationContextInitializer:46) - Using JFrog Access home at '/opt/bitnami/apps/artifactory/artifactory_home/access' 2018-04-25 15:10:54,413 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:48) - Starting AccessApplication v3.2.2 on ip-10-51-35-238 with PID 16169 (/opt/bitnami/apache-tomcat/webapps/access/WEB-INF/lib/access-application-3.2.2.jar started by tomcat in /opt/bitnami/apache-tomcat/bin) 2018-04-25 15:10:54,414 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:597) - The following profiles are active: production,grpc 2018-04-25 15:11:11,402 [localhost-startStop-1] [INFO ] (o.j.a.s.d.u.AccessJdbcHelperImpl:129) - Database: Apache Derby 10.11.1.1 - (1616546). Driver: Apache Derby Embedded JDBC Driver 10.11.1.1 - (1616546) 2018-04-25 15:11:11,410 [localhost-startStop-1] [INFO ] (o.j.a.s.d.u.AccessJdbcHelperImpl:132) - Connection URL: jdbc:derby:/opt/bitnami/apps/artifactory/artifactory_home/data/derby 2018-04-25 15:11:16,127 [localhost-startStop-1] [INFO ] (o.j.a.s.s.c.InternalConfigurationServiceImpl:94) - Loading configuration from db finished successfully 2018-04-25 15:11:18,035 [localhost-startStop-1] [INFO ] (o.j.a.s.s.AccessStartupServiceImpl:79) - Found master.key file at /opt/bitnami/apps/artifactory/artifactory_home/etc/security/master.key, using it as master.key 2018-04-25 15:11:19,486 [localhost-startStop-1] [INFO ] (o.j.a.s.s.t.TokenServiceImpl:100) - Scheduling task for revoking expired tokens using cron expression: 0 0 0/1 * * ? 2018-04-25 15:11:19,651 [localhost-startStop-1] [INFO ] (o.j.a.s.r.c.RpcServiceInvoker:86) - Added service: sync 2018-04-25 15:11:20,764 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:93) - [ACCESS BOOTSTRAP] Starting JFrog Access bootstrap... 2018-04-25 15:11:23,381 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:146) - [ACCESS BOOTSTRAP] Updating server '509b68e8-48e5-4bba-8ea7-6564c65d5b37' private key finger print to: 63c2f42824ead3169fc13eab66d3d254d25c659ceef5cdeed21c3110f47ee0d3 2018-04-25 15:11:24,164 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:108) - [ACCESS BOOTSTRAP] JFrog Access bootstrap finished. 2018-04-25 15:11:30,624 [localhost-startStop-1] [INFO ] (o.j.a.s.s.s.RefreshableScheduledJob:56) - Scheduling heartbeat task to run every 5 seconds 2018-04-25 15:11:43,339 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:57) - Started AccessApplication in 53.93 seconds (JVM running for 93.388) 2018-04-25 15:11:47,756 [localhost-startStop-1] [WARN ] (o.a.t.u.s.StandardJarScanner:311) - Failed to scan [file:/opt/bitnami/apache-tomcat/lib/derbyLocale_cs.jar] from classloader hierarchy enter code here

Additionaly I enabled trace on the logs. Following is what I am seeing in

/home/bitnami/apps/artifactory/artifactory_home/access/logs/request.log. Seems the ones with backup call are tagged as coming from anonymous vs |jfrt@01c94cf2cpdmx60xvtsf8j1jy4 for other calls.

2018-04-25T15:49:03.906+0000|127.0.0.1|jfrt@01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|198|JFrog Access Java Client/3.2.2
2018-04-25T15:50:17.553+0000|127.0.0.1|anonymous|POST|http://127.0.0.1/access/api/v1/system/backup/export|401|0|188|JFrog Access Java Client/3.2.2
2018-04-25T15:50:18.657+0000|127.0.0.1|jfrt@01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/users/|200|0|84|JFrog Access Java Client/3.2.2
2018-04-25T15:50:18.733+0000|127.0.0.1|jfrt@01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|73|JFrog Access Java Client/3.2.2
2018-04-25T15:50:42.823+0000|127.0.0.1|anonymous|POST|http://127.0.0.1/access/api/v1/system/backup/export|401|0|134|JFrog Access Java Client/3.2.2
2018-04-25T15:50:42.995+0000|127.0.0.1|jfrt@01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/users/|200|0|114|JFrog Access Java Client/3.2.2
2018-04-25T15:50:43.086+0000|127.0.0.1|jfrt@01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|64|JFrog Access Java Client/3.2.2

Please let me know if I can provide additional relevant information that can help you to get more insight in my envoirnment. Thank you again for you help

Upvotes: 0

Ariel Kabov
Ariel Kabov

Reputation: 866

Thanks for the feedback. I suspect the reason for the behavior is that for the more common operations Artifactory uses the <adminToken> from the Config Descriptor, however some other operations use credentials saved in $ARTIFACTORY_HOME/etc/security/access/access.creds.

These seem to be bad.

In order to restart these you should:

  1. Create the following file: $ARTIFACTORY_HOME/access/etc/bootstrap.creds
  2. The content of the file should be: access-admin@*=password
  3. The permissions of the file must be 600.

Once that file is in place, restart Artifactory.

Upvotes: 4

Ariel Kabov
Ariel Kabov

Reputation: 866

Since Artifactory version 5.4 there is a separate service in Artifactory named "Access", which manages permissions and security related configurations.

Artifactory authenticates with Access using an access token that is configured in the "Config Descriptor".

It is possible that this issue will be resolved by a simple restart, so that's what i would have started with. If this doesn't work, or what i suspect will happen is Artifactory won't be able to start, please do (while Artifactory is shut down):

  1. Remove the <adminToken> tag from the $ARTIFACTORY_HOME/etc/artifactory.config.latest.xml
  2. $ cp artifactory.config.latest.xml artifactory.config.import.xml
  3. Restart Artifactory.

Upvotes: 1

Related Questions