user3605317
Reputation: 143
jupyterhub starts up fine, but unable to spawn a server
I have installed Anaconda and jupyterhub. When I start jupyterhub on centOS7. The authentication to Linux machine is through Active Directory. it starts fine however the user123 (who is active directory user) tried to login, is unable to spawn a server. This is what I see in the logs:
`
[root@ussd-dev-jptr01 user123]# jupyterhub -f /home/user123/jupyterhub_config.py
[W 2019-10-30 13:33:56.573 JupyterHub app:626] JupyterHub.proxy_cmd is deprecated in JupyterHub 0.8,
use ConfigurableHTTPProxy.command
[I 2019-10-30 13:33:56.573 JupyterHub app:2120] Using Authenticator: j
upyterhub.auth.PAMAuthenticator-
1.0.0
[I 2019-10-30 13:33:56.573 JupyterHub app:2120] Using Spawner:
jupyterhub.spawner.LocalProcessSpawner-
1.0.0
[I 2019-10-30 13:33:56.576 JupyterHub app:1257] Loading cookie_secret from
/home/user123/jupyterhub_cookie_secret
[I 2019-10-30 13:33:56.592 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2019-10-30 13:33:56.593 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2019-10-30 13:33:56.593 JupyterHub app:1534] Add any administrative users to
c.Authenticator.admin_users in config.
[I 2019-10-30 13:33:56.593 JupyterHub app:1563] Not using whitelist. Any authenticated user will be
allowed.
[I 2019-10-30 13:33:56.636 JupyterHub app:2337] Hub API listening on http://127.0.0.1:8081/hub/
[I 2019-10-30 13:33:56.637 JupyterHub proxy:645] Starting proxy @ http://:8000
13:33:56.846 [ConfigProxy] info: Proxying https://*:8000 to (no default)
13:33:56.850 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
13:33:57.098 [ConfigProxy] info: 200 GET /api/routes
[I 2019-10-30 13:33:57.098 JupyterHub proxy:319] Checking routes
[I 2019-10-30 13:33:57.099 JupyterHub proxy:399] Adding default route for Hub: / =>
http://127.0.0.1:8081
13:33:57.101 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
13:33:57.102 [ConfigProxy] info: 201 POST /api/routes/
[I 2019-10-30 13:33:57.102 JupyterHub app:2422] JupyterHub is now running at http://:8000
[W 2019-10-30 13:34:13.664 JupyterHub auth:908] PAM Authentication failed
([email protected]@::ffff:10.12.68.203): [PAM Error 7] Authentication failure
[W 2019-10-30 13:34:13.665 JupyterHub base:670] Failed login for [email protected]
[I 2019-10-30 13:34:13.695 JupyterHub log:174] 200 POST /hub/login?next= (@::ffff:10.12.68.203)
148.89ms
[I 2019-10-30 13:34:28.609 JupyterHub base:663] User logged in: user123
[I 2019-10-30 13:34:28.611 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn
(user123@::ffff:10.12.68.203) 197.70ms
[I 2019-10-30 13:34:28.689 JupyterHub spawner:1387] Spawning /apps/Install/anaconda3/bin/jupyterhub-
singleuser --port=55459 '--notebook-dir=~/notebooks'
[W 2019-10-30 13:34:29.073 SingleUserNotebookApp configurable:168] Config option open_browser not
recognized by SingleUserNotebookApp. Did you mean browser?
[I 2019-10-30 13:34:29.251 SingleUserNotebookApp extension:59] JupyterLab extension loaded from
/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 2019-10-30 13:34:29.251 SingleUserNotebookApp extension:60] JupyterLab application directory is
/apps/Install/anaconda3/share/jupyter/lab
[I 2019-10-30 13:34:29.254 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser
server version 1.0.0
[I 2019-10-30 13:34:29.264 JupyterHub log:174] 200 GET /hub/api (@127.0.0.1) 1.18ms
[I 2019-10-30 13:34:29.266 SingleUserNotebookApp notebookapp:1825] Serving notebooks from local
directory: /home/user123/notebooks
[I 2019-10-30 13:34:29.266 SingleUserNotebookApp notebookapp:1825] The Jupyter Notebook is running
at:
[I 2019-10-30 13:34:29.266 SingleUserNotebookApp notebookapp:1825] https://ussd-dev-
jptr01.illumina.com:55459/user/user123/
[I 2019-10-30 13:34:29.266 SingleUserNotebookApp notebookapp:1826] Use Control-C to stop this server
and shut down all kernels (twice to skip confirmation).
[I 2019-10-30 13:34:29.269 SingleUserNotebookApp singleuser:542] Updating Hub with activity every
300 seconds
[I 2019-10-30 13:34:29.284 JupyterHub log:174] 200 POST /hub/api/users/user123/activity
([email protected]) 13.79ms
[W 2019-10-30 13:34:29.503 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42800): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:29.556 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42802): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[I 2019-10-30 13:34:29.666 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/user123
(user123@::ffff:10.12.68.203) 1011.24ms
[I 2019-10-30 13:34:29.749 JupyterHub pages:303] user123 is pending spawn
[I 2019-10-30 13:34:29.756 JupyterHub log:174] 200 GET /hub/spawn-pending/user123
(user123@::ffff:10.12.68.203) 17.19ms
[W 2019-10-30 13:34:32.714 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42810): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:35.644 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42812): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:38.664 JupyterHub base:932] User user123 is slow to become responsive
(timeout=10)
[W 2019-10-30 13:34:40.650 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42814): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:43.563 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42816): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:48.567 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42818): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:53.571 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42820): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:58.575 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42822): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:59.444 SingleUserNotebookApp iostream:1407] SSL Error on 12 ('127.0.0.1',
42824): [SSL: HTTP_REQUEST] http request (_ssl.c:1076)
[W 2019-10-30 13:34:59.445 JupyterHub user:678] user123's server never showed up at
http://127.0.0.1:55459/user/user123/ after 30 seconds. Giving up
[I 2019-10-30 13:34:59.446 SingleUserNotebookApp notebookapp:1694] Shutting down 0 kernels
[E 2019-10-30 13:34:59.704 JupyterHub gen:599] Exception in Future <Task finished coro=
<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at
/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py:800>
exception=TimeoutError("Server at http://127.0.0.1:55459/user/user123/ didn't respond in 30
seconds")> after timeout
Traceback (most recent call last):
File "/apps/Install/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 593, in
error_callback
future.result()
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 807,
in finish_user_spawn
await spawn_future
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 654, in spawn
await self._wait_up(spawner)
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 701, in
_wait_up
raise e
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 669, in
_wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in
wait_for_http_server
timeout=timeout,
File "/apps/Install/anaconda3/lib/python3.7/site-packages/jupyterhub/utils.py",
line 177, in
exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:55459/user/user123/ didn't respond in 30 seconds
[I 2019-10-30 13:34:59.706 JupyterHub log:174] 200 GET /hub/api/users/user123/server/progress
(user123@::ffff:10.12.68.203) 29594.20ms `
`
Please let me know what am I missing?
Thanks
Upvotes: 1
Views: 10013
Answers (3)
Shaikh Islam
Reputation: 23
In my case, there was a conflict in the python version. When I was using sudo pip, it was using pip 23.0.1 from python 3.11; when I was using pip, it's version was pip 22 from python 3.10.
So it's better to specify which pip by this command
python3.10 -m pip install <package name>
Upvotes: 0
Roshan Chauhan
Reputation: 84
check event logs of user image/pod, for example, if you've logged in with admin user then jupyter-admin named pod must be present in your cluster.
run kubectl get pods <user-po-name> -n <jhub-namespace>
now, check for event section,
It will show an error because of which container didn't spawn.
It must be issue in pulling images or postHook.
Addition information is needed too like your config.yaml, jhub version, etc.
Upvotes: 0
user2314737
Reputation: 29427
I had the same error on a Linux machine and got rid of it by freeing some memory on the system.
I assume the TimeoutError was just a regular 30 seconds timeout due to a slow response of the system. Some process filling the RAM on the system was causing it to start swapping. To check memory usage:
free -h
and then
htop -s PERCENT_MEM
to find out what process is using up the memory.
Upvotes: 0
Related Questions
- Jupyterhub service unavailable error and http :403 forbidden
- JupyterHub fails to spawn new session with 'No module named IPythonHandler'
- JupyterHub: regular error message: "Your server at ... is not running. Would you like to restart it?"
- failed to run jupyterhub when changing default port
- Can't run jupyterhub in docker container
- Can not access Jupyter server on Raspberry Pi
- JupyterHub Connection refused when access using curl
- How to config startup script for jupyterhub?
- Ubuntu Jupyterhub: 500 POST Spawner failed to start [status=1]
- Jupyterhub not running on Ubuntu 14.04