ipyparallel Exception: "You have attempted to connect to an IPython Cluster but no Controller could be found"

Question

I've just started using ipyparallel, I'm using VS2017 and importing it as;

import ipyparallel as ipp

And then attempting to start it using;

def main():
    rc = ipp.Client()


if __name__ == "__main__":

    main()

in the python window I get the message: "Waiting for connection file: ~.ipython\profile_default\security\ipcontroller-client.json"

It hangs on the rc = ip.Client() line and gives the exception in the question. There doesn't appear to be any file called profile_default at that location, let alone a json file.

Anyone know how to fix this?

Answer 1

This error occurs if no cluster is running.

import ipyparallel as ipp
rc = ipp.Client()

Here's the error message:

Waiting for connection file: ~/.ipython/profile_default/security/ipcontroller-client.json

---

OSError Traceback (most recent call last) in ----> 1 rc = ipp.Client() OSError: Connection file '~/.ipython/profile_default/security/ipcontroller-client.json' not found. You have attempted to connect to an IPython Cluster but no Controller could be found. Please double-check your configuration and ensure that a cluster is running.

Here's a function to show which clusters are running:

def show_clusters():
    clusters = ipp.ClusterManager().load_clusters() 
    print("{:15} {:^10} {}".format("cluster_id", "state", "cluster_file")) 
    for c in clusters:
        cd = clusters[c].to_dict()
        cluster_id = cd['cluster']['cluster_id']
        controller_state = cd['controller']['state']['state']
        cluster_file = getattr(clusters[c], '_trait_values')['cluster_file']
        print("{:15} {:^10} {}".format(cluster_id, controller_state, cluster_file))
        

In this case there were no clusters running.

show_clusters()
# cluster_id        state    cluster_file

In Jupyter notebooks one can enable a tab for clusters (ipcluster nbextension enable). This is how it looks when no cluster is running:

Start a cluster

cluster = ipp.Cluster(n=4)
await cluster.start_cluster() # or cluster.start_cluster_sync() without await
# Using existing profile dir: '/Users/X/.ipython/profile_default'
# Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

show_clusters()
# cluster_id        state    cluster_file
# 1648935811-e7r2  running /Users/X/.ipython/profile_default/security/cluster-1648935811-e7r2.json

Now you can define a client using the cluster id:

rc = ipp.Client(cluster_id='1648935811-e7r2')

This is how the Jupyter notebook clusters tab looks after starting a cluster:

If you start the default cluster from the Jupyter notebook tab (not sure how to do this on the command-line), the command rc = ipp.Client() will also work without a cluster id.

Answer 2

Ok this is a dumb newbie question. The answer is in the docs. The .json files are created when you first use ipcluster in powershell. I was looking for profile_default in anaconda but its in .ipython/ in Users.