Best way to package a Python CLI outside of its associated library

Question

I'm packaging a library and a CLI associated to it. I'm using flit

The library makes use of long-to-import dependencies (numpy, torch and whatnot), so I would like to split the CLI part so that it doesn't need to do the imports to have, e.g., shell completion. Of course, it will have to do the import to execute some command - but that's fine and expected.

Let me illustrate with an attempt that does not work:

.
├── pyproject.toml
│   | 
│   | ...
│   |
│   | [project.scripts]
│   | mycli = "mylibrary.cli:main "
│   |
│   | ...
│   |
│
└── src/mylibrary
    ├── __init__.py
    │   | 
    │   | from . import foo, cli
    │   |
    │
    ├── foo.py
    │   | 
    │   | import very_long_import
    │   |
    │
    └── cli.py

Now even something like mycli --help will need to do a import very_long_import, which is unecessary.

On the other hand, I can structure my package as

.
├── pyproject.toml
│   | 
│   | ...
│   |
│   | [project.scripts]
│   | mycli = "mylibrary.cli:main "
│   |
│   | ...
│   |
│
└── src/mylibrary
    ├── __init__.py
    │   | 
    │   | # empty
    │   |
    │
    ├── foo.py
    │   | 
    │   | import very_long_import
    │   |
    │
    └── cli.py

Now mycli --help is fast, but I can no longer write

import mylibrary

mylibrary.foo.do_some_stuff()

I have to instead write

import mylibrary.foo

mylibrary.foo.do_some_stuff()

While before I could do both.

Ideally, I would want something like

.
├── pyproject.toml
│   | 
│   | ...
│   |
│   | [project.scripts]
│   | mycli = "cli:main "
│   |
│   | ...
│   |
│
└── src
    ├── cli
    │   ├── __init__.py
    │   └── main.py
    │
    └── mylibrary
        ├── __init__.py
        │   | 
        │   | from . import foo, cli
        │   |
        │
        └── foo.py
            | 
            | import very_long_import
            |

where the CLI is shipped as a script on top of the library, not an entry point that's part of the library. But of course here - with flit at least - we get

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/bin/mylibrary", line 5, in <module>
    from cli import main
ModuleNotFoundError: No module named 'cli'

Do you know a way to achieve this library/script separation, either with flit or some other building tools (as long as I only have to write a pyproject.toml and no setup.py junk 😇).