CODEWITHSUNDEEP
pythoncachingpackagingmemoization
Sean Lake
Sean Lake

Reputation: 608

Is there a standard location to store function cache files in Python?

I have these scripts that perform a lot of expensive computations. One optimization I do is to evaluate the expensive function on a grid, cache the grid, and interpolate for calls within the grid. It is important that the files be written out to disk so that future sessions can benefit from the cache. When this script was just for my own personal use, I could put the cache wherever. Now that I want to incorporate it into a package for wider distribution, I find that I need to have some way to know where is it kosher to store cache files for a package? Do I just need to make some decisions, or are there some procedures I should follow (e.g. let the user modify the directory at install time, at run-time, etc)?

Edit: this code will also have functions based on interpolating data from other sources. So I will need a place to store those, too, but need to know where it is standard to do so and how to detect where that is on a particular install (something hard coded at install time, or can Python modules detect where they're installed at runtime?). Point being, this location needs to be persistent in a way that I understand temporary directories not to be. Therefore, this would ideally be done inside of the package's directory structure or somewhere in the user's home directory. If in the package's directories, it is acceptable if I have to do shenanigans with permissions to make a directory that the user can modify.

Upvotes: 10

Views: 7623

Answers (4)

Jack G
Jack G

Reputation: 5331

For those who want the awesomeness of Danilia's platformdirs—so reliable/portable that even Pip uses it—without the suckiness of needless bloat, the full implementation logic of user_cache_dir reduces to:

# SPDX-License-Id: MIT
# Credit: Jack Giffin and platformdirs: github.com/tox-dev/platformdirs
# Source: https://stackoverflow.com/a/79403791/5601591
def user_cache_dir_from_platformdirs():
    from sys import platform, path
    from os import getenv, path
    if platform == "darwin":
        return os.path.expanduser("~/Library/Caches")
    elif platform == "win32":
        try: # https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid
            from ctypes import windll, wintypes, create_unicode_buffer, c_int
            buf, gfpW = create_unicode_buffer(1024), windll.shell32.SHGetFolderPathW
            gfpW.argtypes = [wintypes.HWND,c_int,wintypes.HANDLE,wintypes.DWORD,wintypes.LPWSTR]
            gfpW.restype = wintypes.HRESULT
            if 0 == gfpW(None, 28, None, 0, buf) and buf[0] != 0:
                return buf.value # CSIDL_LOCAL_APPDATA = 28
        except Exception:
            pass
        if getenv("LOCALAPPDATA") and path.isdir(getenv("LOCALAPPDATA")):
            return getenv("LOCALAPPDATA")
        from winreg import OpenKey, QueryValueEx, HKEY_CURRENT_USER
        key = OpenKey(HKEY_CURRENT_USER, r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders")
        return str( QueryValueEx(key, "Local AppData")[1] )
    elif getenv("ANDROID_DATA") == "/data" and getenv("ANDROID_ROOT") == "/system" and not getenv("SHELL") and not getenv("PREFIX"):
        try:
            from android import mActivity
            context = cast("android.content.Context", mActivity.getApplicationContext())
            return context.getFilesDir().getAbsolutePath() + "/cache"
        except Exception:
            try:
                from jnius import autoclass  # noqa: PLC0415
                context = autoclass("android.content.Context")
                return context.getFilesDir().getAbsolutePath() + "/cache"
            except Exception:
                from re import match
                for dir in path:
                    if match(r"/data/(data|user/\d+)/(.+)/files", dir):
                        return path.split("/files")[0] + "/cache"
                for dir in path:
                    if match(r"/mnt/expand/[a-fA-F0-9-]{36}/(data|user/\d+)/(.+)/files", dir):
                        return path.split("/files")[0] + "/cache"
    # For all Linux and *nix including Haiku, OpenIndiana, and the BSDs:
    return getenv("XDG_CACHE_HOME","").strip() or path.expanduser("~/.cache")

NOTICE: above differs subtly from platformdirs in that it attempts to return the generic local-user cache dir, whereas platformdirs bases its behavior upon the provided name of your software.

Upvotes: 1

Danila Vershinin
Danila Vershinin

Reputation: 9875

I have the same requirements.

It seems like the way to go is storing to ~/.cache/<app name> (Linux).

For covering different operating systems, getting similar location can be simplified with platformdirs (src, PyPI, conda-forge), e.g.:

from platformdirs import user_cache_dir
cachedir = user_cache_dir("<app name>", "<app author>")

Upvotes: 13

Akarshit Sharma
Akarshit Sharma

Reputation: 11

I found some cached files here C:\Users\User_Name\AppData\Local\pip\cache

It is the folder where pip is installed and cache files/libraries are present.

If you uninstall and reinstall python, then these cached libraries will be used, instead of downloading new ones.

Upvotes: 1

Jacques Gaudin
Jacques Gaudin

Reputation: 16988

I would consider creating a named temporary file with a prefix that can be recognized by your scripts, so that the file can be reopened in a later session. the file creation would look like:

tempfile.NamedTemporaryFile(prefix='myApp', delete=False)

Otherwise, I would default to the current working directory and provide an option to set a folder at run-time with argparse.

Upvotes: 0

Related Questions