npm install package from github repo subfolder

Question

Is it possible to install npm package from github when the package located inside subfolder?

For example, we have Microsoft BotBuilder repository: https://github.com/Microsoft/BotBuilder

But I need to install package inside subfolder "Node/core/": https://github.com/Microsoft/BotBuilder/tree/master/Node/core/

So how can I install it with npm?

Answer 1

If GitPkg doesn't work for you, npm has an awesome feature called 'workspaces' since v7. These are your friend for managing, deploying, etc from a subfolder of your main repo. They allow you to define an npm identity for your subfolder, and to make it npm install-able in both the local repo and in others.

Defining your workspace is as simple as defining a folder where packages will live. Name it after your GitHub org, which we'll call '@namespace'. Put your package-name subfolder within that, and then add a single line to the package.json at the repo root:

"workspaces": ["@namespace/package-name"],

Then, run npm init -w @namespace/package-name to create a package.json in the subfolder/workspace. (The -w flag defines your workspace as the 'scope' of the npm command.)

Finally, run npm install and all this magic will result in your node_modules for the root project now symlinking to that workspace subfolder as a dependency. You can now start referencing your workspace code like any other dependency, with import { thing } from '@namespace/package-name'.

But, your real goal was to publish it into other projects, right? Well, for local dev, you can just npm install from the repo on your local machine, by referring to that workspace, such as (for a sibling folder): npm i -D ../repo-name/@namespace/package-name.

But, what about when you get beyond local dev? What about when you need to install dependencies in CI for deployment, or let devs install without needing the whole repo that houses the package? For that, you need an npm-compatible package directory. You can use the npmjs.com directory, of course, and it's free if your repo is public. You just npm publish -w @namespace/package-name to publish from that workspace subfolder. If your repo is private though, publishing to npmjs.com will cost you.

The new dark-horse option: especially if you're already using a paid GitHub plan, you can publish to and install from GitHub Packages for free. The GitHub Packages docs will cover this in detail, but it basically all boils down to using your .npmrc to define which registry has control over that @namespace you defined in the package.json. Place this line in .npmrc in the home repo containing the workspace, plus in all other repos that want to install the workspace package:

@namespace:registry=https://npm.pkg.github.com

Henceforth, npm will know to publish or install any dependencies within @namespace from GitHub Packages. And you'll have accomplished publication of a sub-folder at no extra cost.

(Note: GitPkg is a wonderful community service, and I tried it before going the workspaces/GitHubPackages route. However, it's the side project of a busy graduate student, and there are currently lots of 500 errors reported by users like me who could never make it work with their repo for some reason. Great if it works for you, but otherwise the above will get it done, and perhaps more scalably.)

Answer 2

If the package source is hosted on GitHub, you can use GitPkg like this:

# using npm:
npm install https://gitpkg.now.sh/<user>/<project>/<subdir>?<commit-ish>
# using yarn:
yarn add https://gitpkg.now.sh/<user>/<project>/<subdir>?<commit-ish>

For your particular case, the URL would be this:

https://gitpkg.now.sh/Microsoft/BotBuilder/Node/core?master

There is a nice wizard-like form at their site that helps build the URL, and the command to install it.

Answer 3

Inspired by @Tomasz Jakub Rup's answer. I updated his example and pointed to the 3.0 branch which his example was based on and instead used the new-ish git feature called sparse-checkout. This feature will save time/bandwidth as it doesn't need to clone the entire repository ahead of time and will only grab what you directed. Many servers however don't support --filter option which does the bulk of the space saving, but --depth 1 in many cases will still cut down on bandwidth.

I used a .tmp_npm folder to minimize overwriting and also to possibly be .gitignored due to being a hidden file.

"scripts": {
  "postinstall": "mkdir .tmp_npm; cd .tmp_npm; git init; git clone --filter=blob:none --no-checkout --depth 1 --sparse -b 3.0 https://github.com/Microsoft/BotBuilder.git; cd BotBuilder/; git sparse-checkout init --cone; git sparse-checkout add Node/core; git checkout; cd ../..; npm i .tmp_npm/BotBuilder/Node/core/"
  ...
},

Answer 4

Paste the github link to the subfolder into gitpkg. You can then use this along with yarn or npm to install the package from a github sub folder.

https://gitpkg.now.sh/

Answer 5

Might be slightly off topic, just still relevant to the question

https://git-scm.com/book/en/v2/Git-Tools-Submodules

Git Submodules are git repos that you can use in other repos (henceforth, referred to as Supermodules). With each submodule having the usual assortment of branches features and tags, the benefit comes from each supermodule being a version controlled, pluggable components, that can be worked on separately or developed alongside the supermodule.

A Few Useful Commands

To add a submodule, you run the following inside your supermodule:

git submodule add <url-to-submodule-repo>

The submodule(s) still have to be initialized and fetched from the repo:

git submodule init git submodule update

A supermodule with submodules can be cloned and all submodules fetched by running:

git clone --recursive <url-to-supermodule>

You can pull upstream changes to a submodule's branch by running the following inside the submodule directory:

git fetch

Then run the following to update local code:

git merge

The following will fetch and merge for all submodules in your supermodule:

git submodule update --remote

If you want to track a specific branch of a submodule you can use the following:

git config -f .gitmodules submodule.<my-submodule>.branch fantastic_new_implementation

If you have worked on your supermodules and submodules and you push your supermodule, changes made to submodules will only exist locally and those you are collaborating with will not know of these changes. The following command will check if your submodules have been pushed BEFORE attempting to push your supermodule

git push --recurse-submodules=check

Finally, here is a useful ForEach command, that allows us to run a command for each submodule

git submodule foreach 'git checkout -b featureA

Answer 6

Add to package.json:

...
"scripts": {
  "postinstall": "mkdir BotBuilder; cd BotBuilder; git init; git remote add -f origin https://github.com/Microsoft/BotBuilder.git; git config core.sparseCheckout true; echo \"Node/core\" >> .git/info/sparse-checkout; git pull --depth=1 origin master; cd ..; npm i ./BotBuilder/Node/core/"
  ...
},
...

postinstall script is running after the package is installed.

And step by step:

1. Make folder to clone repo: mkdir BotBuilder
2. enter to the folder: cd BotBuilder
3. init git repo: git init
4. set git origin to Microsoft/BotBuilder repo: git remote add -f origin https://github.com/Microsoft/BotBuilder.git
5. enable sparse checkout: git config core.sparseCheckout true
6. add Node/core to checkout list: echo "Node/core" >> .git/info/sparse-checkout
7. pull part of repo: git pull --depth=1 origin master
8. enter to Your app folder: cd ..
9. install BotBuilder: npm i ./BotBuilder/Node/core/