getting npm modules in node_modules under git control

Question

I have many git repositories each with their own npm package.json file that reference each other. When I install the main package with npm install it successfully clones all git repos referenced in all package.json into the node_modules folder. However they are not cloned with the .git folder for me to keep any changes under source control. Is it possible to npm install these packages and to get the .git folder?

Thanks

Answer 1

They're not for everyone but I really like using git submodules for such things so I wanted to work out a way to use them for this:

1. Add submodule under local_modules/:

   git submodule add -b develop git@github.com:JarvusInnovations/git-client.git local_modules/git-client
   

2. Use file: prefix to declare dependency in package.json:

   {
     "dependencies": {
       "git-client": "file:local_modules/git-client"
     }
   }
   

3. Refresh node_modules/:

   npm install
   

Now NPM creates a symlink under node_modules/ to ../local_modules/git-client and you can commit whatever HEAD you want your submodule to be at to the main project. So your dev workflow can look like

1. Make a set of changes across the main project and the module that go hand-in-hand
2. Make a commit within the submodule to add/change features
3. Make a commit within the main project that includes both the change in submodule version and the changes in the main project that go along with it locked together

Now when other developers clone (--recursive) or pull (and then run git submodule update --init) your pre-release work they can be guaranteed to have the right pre-release module code that goes with it

Answer 2

Late to the party but after reading several posts including this one and a few head bashing/testing hours later I feel compelled to share my take on a local repo under version control being used as a package in a node project.

Do this all from the root of your project:

npm link /path/to/your/local/package/

npm link <"name:"> "name:" key in your local package's package.json

npm install --save /path/to/your/local/package/

the last one you will not find in the npm documentation on using "link" but it's critical

note: you can use relative paths (e.g. ../<localpackagefoldername> if sharing a parent folder)

important: your local package package.json MUST have a package "name:" key and a "main:" key pointing to an entry point js file otherwise this all fails.

now you can use require('name') in your code and later if you publish to npm you don't have to change anything expect the line in project package.json which would be just as easy to delete and npm install as edit.

If you add a package to your local module with npm install then do the same in your project and it will be added to the project's node_modules. If you do an npm uninstall in your local package then do an npm prune in your project

note: if you have run npm install in your local package root then node_modules was created there for that package and now when you npm install in your project you'll get warnings like this you can ignore skippingAction Module is inside a symlinked module: not running remove. If that bugs you then delete the node_modules folder in your local package if you are not running it separately.

Starting with npm 3 node_modules are now flattened so doing it this way means your local package dependencies within the project are flattened too! Further, you can't just put your full git repo into node_modules as npm will despise the .git folder and of course you'll be nesting your node_modules.

My Big Trick: Sure you can have your local package repo in a directory separate from your project but if you make it a git submodule in a subdirectory of your project you get the best of both worlds, flattened dependencies in node_modules and combined pushes of the project and package (submodule) at the same time. Plus it makes those link command paths trivial.

Tip: If you go the separate directory (no submodule) route then use your ide editor (e.g. atom) to add the local package folder to your project tree for easy editing with your project. Too, if you go this route it's up to you to commit and push changes for the local package since it's not a submodule.

Probably the only caveat I can think of at this time is to be sure that you have dependencies entries in your local package even if they are in the project's package.json otherwise if someone uses the local package somewhere else (on it's own) it will be missing dependencies for npm install

Answer 3

In general files inside node_modules are expected to be managed by npm. Node wasn't originally designed that way but npm has evolved to become the standard package management tool for node.js (it even ships with node!).

If you have your own modules that you would like to maintain there are several ways to handle it.

Package specific modules

(also sometimes called "libs")

The simplest way to write your own module is to require with relative or absolute path:

var my_module = require('./my_module'); // notice '.js' is not needed

Most people put all package specific modules in a directory called lib or src. So you can require them as:

var my_module = require('./lib/my_module');

Note that the path is relative to the file you're editing. So if a lib file requires another lib file you don't need the '/lib'.

Fancy package specific module

Some people don't like the look of './lib/..' in require(). But they still want their package specific modules to be under the control of the package repo. In which case one solution is to symlink the modules to node_modules.

In one of my projects I have this in package.json:

"postinstall": "bash -c 'cd node_modules;ln -sf ../lib/*.js .'"

What it does is creates a symlink of all .js files inside ./lib to the node_modules folder when you run npm install.

Independent modules shard by several projects

If you have several modules shared by several of your projects you can (ab)use how node.js searches for modules. Node.js will search for the folder node_modules in the current directory and search for the module in there and if it can't find it it will recurse up the parent directory for a node_module folder etc. all the way to the root directory.

So, one way to have your own self-managed modules is to put a node_modules folder in the parent directory:

/home/myself/
    code/
        node_modules/
            my_module1.js
            my_module2/
                node_modules
                .git
                index.js
                package.json
        project1/
            node_modules/
            .git
            main.js
            package.json
        project2/
            node_modules/
            .git
            main.js
            package.json

In this way both project1 and project2 have access to my_module1 and my_module2. In addition my_module2 can have its own git repo.