pnpm provides different hashes for the exact same version of the same dependency, breaking nestjs

Question

I have a monorepo with a very basic setup available for reproducing this issue here:

It is a single nestjs app with 2 packages that it reads from.

@nestjs/core among other dependencies is needed for both the packages and the main app to work, and it is enforced to be the exact same fixed version not only on their own local package.json's but also with the resolutions {} config in the main package.json.

I can inspect the lockfile and find out that although the same version is used -- the hashes are different, causing major issues with nestjs, not being able to import injectable dependencies reliably causing it to break on bootstrap.

Is there a way to prevent this? to force linking the exact same hash/dependency?

Answer 1

Edit:

With pnpm v7.29.0, you no longer have to perform the hack described below, but just leaving it here for educational purposes.

Now the solution is just to set dedupe-peer-dependents=true (e.g. in your .npmrc).

---

From pnpm docs

- foo-parent-1
  - bar@1.0.0
  - baz@1.0.0
  - foo@1.0.0
- foo-parent-2
  - bar@1.0.0
  - baz@1.1.0
  - foo@1.0.0

In the example above, foo@1.0.0 is installed for foo-parent-1 and foo-parent-2. Both packages have bar and baz as well, but they depend on different versions of baz. As a result, foo@1.0.0 has two different sets of dependencies: one with baz@1.0.0 and the other one with baz@1.1.0. To support these use cases, pnpm has to hard link foo@1.0.0 as many times as there are different dependency sets.

For your specific case, foo === @nestjs/core, baz === @nestjs/microservices. Although the example used here is for "different versions", the same applies for optional peer dependencies. So to re-illustrate the example, in your context:

- my-nestjs-app
  - @nestjs/microservices@9.1.4
  - @nestjs/core@9.1.4
- my-other-nestjs-app
  - @nestjs/core@9.1.4

Normally, if a package does not have peer dependencies, it is hard linked to a node_modules folder next to symlinks of its dependencies, like so:

However, if foo [@nestjs/core] has peer dependencies, there may be multiple sets of dependencies for it, so we create different sets for different peer dependency resolutions

^ This is usually ok for most packages out there. However @nestjs/core is special. It's stateful so that it can take care of all the runtime dependency injections. pnpm creating multiple copies of @nestjs/core in a monorepo will result in the confusing behaviour you're seeing, as your app could depend on 1 copy, while other NestJS libs depend on another. This seems like a common problem felt by devs using pnpm + nest, according to the NestJS discord.

Solution

Use pnpm hooks to modify nestjs packages' peerDependenciesMeta at resolution time:

// .pnpmfile.cjs in your monorepo's root

function readPackage(pkg, context) {
  if (pkg.name && pkg.name.startsWith('@nestjs/')) {
    context.log(`${pkg.name}: make all peer dependencies required`);
    pkg.peerDependenciesMeta = {}; 
  }
  return pkg;
}

module.exports = {
  hooks: {
    readPackage,
  }
};

This is a hack IMO, and it's really annoying to deal with because Renovate / Dependabot will ignore the .pnpmfile.cjs when it performs dependency updates. I'd suggest going with Nx or some other package manager that Nest / stateful packages work better with.

Answer 2

1. Set use-lockfile-v6=true in your .npmrc

2. Then pnpm install to get the new lockfile

3. Then analyze the top of the pnpm-lock.yaml to see which packages (yours) of your monorepo appear with different versions.

It's a bit a manual work but starts from top to bottom and look into the importers: block. It's all the direct dependencies you listed into your package.json files.

Stop at each dependency that looks with a suffix, like version: 0.31.1(react@18.2.0) (note that something with no risk of duplication would be version: 0.31.1).

In my case for example I found for my packageA:

      '@mui/material':
        specifier: ^5.10.16
        version: 5.10.16(@emotion/react@11.10.5)(@emotion/styled@11.10.5)(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0)

Then copy into the clipboard '@mui/material': and search for all occurrences into the importer: section. If you find a different pattern of version it means it's deduplicated, for example I had for my packageB:

      '@mui/material':
        specifier: ^5.10.16
        version: 5.10.16(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0)

In my case my packageA has @emotion/* dependencies that are not specified into packageB, making them a mismatch since @mui/material lists them as peerDependencies. Since I was no longer needing @emotion/* dependencies I just deleted them. Then pnpm install and there is only now version: 5.10.16(@types/react@18.0.26)(react-dom@18.2.0)(react@18.2.0). Meaning it will uses the exact same package for both packageA and packageB.

In case I would have needed @emotion/* in packageA I would have been able to add them into packageB, it would make the same result of merging the dependency. The idea is just to fix the versions of peerDependencies of my different @mui/material so they can match and be merged. It's explained in https://pnpm.io/how-peers-are-resolved .

I think it can also help to make sure YOUR direct dependencies use same version everywhere in the monorepo (it's a first step before aligning peerDependencies. For this I use into my root package.json:

  "pnpm": {
    "overrides": {
      "@mui/material": "^5.10.16",
      "react": "^18.2.0",
      "react-dom": "^18.2.0",
    }
  },

(repeat the process for each dependency that you don't want to see "duplicated"...)

(note that it's from my own experience, it's maybe not perfect, but it helped me at least ^^...)

Answer 3

When a dependency has peer dependencies, it might be written to node_modules several times if the peer dependencies are resolved differently in various parts of the dependency graph.

In your case, @nestjs/core is in the dependencies of the graphql-server project and the @myapp/entities project. @nestjs/core has @nestjs/platform-express as an optional peer dependency.

@nestjs/platform-express is in the dependencies of the graphql-server project, so pnpm links it to @nestjs/platform-express. You can see it in the lockfile:

  /@nestjs/core/8.4.7_fkqgj3xrohk2pflugljc4sz7ea:
    resolution: {integrity: sha512-XB9uexHqzr2xkPo6QSiQWJJttyYYLmvQ5My64cFvWFi7Wk2NIus0/xUNInwX3kmFWB6pF1ab5Y2ZBvWdPwGBhw==}
    requiresBuild: true
    peerDependencies:
      '@nestjs/common': ^8.0.0
      '@nestjs/microservices': ^8.0.0
      '@nestjs/platform-express': ^8.0.0
      '@nestjs/websockets': ^8.0.0
      reflect-metadata: ^0.1.12
      rxjs: ^7.1.0
    peerDependenciesMeta:
      '@nestjs/microservices':
        optional: true
      '@nestjs/platform-express':
        optional: true
      '@nestjs/websockets':
        optional: true
    dependencies:
      '@nestjs/common': 8.4.7_47vcjb2de6lyibr6g4enoa5lyu
      '@nestjs/platform-express': 8.4.7_7tsmhnugyerf5okgqzer2mfqme # <------HERE
      '@nuxtjs/opencollective': 0.3.2
      fast-safe-stringify: 2.1.1
      iterare: 1.2.1
      object-hash: 3.0.0
      path-to-regexp: 3.2.0
      reflect-metadata: 0.1.13
      rxjs: 7.5.5
      tslib: 2.4.0
      uuid: 8.3.2
    transitivePeerDependencies:
      - encoding

In the other project (@myapp/entities), @nestjs/platform-express is not in the dependencies, so when installing @nestjs/core, pnpm cannot resolve the optional peer dependency. As a result, pnpm needs to create another instance of @nestjs/core, which doesn't have this optional peer linked in. As you can see in the lockfile, the other entry doesn't have @nestjs/platform-express:

  /@nestjs/core/8.4.7_g7av3gvncewo44y4rurz3mgav4:
    resolution: {integrity: sha512-XB9uexHqzr2xkPo6QSiQWJJttyYYLmvQ5My64cFvWFi7Wk2NIus0/xUNInwX3kmFWB6pF1ab5Y2ZBvWdPwGBhw==}
    requiresBuild: true
    peerDependencies:
      '@nestjs/common': ^8.0.0
      '@nestjs/microservices': ^8.0.0
      '@nestjs/platform-express': ^8.0.0
      '@nestjs/websockets': ^8.0.0
      reflect-metadata: ^0.1.12
      rxjs: ^7.1.0
    peerDependenciesMeta:
      '@nestjs/microservices':
        optional: true
      '@nestjs/platform-express':
        optional: true
      '@nestjs/websockets':
        optional: true
    dependencies:
      '@nestjs/common': 8.4.7_47vcjb2de6lyibr6g4enoa5lyu
      '@nuxtjs/opencollective': 0.3.2
      fast-safe-stringify: 2.1.1
      iterare: 1.2.1
      object-hash: 3.0.0
      path-to-regexp: 3.2.0
      reflect-metadata: 0.1.13
      rxjs: 7.5.5
      tslib: 2.4.0
      uuid: 8.3.2
    transitivePeerDependencies:
      - encoding

To solve this, you can add @nestjs/platform-express to the dependencies of the @myapp/entities project. It should be the same version as in the other project.