How to avoid having webpack to load unnecessary package files?

Question

cldr-data is an npm module that holds Unicode CLDR data, which is huge (~250 MB, lots of JSON files). Developers can cherry-pick the JSON files directly, e.g., require("cldr-data/supplemental/likelySubtags") or they can use its API, e.g., require("cldr-data")("supplemental/likelySubtags") (which will return require("./" + path) relatively to the module obviously).

When cherry-picking the JSON files directly, webpack resolves the straightforward requirements and it works just fine.

// "cldr-data/supplemental/likelySubtags.json" is a JSON file.
var likelySubtags = require("cldr-data/supplemental/likelySubtags");

quick build:

$ npm run build

> globalize-hello-world-webpack@ build /tmp/globalize-hello-world-webpack
> webpack

Hash: 4cb435ab9977c41bb6e0
Version: webpack 1.9.12
Time: 420ms
    Asset    Size  Chunks             Chunk Names
bundle.js  323 kB       0  [emitted]  main
   [0] ./main.js 1.26 kB {0} [built]
    + 18 hidden modules

When using the API, webpack gets crazy with "request of a dependency is an expression".

// "cldr-data/supplemental/likelySubtags.json" is a JSON file.
var cldrData = require("cldr-data");
var likelySubtags = cldrData("supplemental/likelySubtags");

looooooong intensive-CPU broken build:

npm run build
Hash: 74a65b11c84ec356ac8d
Version: webpack 1.9.12
Time: 496679ms
    Asset    Size  Chunks             Chunk Names
bundle.js  173 MB       0  [emitted]  main
   [0] ./main.js 1.54 kB {0} [built]
   [1] . ^\.\/.*$ 363 bytes {0} [built] [3 warnings]
   [2] ./README.md 0 bytes [optional] [built] [failed]
   [4] ./output/bundle.js 323 kB {0} [optional] [built]
   [6] ./webpack.config.js 359 bytes {0} [optional] [built]
    + 12155 hidden modules

WARNING in ./main.js
Critical dependencies:
10:1-39 the request of a dependency is an expression
7:1-44 the request of a dependency is an expression
8:1-42 the request of a dependency is an expression
9:1-42 the request of a dependency is an expression
11:1-49 the request of a dependency is an expression
12:1-50 the request of a dependency is an expression
13:1-44 the request of a dependency is an expression
14:1-45 the request of a dependency is an expression
15:1-45 the request of a dependency is an expression
 @ ./main.js 10:1-39 7:1-44 8:1-42 9:1-42 11:1-49 12:1-50 13:1-44 14:1-45 15:1-45

WARNING in ./~/cldr-data/DCO.md
Module parse failed: /tmp/globalize-hello-world-webpack/node_modules/cldr-data/DCO.md Line 1: Unexpected identifier
You may need an appropriate loader to handle this file type.
| If you would like to make a contribution to cldr-data-npm, please certify to the following:
| 
| ---
 @ ./~/cldr-data ^\.\/.*$

WARNING in ./~/cldr-data/LICENSE
Module parse failed: /tmp/globalize-hello-world-webpack/node_modules/cldr-data/LICENSE Line 1: Unexpected identifier
You may need an appropriate loader to handle this file type.
| UNICODE, INC. LICENSE AGREEMENT - DATA FILES AND SOFTWARE
| 
|     Unicode Data Files include all data files under the directories
 @ ./~/cldr-data ^\.\/.*$

WARNING in ./~/cldr-data/LICENSE-MIT
Module parse failed: /tmp/globalize-hello-world-webpack/node_modules/cldr-data/LICENSE-MIT Line 1: Unexpected number
You may need an appropriate loader to handle this file type.
| Copyright (c) 2013 Rafael Xavier de Souza http://rafael.xavier.blog.br
| 
| Permission is hereby granted, free of charge, to any person
 @ ./~/cldr-data ^\.\/.*$

WARNING in ./~/cldr-data/README.md
Module parse failed: /tmp/globalize-hello-world-webpack/node_modules/cldr-data/README.md Line 1: Unexpected token ILLEGAL
You may need an appropriate loader to handle this file type.
| # cldr-units-modern
| 
| This repository provides the a portion of the JSON distribution of CLDR locale data
 @ ./~/cldr-data ^\.\/.*$

WARNING in ./~/cldr-data/install.js
Module parse failed: /tmp/globalize-hello-world-webpack/node_modules/cldr-data/install.js Line 50: Illegal return statement
You may need an appropriate loader to handle this file type.
|     "listed under `dependencies` of the `package.json` of your application."
|   );
|   return process.exit(0);
| }
| 
 @ ./~/cldr-data ^\.\/.*$

WARNING in . ^\.\/.*$
Module not found: Error: a dependency to an entry point is not allowed
 @ . ^\.\/.*$

WARNING in . ^\.\/.*$
Module not found: Error: a dependency to an entry point is not allowed
 @ . ^\.\/.*$

WARNING in ./README.md
Module parse failed: /tmp/globalize-hello-world-webpack/README.md Line 1: Unexpected token ILLEGAL
You may need an appropriate loader to handle this file type.
| # Hello World (Webpack)
| 
| Project loading `globalize` through `webpack`:
 @ . ^\.\/.*$

ERROR in ./~/cldr-data/Gruntfile.js
Module not found: Error: Cannot resolve module 'matchdep' in /tmp/globalize-hello-world-webpack/node_modules/cldr-data
 @ ./~/cldr-data/Gruntfile.js 40:2-23

cldr-data exported API uses dynamic requierments and it's making webpack crazy. I suppose require contexts could help me out in this scenario given #196. But, I have no idea how.

Find the question files here: https://github.com/unindented/globalize-hello-world-webpack/.

Any help is very much appreciated.

Thanks

Answer 1

The short answer is use static strings like require("cldr-data/<path>").

If you really would like to use the API, use ContextReplacementPlugin plugin according to How to prevent moment.js from loading locales with webpack?. Although, the benefits of the API (e.g., .entireMainFor("en")) are lost, because you would have to duplicate effort changing the plugin to reflect what you need. For example:

var webpack = require("webpack");

module.exports = 
...
  plugins: [
    new webpack.ContextReplacementPlugin(/cldr-data$/, /supplemental/)
  ]
};

Thanks for all the help @bebraw and @sokra.

Answer 2

If you want to go through require.context, it would look like this on consumer side:

var req = require.context('cldr-data/supplemental');
var likelySubtags = req('./likelySubtags.json');

That doesn't look very nice to me. I tried against require.context('cldr-data') but then we're again in the world of slow builds. I think in that case it will have to traverse whole package so it can build the lookup.

var likelySubtags = require('cldr-data/supplemental/likelySubtags'); seems like the sanest option to me at the moment.

Perhaps there could be a Webpack adapter in between that would hide that require.context thing as above? You could easily end up with an API like require('cldr-data').supplemental('likelySubtags'). Thatreqpart can be dynamic so this should work. You would just write the missing./and.json` there. Of course you are free to change the API to your liking. This was just the first idea that I came up with.