Add incremental compiler to webpack dev server

What does this MR do?

Add incremental compiler to webpack dev server

In order to have quicker compilations while developing we are adding an incremental compiler to webpack to render routes on demand.

When the developer is working on the dev server and enables the incremental compiler with DEV_SERVER_INCREMENTAL=true, the new functionality is enabled.

The biggest problem to solve here: How can we know which entry point to render and which not? Our current webpack integration with rails requires the webpack manifest to have a list of all existing entry points. So our incremental compiler takes the following approach:

Every compilation of webpack will run generateEntries which generates a list of all our entry points. In that function we are able to replace all page specific entrypoints and point them to an empty file, unless we explicitly want them to compile.

In the webpack-dev-server itself we register a middleware which keeps track of all the page specific bundles requested. Whenever a page specific bundle is requested that hasn't been requested before, we add it to the list of bundles we want compiled.

This approach allows us to dynamically change the entry points without a need to restart webpack alltogether and it works with hot module reloading.

Rather than pointing to a blank javascript we are pointing to one which renders an overlay to let the user know that webpack compiles the page for the first time.

Additionally we keep a history of requested routes in tmp/cache in order to keep the list of compiled entry points between sessions. In a next iteration we can add a bit of logic and e.g. remove entry points the developer hasn't been visiting in a week.

First results are really promising (on my machine):

• Memory consumption when idling: 1600MB => 340MB
• Max memory: ~2200MB => ~1000MB
• Initial Compilation time: 58s => 15s
• Recompile after a file change: 13s => 3s
• Visiting a new page that hasn't been visited before, it takes about four seconds to reload the page, seven seconds to completely load the page.

Currently the technique still watches all of the source files, so changing an unrelated file will trigger a recompilation. This is however a minor caveat and the same behavior that we currently have, maybe we can optimize in the future.

• Relates to #214297
• Relates to #215608 (closed)

How to enable?

1. Make sure that gdk is up to date
2. In your gdk.yml simply set

   webpack:
     incremental: true

3. Run gdk reconfigure

Screenshot

Here you can see how this behaves when enabled:

incremental-compiler

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team