Merge pull request #35129 from dimagi/bmb/webpack-docs

Webpack: Add Documentation Updates to the JavaScript Guide (RTD)

DEV_FAQ.md

@@ -38,6 +38,11 @@ Formplayer isn't running properly. See [Formplayer](https://github.com/dimagi/co
 
 It happens, try restarting.
 
+> I'm getting TemplateSyntaxErrors related to Webpack.
+
+Please run `yarn dev`. If this build is failing, and you didn't make any changes to JavaScript files on your branch,
+please raise this as an issue.
+
 ## Generating Sample Data
 
 ### SMS

DEV_SETUP.md

@@ -762,7 +762,23 @@ This can also be used to promote a user created by signing up to a superuser.
 Note that promoting a user to superuser status using this command will also give them the
 ability to assign other users as superuser in the in-app Superuser Management page.
 
-### Step 11: Running CommCare HQ
+### Step 11: Running `yarn dev`
+
+In order to build JavaScript bundles with Webpack, you will need to have `yarn dev`
+running in the background. It will watch any existing Webpack Entry Point, aka modules
+included on a page using the `webpack_main` template tag.
+
+When you add a new entry point (`webpack_main` tag), please remember to restart `yarn dev` so
+that it can identify the new entry point it needs to watch.
+
+To build Webpack bundles like it's done in production environments, pleas use `yarn build`.
+This command does not have a watch functionality, so it needs to be re-run every time you make
+changes to javascript files bundled by Webpack.
+
+For more information about JavaScript and Static Files, please see the
+[Dimagi JavaScript Guide](https://commcare-hq.readthedocs.io/js-guide/README.html) on Read the Docs.
+
+### Step 12: Running CommCare HQ
 
 Make sure the required services are running (PostgreSQL, Redis, CouchDB, Kafka,
 Elasticsearch).
@@ -794,6 +810,9 @@ yarn install --frozen-lockfile
 ./manage.py fix_less_imports_collectstatic
 ```
 
+See the [Dimagi JavaScript Guide](https://commcare-hq.readthedocs.io/js-guide/README.html) for additional
+useful background and tips.
+
 ## Running Formplayer and submitting data with Web Apps
 
 Formplayer is a Java service that allows us to use applications on the web  instead of on a mobile device.

docs/js-guide/README.rst

@@ -15,24 +15,31 @@ Table of contents
     :maxdepth: 1
 
     code-organization
+    static-files
 
 .. toctree::
     :caption: Dependencies
     :maxdepth: 1
 
     dependencies
     module-history
-    migrating
     libraries
     external-packages
 
+.. toctree::
+    :caption: Migrations
+    :maxdepth: 1
+
+    migrating
+    requirejs-to-webpack
+    amd-to-esm
+
 .. toctree::
     :caption: Best practices
     :maxdepth: 1
 
     integration-patterns
     security
-    static-files
     inheritance
     code-review
 

docs/js-guide/amd-to-esm.rst

@@ -0,0 +1,194 @@
+Updating Module Syntax from AMD to ESM
+======================================
+
+Most entry points for legacy modules that have recently been migrated from RequireJS to
+Webpack as part of the `RequireJS to Webpack Migration
+<https://github.com/dimagi/commcare-hq/blob/master/docs/js-guide/requirejs-to-webpack.rst>`__
+are eligible for this update.
+
+See the `Historical Background on Module Patterns
+<https://github.com/dimagi/commcare-hq/blob/master/docs/js-guide/module-history.rst>`__
+for a more detailed discussion of module types. As a quick refresher, here are some definitions:
+
+Modified AMD (Asynchronous Module Definition)
+    The legacy module type used for older JavaScript modules on HQ, identified by having an ``hqDefine``
+    statement near the top of the file. AMD was the only module type compatible with RequireJS,
+    our first JavaScript bundler. It is still needed as a format for modules required by No-Bundler pages.
+
+ESM (ES Modules)
+    The newest module type with updated powerful import and export syntax. This is the module
+    format that you will see referenced by documentation in modern javascript frameworks.
+    This is quickly identified by the ``import`` statements at the top used for including dependencies.
+
+The different types of modules you will encounter are:
+
+Entry Point Modules
+    Modules that are included directly on a page using a bundler template tag, like
+    ``webpack_main``. These are the modules that the bundler (Webpack) uses to build
+    a dependency graph so that it knows what bundle of javascript dependencies and
+    page-specific code is needed to render that page / entry point.
+
+Dependency Modules
+    These are modules that are never referenced by ``webpack_main`` and are only
+    in the list of dependencies for other modules. Often these modules are used as utility modules
+    or a way to organize JavaScript for a page that is very front-end heavy.
+
+
+Step 1: Determine if the Module is Eligible for a Syntax Update
+---------------------------------------------------------------
+
+The HQ AMD-style module will look something like:
+
+::
+
+    hqDefine('hqwebapp/js/my_module', [
+        'jquery',
+        'knockout',
+        'underscore',
+        'hqwebapp/js/initial_page_data',
+        'hqwebapp/js/assert_properties',
+        'hqwebapp/js/bootstrap5/knockout_bindings.ko',
+        'commcarehq',
+    ], function (
+        $,
+        ko,
+        _,
+        initialPageData,
+        assertProperties
+    ) {
+        ...
+    });
+
+
+Entry Points
+~~~~~~~~~~~~
+
+If this module is a webpack entry point, then it is eligible for an update. In the example above, you would find
+``hqwebapp/js/my_module`` used on a page with the following:
+
+::
+
+    {% webpack_main "hqwebapp/js/my_module %}
+
+The entry point can also be specified with ``webpack_main_b3`` if the module is part of the Bootstrap 3 build
+of Webpack.
+
+If this module is inside a ``requirejs_main`` or ``requirejs_main_b5`` tag, then it is NOT eligible for an update.
+Instead, please first
+`migrate this module from RequireJS to Webpack <https://github.com/dimagi/commcare-hq/blob/master/docs/js-guide/requirejs-to-webpack.rst>`__
+
+Dependency Modules
+~~~~~~~~~~~~~~~~~~
+
+If this module is a dependency of any modules that are ``requirejs_main`` entry points,
+then this module is not eligible for migration. If a module's syntax is updated when it's still
+required by RequireJS modules, then it will result in a RequireJS build failure on deploy.
+
+You can check the status of a dependency module's RequireJS usage by looking at the
+`Bootstrap 3 <https://www.commcarehq.org/static/build.b3.txt>`__ and
+`Bootstrap 5 <https://www.commcarehq.org/static/build.b5.txt>`__ module list.
+
+If this module is referenced by any ``hqImport`` calls (for instance ``hqImport('hqwebapp/js/my_module')``),
+then this module is NOT yet eligible, and must continue using the older AMD-style syntax until
+the ``hqImport`` statements are no longer needed. See the
+`JS Bundler Migration Guide <https://github.com/dimagi/commcare-hq/blob/master/docs/js-guide/migrating.rst>`__ for
+how to proceed in this case.
+
+Slightly Different Syntax
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the AMD-style module looks a bit different than the syntax above--for instance, the list of dependencies are missing or
+``hqImport`` and/or global variables can be found in the main body of the module--then this module must be
+`migrated to use a JS Bundler <https://github.com/dimagi/commcare-hq/blob/master/docs/js-guide/migrating.rst>`__.
+
+
+Step 2: Update the Module Syntax
+--------------------------------
+
+Key Points
+~~~~~~~~~~
+
+-   ESM no longer needs to define the module name within the module itself. Instead, Webpack (our bundler) is configured
+    to know how to reference this module by its filename and relative path within an application.
+-   By default, you can use the same dependency names with the ``import`` syntax. If the ``import`` statement results
+    in a Webpack Build error, look at ``webpack.common.js`` because it might require an alias. If you still have
+    a problem, check ``requirejs_config.js``, because there might have been an alias defined there that hasn't
+    been added to ``webpack.common.js``.
+
+
+Example Structural Change
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a rough example of what the changes will look like:
+
+::
+
+    hqDefine('hqwebapp/js/my_module', [
+        'jquery',
+        'knockout',
+        'underscore',
+        'hqwebapp/js/initial_page_data',
+        'hqwebapp/js/assert_properties',
+        'hqwebapp/js/bootstrap5/knockout_bindings.ko',
+        'commcarehq',
+    ], function (
+        $,
+        ko,
+        _,
+        initialPageData,
+        assertProperties
+    ) {
+        ...
+    });
+
+to
+
+::
+
+    import "commcarehq";  // Note: moved to top
+
+    // named yarn/npm dependencies
+    import $ from "jquery";
+    import ko from "knockout";
+    import _ from "underscore";
+
+    // named internal dependencies:
+    import initialPageData from "hqwebapp/js/initial_page_data";
+    import assertProperties from "hqwebapp/js/assert_properties";
+
+    // unnamed internal dependencies:
+    import "hqwebapp/js/bootstrap3/knockout_bindings.ko";
+
+    // module specific code...
+    ...
+
+Note that ``import "commcarehq";`` has been moved to the top of the file. The ordering is
+for consistency purposes, but it's important that either ``import "commcarehq";`` or
+``import "commcarehq_b3";`` (for Bootstrap 3 / ``webpack_main_b3``) is present in the list
+of imports for Webpack Entry Point modules. If this import is not present in an entry point,
+then site-wide navigation, notifications, modals, and other global widgets will not
+work on that page.
+
+Remember, an Entry Point is any module that is included directly on a page using the
+``webpack_main`` or ``webpack_main_b3`` template tags.
+
+Modules that are not entry points are not required to have this import. If you are updating the
+syntax of a dependency (non-entry point) module, do not worry about including this import if
+it is not already present.
+
+
+Step 4: Other Code Updates
+--------------------------
+
+If this module is an entry point, then the rest of the module-specific code can remain as is,
+with the indentation level updated. However, some entry points are also dependencies of other
+entry points. If that's the case, proceed to the next part.
+
+If this module is a dependency module, meaning it is referenced by other modules,
+then the ``return`` line at the end of the module should follow the appropriate ``export`` syntax
+needed by the modules that depend on this module.
+
+The most likely change is to replace ``return`` with ``export`` and leave everything else as is.
+Otherwise, see the
+`export documentation <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export>`__
+for details and inspiration in case you want to do some additional refactoring.

docs/js-guide/code-organization.rst

@@ -2,7 +2,8 @@ Static Files Organization
 -------------------------
 
 All\* JavaScript code should be in a .js file and encapsulated as a
-module using ``hqDefine``.
+module either using the ES Module syntax or modified-AMD syntax in
+legacy code using using ``hqDefine``.
 
 JavaScript files belong in the ``static`` directory of a Django app,
 which we structure as follows:
@@ -15,15 +16,17 @@ which we structure as follows:
        font/
        images/
        js/       <= JavaScript
-       less/
+       scss/
        lib/      <= Third-party code: This should be rare, since most third-party code should be coming from yarn
        spec/     <= JavaScript tests
        ...       <= May contain other directories for data files, i.e., `json/`
      templates/myapp/
        mytemplate.html
 
-\* There are a few places we do intentionally use script blocks, such as
-configuring less.js in CommCare HQ’s main template,
-``hqwebapp/base.html``. These are places where there are just a few
-lines of code that are truly independent of the rest of the site’s
-JavaScript. They are rare.
+To develop with javascript locally, make sure you run ``yarn dev`` and
+restart ``yarn dev`` whenever you add a new Webpack Entry Point.
+
+\* There are a few places we do intentionally use script blocks.
+These are places where there are just a few lines of code that are
+truly independent of the rest of the site's JavaScript. They are rare.
+For instance, third-party analytics script blocks are an example.