Add tests (#16)

* Improve settings, remove warnings * Ignore env * Add tests, improve translation of blocks, add translate related objects * Add patch for **model** copy_for_translation_done * Use .env file to store secrets, move WT settings to the bottom * Assume https on URL fields * Serve media in development * Add explanation docs * Add simple blog_post_page template * Add link from readme to explanation
allcaps · Aug 22, 2024 · aa94ca0 · aa94ca0
1 parent 1e24e4e
commit aa94ca0
Show file tree

Hide file tree

Showing 19 changed files with 598 additions and 136 deletions.
diff --git a/.env-example b/.env-example
@@ -0,0 +1,6 @@
+# https://pypi.org/project/python-dotenv/
+
+# Rename this file to .env and fill in the values.
+# The .env is not onder version control. Keeping your secrets safe.
+
+WAGTAIL_TRANSLATE_DEEPL_KEY=
diff --git a/.gitignore b/.gitignore
@@ -8,10 +8,12 @@ __pycache__/
 /.tox
 /.venv
 /venv
+/env
+/.env
 /.vscode
 /site
 /test_wagtail_translate.db
 /node_modules
 /test-static
 /test-media
-.python-version
+/.python-version
diff --git a/README.md b/README.md
@@ -1,10 +1,13 @@
 # Wagtail Translate
 
-**Wagtail Translate** adds machine translations to your Wagtail site, with built-in support for [DeepL](https://www.deepl.com) and the flexibility to integrate other translation services. It automatically detects when a page is copied to a new locale and initiates the translation process.
+**Wagtail Translate** adds machine translations to your Wagtail site, with built-in support for [DeepL](https://www.deepl.com) and the flexibility to integrate other translation services.
+It automatically detects when a page is copied to a new locale and initiates the translation process.
 
-For multilingual websites, the recommended packages are [Wagtail Localize](https://wagtail-localize.org/) or [Wagtail Simple Translation](https://docs.wagtail.org/en/stable/reference/contrib/simple_translation.html). Wagtail Localize offers advanced features that may be excessive for many projects, while Wagtail Simple Translation only copies pages to new locales, requiring manual translation.
+[Wagtail Localize](https://wagtail-localize.org/) and [Wagtail Simple Translation](https://docs.wagtail.org/en/stable/reference/contrib/simple_translation.html) are the go-to solutions for multi-language Wagtail projects.
+Wagtail Localize offers advanced features that may be excessive for many projects, while Wagtail Simple Translation only copies pages to new locales, requiring manual translation.
 
-**Wagtail Translate** adds machine translations to Wagtail and works in combination with Simple Translation, offering the ideal solution for projects seeking a simple interface with powerful translation support.
+**Wagtail Translate** adds machine translations to Wagtail.
+It works in combination with Simple Translation, offering the ideal solution for projects seeking a simple interface with powerful translation support.
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 [![PyPI version](https://badge.fury.io/py/wagtail-translate.svg)](https://badge.fury.io/py/wagtail-translate)
@@ -100,9 +103,11 @@ Install and configure it as follows:
 - Add `WAGTAIL_TRANSLATE_DEEPL_KEY = "..."` to your settings.
 - Set `WAGTAIL_TRANSLATE_TRANSLATOR = "wagtail_translate.translators.deepl.DeepLTranslator"`
 
-## Customise Wagtail Translate
+## Documentation
 
-See: [docs/customise_wagtail_translate.md](https://github.com/allcaps/wagtail-translate/tree/main/docs/customise_wagtail_translate.md).
+- This readme for installation and basic usage.
+- [How to customise Wagtail Translate](https://github.com/allcaps/wagtail-translate/tree/main/docs/customise_wagtail_translate.md)
+- [Explanation](https://github.com/allcaps/wagtail-translate/tree/main/docs/explanation.md)
 
 ## Contributing
 

diff --git a/docs/customise_wagtail_translate.md b/docs/customise_wagtail_translate.md
@@ -1,4 +1,4 @@
-## Customise Wagtail Translate
+## How to customise Wagtail Translate
 
 Multi-language projects often have specific requirements for the translation process.
 This page provides examples of how to customize Wagtail Translate to fit your project's needs.

diff --git a/docs/explanation.md b/docs/explanation.md
@@ -0,0 +1,81 @@
+# Explanation
+
+This document details the concepts underlying Wagtail Translate and explains the reasoning behind its design choices. It aims to enhance and expand your understanding of Wagtail Translation.
+
+## Idea and positioning
+
+[Wagtail Localize](https://wagtail-localize.org/) and [Wagtail Simple Translation](https://docs.wagtail.org/en/stable/reference/contrib/simple_translation.html) are the go-to solutions for multilingual Wagtail projects.
+Wagtail Localize offers advanced features that may be excessive for many projects, while Wagtail Simple Translation only copies pages to new locales, requiring manual translation.
+
+**Wagtail Translate** adds machine translations to Wagtail. It works with Simple Translation, providing an ideal solution for projects seeking a straightforward interface with robust translation support.
+
+Your content editors will appreciate the simplicity of Wagtail Translate. 🥳
+
+## Features
+
+- Automatic translation
+- Integration with DeepL
+- Highly customizable
+
+## Architecture
+
+Wagtail Translate operates at the model level and does not include a user interface. While using Wagtail Translate in combination with Wagtail Simple Translation is the obvious choice, it is not a strict requirement.
+
+Wagtail Translate listens for Wagtail's `copy_for_translation_done` signal to initiate the machine translation.
+Wagtail Translate is indifferent on who sends the signal. This could be Simple Translation, other products, or custom code.
+
+## Customizing default behaviors
+
+Wagtail Translate is designed to be flexible, allowing you to tailor its behavior to your needs.
+
+- Provide your own translation service
+- Define your own signal handler
+- Customize the translation process
+
+## Background workers
+
+Wagtail Translate uses external services like DeepL to provide translations. These services can take time, fail to respond, or be offline.
+
+By default, Wagtail Translate handles translations synchronously, meaning they block the current thread, and the user must wait for all translations to complete. This can negatively impact performance and user experience.
+
+Django's [background workers](https://www.djangoproject.com/weblog/2024/may/29/django-enhancement-proposal-14-background-workers/) are an accepted proposal and are in development. Once available, they will offer a unified way to offload tasks. Wagtail Translate is expected to support them. For now, if you need translations offloaded to a background task, you can customize the default behaviors and implement this yourself.
+
+## Fields to translate
+
+Wagtail Translate introspects the model definition to identify the fields that need translation. You can control which fields are translated by implementing a `get_translatable_fields` method on your model.
+
+## HTML
+
+Wagtail has rich text and raw fields, which may contain HTML. Translating HTML is challenging because the structure (tags and attributes) must remain unchanged, while the text between tags should be translated.
+
+Wagtail Translate deconstructs HTML and translates the text parts one by one, then reconstructs the HTML with the translated text. However, this process can lead to a lack of context for the translation service. For example:
+
+```html
+<p>The <em>black</em> cat</p>
+```
+Translating word-for-word into French yields:
+```html
+<p>Le <em>noir</em> chat</p>
+```
+But it should be translated as:
+```html
+<p>Le chat <em>noir</em></p>
+```
+
+Word order can vary across languages, and words may have different meanings depending on the context. For instance, the word "bow" can refer to a weapon for shooting arrows or a gesture of respect. In French, these are translated as "arc" and "révérence," respectively. If translation service has too little context, there is a risk of incorrect translation.
+
+Wagtail Translate preserves the HTML structure to prevent broken HTML, as translating tags and attributes could lead to invalid markup.
+
+In practice, most HTML translates well, but if you encounter odd translations, this may be the cause. A workaround might be removing the styling from the text before translation and re-apply the styling after.
+
+You can alter the HTML translation behavior by providing a custom translation class and overriding the `BaseTranslator.translate_html` method.
+
+## Translation of related objects
+
+For related objects (foreign keys) Wagtail Translate checks if the object is translatable, and if the translations exist. If the translation exists, it is used; otherwise, the original object is used.
+
+This approach prevents deep traversal as related objects can have related objects of their own. It also accommodates intentional cross-language references. For example, a multilingual site might have blogs exclusively in English, requiring links to the English posts regardless of the referencing page's locale.
+
+Since Wagtail Translate can't forsee the intended behaviour, it uses the simplest approach: use if the translation exists, otherwise use the original object. This  means that sometimes the content editor needs to step in and translate the related object, and select that related object.
+
+To adjust this behaviour, override `BaseTranslator.translate_related_object`.
diff --git a/pyproject.toml b/pyproject.toml
@@ -33,9 +33,11 @@ dependencies = [
     "Django>=4.2",
     "Wagtail>=5.2"
 ]
+
 [project.optional-dependencies]
 testing = [
     "dj-database-url==2.1.0",
+    "python-dotenv",
     "pre-commit==3.4.0",
     "pytest==8.1.1",
     "pytest-cov==5.0.0",

diff --git a/src/wagtail_translate/apps.py b/src/wagtail_translate/apps.py
@@ -3,19 +3,29 @@
 from django.apps import AppConfig
 
 
-def patch_needed(version=wagtail.VERSION) -> bool:
+def page_patch_needed(version=wagtail.VERSION) -> bool:
     """
-    Wagtail 6.2 introduces the `copy_for_translation_done` signal.
+    Wagtail 6.2 introduces the `copy_for_translation_done` signal for **pages**.
     Older Wagtail versions need to be patched.
     """
     return version[0] < 6 or (version[0] == 6 and version[1] < 2)
 
 
+def model_patch_needed(version=wagtail.VERSION) -> bool:
+    """
+    Wagtail 6.3 introduces the `copy_for_translation_done` signal for **models**.
+    Older Wagtail versions need to be patched.
+    """
+    return version[0] < 6 or (version[0] == 6 and version[1] < 3)
+
+
 class WagtailTranslateAppConfig(AppConfig):
     label = "wagtail_translate"
     name = "wagtail_translate"
     verbose_name = "Wagtail Translate"
 
     def ready(self):
-        if patch_needed():
-            from . import monkeypatches  # noqa
+        if page_patch_needed():
+            from . import monkeypatch_page  # noqa
+        if model_patch_needed():
+            from . import monkeypatch_model  # noqa
diff --git a/src/wagtail_translate/fields.py b/src/wagtail_translate/fields.py
@@ -7,14 +7,13 @@
 
 def get_translatable_fields(model):
     """
-    Derives a list of translatable fields from the given model class.
+    Derives a list of translatable fields (strings) from the given model class.
 
     Arguments:
         model (Model class): The model class to derive translatable fields from.
 
     Returns:
-        list[TranslatableField or SynchronizedField]:
-            A list of TranslatableField and SynchronizedFields that were derived from the model.
+        list: A list of stings, containing the names of translatable fields.
     """
 
     # Set translatable_fields on the model to override the default behaviour.
@@ -63,7 +62,7 @@ def get_translatable_fields(model):
             continue
 
         # Ignore choice fields
-        # These are usually used for enums and should not be translated.
+        # These are usually enums and should not be translated.
         if isinstance(field, models.CharField) and field.choices:
             continue
 
@@ -94,65 +93,11 @@ def get_translatable_fields(model):
                 continue
 
             # Foreign keys to translatable models should be translated.
-            # With the exception of pages that are special because we can localize them at runtime easily.
-            # TODO: Perhaps we need a special type for pages where it links to the translation if availabe,
-            # but falls back to the source if it isn't translated yet?
-            # Note: This exact same decision was made for page chooser blocks in segments/extract.py
             if issubclass(field.related_model, TranslatableMixin) and not issubclass(
                 field.related_model, Page
             ):
-                continue
-                # TODO, implement related Translatable support
-                # translatable_fields.append(field)
+                translatable_fields.append(field)
             else:
                 continue
 
-        # TODO
-        # # Fields that support extracting segments are translatable
-        # elif hasattr(field, "get_translatable_segments"):
-        #     translatable_fields.append(TranslatableField(field))
-        #
-        # else:
-        #     # Everything else is synchronised
-        #     translatable_fields.append(SynchronizedField(field))
-
-    # Add child relations for clusterable models
-    # if issubclass(model, ClusterableModel):
-    #     for child_relation in get_all_child_relations(model):
-    #         # Ignore comments
-    #         if (
-    #             issubclass(model, Page)
-    #             and child_relation.name == COMMENTS_RELATION_NAME
-    #         ):
-    #             continue
-    #
-    #         if issubclass(child_relation.related_model, TranslatableMixin):
-    #             translatable_fields.append(child_relation.name)
-    #         else:
-    #             continue
-
-    # Combine with any overrides defined on the model
-    # override_translatable_fields = getattr(model, "override_translatable_fields", [])
-    #
-    # if override_translatable_fields:
-    #     override_translatable_fields = {
-    #         field.field_name: field for field in override_translatable_fields
-    #     }
-    #
-    #     combined_translatable_fields = []
-    #     for field in translatable_fields:
-    #         if field.field_name in override_translatable_fields:
-    #             combined_translatable_fields.append(
-    #                 override_translatable_fields.pop(field.field_name)
-    #             )
-    #         else:
-    #             combined_translatable_fields.append(field)
-    #
-    #     if override_translatable_fields:
-    #         combined_translatable_fields.extend(override_translatable_fields.values())
-    #
-    #     return combined_translatable_fields
-    #
-    # else:
-
     return translatable_fields
diff --git a/src/wagtail_translate/monkeypatch_model.py b/src/wagtail_translate/monkeypatch_model.py
@@ -0,0 +1,44 @@
+"""
+Wagtail 6.3 introduces the copy_for_translation_done signal for models (snippets).
+Wagtail Translate will patch older versions of Wagtail.
+
+The `CopyForTranslationAction.execute` method is patched.
+The new method sends the `copy_for_translation_done` signal.
+"""
+
+import logging
+
+from wagtail.actions.copy_for_translation import CopyForTranslationAction
+
+
+logger = logging.getLogger(__name__)
+
+
+def new_execute(self, skip_permission_checks=False):
+    self.check(skip_permission_checks=skip_permission_checks)
+
+    translated_object = self._copy_for_translation(
+        self.object, self.locale, self.exclude_fields
+    )
+
+    # Depending on the Wagtail version,
+    # the signal may be defined in Wagtail or in Wagtail Translate.
+    try:
+        from wagtail.signals import copy_for_translation_done
+    except ImportError:
+        from wagtail_translate.signals import copy_for_translation_done
+
+    # Send signal
+    copy_for_translation_done.send(
+        sender=self.__class__,
+        source_obj=self.object,
+        target_obj=translated_object,
+    )
+
+    return translated_object
+
+
+logger.warning(
+    "Monkeypatching wagtail.actions.copy_for_translation.CopyForTranslationAction.execute, send copy_for_translation_done signal"
+)
+CopyForTranslationAction.execute = new_execute
diff --git a/src/wagtail_translate/monkeypatches.py → src/wagtail_translate/monkeypatch_page.py b/src/wagtail_translate/monkeypatches.py → src/wagtail_translate/monkeypatch_page.py
diff --git a/src/wagtail_translate/translators/base.py b/src/wagtail_translate/translators/base.py
@@ -1,6 +1,8 @@
 from bs4 import BeautifulSoup, NavigableString
+from django.db.models import ForeignKey
 from wagtail import blocks
 from wagtail.fields import RichTextField, StreamField
+from wagtail.models import TranslatableMixin
 from wagtail.rich_text import RichText
 
 from ..fields import get_translatable_fields
@@ -158,6 +160,24 @@ def translate_list_block(self, item):
             self.translate_block(block_item)
             item.value[idx] = block_item.value
 
+    def translate_related_object(self, item):
+        """Translate related object,
+
+        Select the translated object if it exists,
+        otherwise keep the current object.
+        """
+        if not isinstance(item, TranslatableMixin):
+            return item
+
+        if (
+            translation := item.get_translations()
+            .filter(locale__language_code=self.target_language_code)
+            .first()
+        ):
+            return translation
+
+        return item
+
     def translate_block(self, item) -> None:
         """
         Translate block,
@@ -180,9 +200,7 @@ def translate_block(self, item) -> None:
         elif isinstance(item.block, blocks.BlockQuoteBlock):
             item.value = self.translate(item.value)
         elif isinstance(item.block, blocks.ChooserBlock):
-            ...  # TODO, implement
-        elif isinstance(item.block, blocks.PageChooserBlock):
-            ...  # TODO, implement
+            item.value = self.translate_related_object(item.value)
 
         # And to recurse, we need to handle iterables.
         elif isinstance(item.block, blocks.StructBlock):
@@ -248,6 +266,8 @@ def translate_obj(self, source_obj, target_obj):
                 translation = self.translate_html(src)
             elif isinstance(field, StreamField):
                 translation = self.translate_blocks(src)
+            elif isinstance(field, ForeignKey):
+                translation = self.translate_related_object(src)
             else:
                 translation = self.translate(src)
             setattr(target_obj, field.name, translation)