Improved documentation that allows a larger audience to contribute

[raner]

Is your feature request related to a problem? Please describe.
Clearly, Iosevka is a popular project that is enjoyed by many people for numerous reasons. However, other than @be5invis, @jmcwilliams403, and maybe a handful of other folks, the group of contributors is relatively small. Personally, when I open an issue for an open-source project and the issue has been deemed valid, I like to contribute a pull request to solve the issue myself (if I have the time). I would like to do this for Iosevka as well, but unfortunately I cannot really wrap my head around how things are done in this code base. When I look at the PRs for similar issues, I usually end up scratching my head, and I'm often unable to understand how the code in the PR actually solved the problem. I looked at the documentation for the PatEL language, which is fairly straightforward, but clearly there is a lot more to Iosevka than just PatEL.

Describe the solution you'd like
I would be delighted to see some more extensive documentation (beyond build set-up, etc.) that explains the philosophy behind the Iosevka code base and explains how to add new characters. I understand that the idea is to reuse geometric primitives so that changes can be applied consistently without the need to edit hundreds of glyphs. However, to the uninitiated it is difficult to understand what primitives should be reused and when possibly a new primitive should be introduced.

For example, it would be very useful if there were some documentation that explained how to achieve the following features:

• Changing the metrics of a glyph or creating a new glyph that only differs in its proportions from an existing glyph:
  • Stretching a glyph vertically so that that its top and bottom aligns with, e.g., the top and bottom horizontal features of brackets
  • Changing a glyph's width from monospace to duospace (e.g., creating a HEBREW LETTER WIDE ALEF U+FB21 from the regular alef U+05D0)
  • Scaling and rotating characters
• Combining two or more existing glyphs to form a new glyph (e.g., circled symbols, characters with diacritics)
• Creating new characters from scratch by reusing the correct primitives or existing similar characters

Before describing such specific tasks, it might be good to outline the general design philosophy of Iosevka (it is very clear that there is one, but it does not seem to be written down anywhere).

Describe alternatives you've considered
As a starting point and possible alternative, one could single out some existing past PRs with exceptionally good documentation, or start including some more explanatory notes in PRs that explain why certain problems were solved certain ways.

Additional context
Sadly, I've seen many great open-source projects that were sentenced to death once the main contributors moved on to other things (or no longer had the time to maintain the project). Some of these projects even had dozens of committers and still suffered this fate. With Iosevka you can count the committers on one hand, so my feeling is that the project could be at risk long term unless more developers are able to make meaningful contributions. This could also help with reducing the current issue backlog that clearly demonstrates that the interest in Iosevka is larger than what the current group of contributors can maintain.

[be5invis]

Right, I want to do some stabilization work in the next year or so. One key step at current stage could be migrating more stuff from PTL to MJS -- The Iosevka's PTL code relies heavily on macros (for example, [glyph-proc] is a macro), which is not simple to understand.

The problem is, Iosevka is already very complicated -- I guess it is one of the most complicated LGC font project... As a result writing docs for it is not a simple work especially considering I am not an English native speaker.

[Logo121]

Handful other folks here. I doubt anyone who contributes here (except be5invis himself) has a full idea of what's going on in the code (especially the more complex ones). You can generally do fine by looking at other code and copying/following them, then learning the mechanisms bit by bit.

But here's some basic ideas on how this font works (at least how I perceive it; correct me if I'm wrong):

• The glyphs are drawn using spiro curves, which works by specifying points of a curve with other constraints and having the program "solve" the curve for you, instead of specifying Beziers directly.
  • There is [spiro-outline] that creates a shape using a list of spiro knots and [dispiro], which is basically a spiro with thickness. The latter is how most letters are drawn.
• The style of Iosevka is somewhere between Geometric and Neo-grotesque with a slab-serif variant, though some character variants are being suggested from time to time that may deviate from this style a bit.
• Some useful functions (or macros) to create a glyph can be found here:

  Iosevka/packages/font-glyphs/src/meta/macros.ptl

  Lines 121 to 161 in c982560

  	### Necessary macros
  	# A glyph construction is a function which "modifies" a glyph.
  	define-macro glyph-proc : syntax-rules
  	`[glyph-proc @::steps] : dirty `[new $Capture$.GlyphProc : function [currentGlyph] [begin \\
  	# local currentGlyph this
  	begin @::[steps.map formOf]
  	return nothing
  	]]
  	define-macro composite-proc : syntax-rules
  	`[composite-proc @::steps] : dirty `[new $Capture$.GlyphProc : function [currentGlyph] [begin \\
  	# local currentGlyph this
  	begin @::[steps.map : lambda [x j] : if j `[include @[formOf x]] `[include @[formOf x] true true]]
  	return nothing
  	]]
  	# Remap Glyph's methods to macros in order to simplify writing
  	define-macro set-width : syntax-rules
  	`[set-width @::args] {'.syntactic-closure' `[currentGlyph.setWidth @::args] env}
  	define-macro include : syntax-rules
  	`[include @::args] {'.syntactic-closure' `[currentGlyph.include @::args] env}
  	define-macro set-mark-anchor : syntax-rules
  	`[set-mark-anchor @::args] {'.syntactic-closure' `[currentGlyph.setMarkAnchor @::args] env}
  	define-macro set-base-anchor : syntax-rules
  	`[set-base-anchor @::args] {'.syntactic-closure' `[currentGlyph.setBaseAnchor @::args] env}
  	define-macro eject-contour : syntax-rules
  	`[eject-contour @::args] {'.syntactic-closure' `[currentGlyph.ejectTagged @::args] env}
  	###### Canvas-based mechanism
  	define-macro new-glyph : syntax-rules
  	`[new-glyph @body] : begin
  	dirty `[$GlyphSaveSink$.save null null @[formOf body]]
  	define-macro create-glyph : syntax-rules
  	`[create-glyph @body] : begin
  	dirty `[$GlyphSaveSink$.save null null @[formOf body]]
  	`[create-glyph @name @body] : begin
  	dirty `[$GlyphSaveSink$.save @[formOf name] null @[formOf body]]
  	`[create-glyph @name @code @body] : begin
  	dirty `[$GlyphSaveSink$.save @[formOf name] @[formOf code] @[formOf body]]

  • [create-glyph] is (almost always) used to create a new glyph and link it to a name or/and a codepoint. It essentially starts a new glyph "canvas"(?) where you can [include] shapes or other operations to form a full glyph.
  • [set-width] sets the width of a shape, though you can [include] a DivFrame (see below) as a shorthand.
  • [set-base-anchor] maps a certain anchor to a coordinate, indicating where the diacritics (or other composition glyphs) should be placed at. [set-mark-anchor] maps the origin of a diacritic that corresponds to the mark-anchor of the same name.
    • [set-base-anchor] can be sometimes replaced by including MarkSets as a shorthand, and [set-mark-anchor] by including StdAnchors.
  • [eject-contour] is used remove a certain [tagged] shape part, useful if a glyph derives from another by removing/replacing some parts of the reference glyph.
    • On that note, you can include a [refer-glyph] to use other named glyphs as a base.
  • [glyph-proc] is a function-like process that can modify the "current glyph" by including it in a glyph.
    • [composite-proc] is a simplified glyph-proc that just includes several glyph-procs in succession.

---

And for your examples:

• Changing the glyph that just differ by a metric or a transformation is not as easy as you may think.
  • Usually glyphs are created within a [create-glyph], with no consideration of reusing shapes, until such usage is needed (e.g. variants). So unless their shapes are wrapped under some function that allows specification of a "top/bottom/left/right" bound or DivFrame, they just assume the usage of whole cell, following the standard baselines (Descender, 0, XH, CAP, Ascender). If you need to make a stretched version of these shapes, you would need to wrap the existing code under a glyph-proc by yourself (which I have done a few times).
  • See above. Unless glyphs are designed with Narrow/Wide variants in mind (i.e. "Mosaics"), making wide variants are not that straightforward. Possibly part of the reason why issues like Fullwidth part in block "Halfwidth and Fullwidth Forms" #759 are not resolved yet.
  • Transformations can be done by [include]ing these transformations:

    Iosevka/packages/font-glyphs/src/meta/aesthetics.ptl

    Lines 36 to 44 in c982560

    	# Transform constructors
    	define [Italify angle shift] : begin
    	local slope [Math.tan ([fallback angle para.slopeAngle] / 180 * Math.PI)]
    	return : new Transform 1 slope 0 1 [fallback shift : -slope * SymbolMid] 0
    	define [Upright angle shift] [Italify angle shift :.inverse]
    	define [Scale sx sy] : new Transform sx 0 0 [fallback sy sx] 0 0
    	define [Translate x y] : new Transform 1 0 0 1 x y
    	define [ApparentTranslate x y] : Translate (x + TanSlope * y) y
    	define [Rotate angle] : new Transform [Math.cos angle] (-[Math.sin angle]) [Math.sin angle] [Math.cos angle] 0 0

    • These apply to the whole glyph canvas prior to the inclusion of the transformation. If that's not desired, [with-transform (transformation) (glyph-proc)] can be used, but sometimes it may have problems with global transformations (e.g. italics) as well.
    • Note that not every derived glyphs use these even if they look like "glyph but transformed". They may be built from scratch, due to other considerations such as serif placement, italics, character variants and stroke distortions.
    • The most "applicable" use case would be simple 180deg rotation, which has its own shorthand [turned]
• Check out the autobuild folder for these type of characters. These derived characters usually share a common process, so they are just recorded to a list to be auto-built.
  • composite.ptl takes care of the "multiple character inside shape" cases, transformed.ptl takes care of the super/subscript cases, and unicode-knowledge.ptl stores the list of diacritic compositions that can be precomposed.
  • Mind you, since these are the easiest glyphs to add, the immediately doable ones are likely already filled in by williams or me.
  • Unless you mean the kind of melded composite digraphs like "Æ", then they are implemented one-by-one, usually by creating 2 half-glyphs and composing them. Refer to the modules like these.
• Well, the simplest way is just to search for the "similar characters" or other characters that share the primitive part, then copy from their implementations. You can often find them by their Unicode codepoint. That's how we all started.

---

As for documented PRs... well I sometimes do a very simple explanation on how I interpreted/implemented some glyphs (e.g. #2190), but definitely not to the level of "using what primitives". The main purpose of those notes are simply for Belleve to review, so I doubt the fine-grained details are really necessary (imo).

[Logo121]

As for the issue itself, I guess we can use the Discussion feature for matters that don't really qualify as issues so that it's easier to discuss about things like these. Just a thought though.

[raner]

@Logo121 Thanks for taking the time to write down some of your knowledge about the codebase and answering my questions. This is a great start for the documentation that I had in mind!

[be5invis]

A correction is that MarkSets (or DivFrame's markSet's) add base anchors instead of mark anchors. Mark anchors are used by, well, mark/diacritic glyphs. A mark/diacritic glyph could also contain mark anchors, which is used to build mark stacking feature.

DivFrame is an utility object that used to handle X metrics. Usually Y-direction metrics are simple but I could create a new helper to do that...

Autobuild is one of the most obscure part in the build system -- to save glyph IDs, I did a lot of black magic inside them. I want to do a refactor of these...