update frozen_mpy(modules) without recompile everything

[puppet13th]

I have successfully compile mpy 1.19.1 for esp32/c ports using IDF 4.3.4 with some addition modules at ports/esp32/modules.
is it possible to add/update ports/esp32/modules without recompile everything?

[robert-hh]

is it possible to add/update ports/esp32/modules without recompile everything?

Yes and no. You have to re-create the firmware binary. If you just change individual files and keep the build directory, calling Make will just re-compile the changed files. You will still have a lot of output form CMake, which just tells that it did nothing. That's irritating. If you add or remove not-registered files from the modules directory, then Make might not notice the change. Then it helps to delete the build-xxxx/frozen_mpy directory before calling make. That will force to rebuild the frozen bytecode.

[puppet13th]

I was thinking to remove frozen_mpy folder and firmware.bin too,before calling make.i will try it and report back.thank you for your answer.

[puppet13th]

I add some modules to ports/esp32/modules and removed these file/folder:

build-xxxx/frozen_mpy
build-xxxx/frozen_content.c
build-xxxx/firmware.bin

then i call make to rebuild firmware.bin.it only do these.

make: Entering directory '/home/puppet13th/micropython-1.19.1/ports/esp32'
idf.py -D MICROPY_BOARD=GENERIC -B build-GENERIC  build
Executing action: all (aliases: build)
Running ninja in directory /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC
Executing "ninja all"...
[2/11] Generating ../../frozen_content.c
MPY apa106.py
MPY _boot.py
MPY aio_ble.py
MPY aio_ble_bps.py
MPY aio_ble_bps_services.py
MPY flashbdev.py
MPY aio_ble_bps_key_services.py
MPY inisetup.py
MPY aioble/central.py
MPY aioble/device.py
MPY aioble/core.py
MPY aioble/security.py
MPY aioble/client.py
MPY aioble/peripheral.py
MPY aioble/l2cap.py
MPY aioble/__init__.py
MPY aioble/server.py
MPY upip.py
MPY upip_utarfile.py
MPY ntptime.py
MPY dht.py
MPY onewire.py
MPY ds18x20.py
MPY uasyncio/__init__.py
MPY uasyncio/core.py
MPY uasyncio/event.py
MPY uasyncio/funcs.py
MPY uasyncio/lock.py
MPY uasyncio/stream.py
MPY webrepl.py
MPY webrepl_setup.py
MPY websocket_helper.py
MPY neopixel.py
GEN /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/frozen_content.c
[3/11] Performing build step for 'bootloader'
[1/1] cd /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/bootloader/esp-idf/esptool_py && /home/puppet13th/.espressif/python_env/idf4.4_py3.9_env/bin/python /mnt/d/mpy/IDF/esp-idf-v4.4.2/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/bootloader/bootloader.bin
Bootloader binary size 0x54d0 bytes. 0x1b30 bytes (24%) free.
[8/9] Generating binary image from built executable
esptool.py v3.3.2-dev
Creating esp32 image...
Merged 25 ELF sections
Successfully created esp32 image.
Generated /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/micropython.bin
[9/9] cd /home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/esp-id.../home/puppet13th/micropython-1.19.1/ports/esp32/build-GENERIC/micropython.bin
micropython.bin binary size 0x1721c0 bytes. Smallest app partition is 0x1f0000 bytes. 0x7de40 bytes (25%) free.

Project build complete. To flash, run this command:
/home/puppet13th/.espressif/python_env/idf4.4_py3.9_env/bin/python ../../../../../mnt/d/mpy/IDF/esp-idf-v4.4.2/components/esptool_py/esptool/esptool.py -p (PORT) -b 460800 --before default_reset --after hard_reset --chip esp32  write_flash --flash_mode dio --flash_size detect --flash_freq 40m 0x1000 build-GENERIC/bootloader/bootloader.bin 0x8000 build-GENERIC/partition_table/partition-table.bin 0x10000 build-GENERIC/micropython.bin
or run 'idf.py -p (PORT) flash'
bootloader  @0x001000    21712  (    6960 remaining)
partitions  @0x008000     3072  (    1024 remaining)
application @0x010000  1515968  (  515648 remaining)
total                  1581504
make: Leaving directory '/home/puppet13th/micropython-1.19.1/ports/esp32'

the process only take a few minutes.

[puppet13th]

first build i only add aioble module(1st compile)
second build i add aio_ble* module(2nd recompile)

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4396
ho 0 tail 12 room 4
load:0x40078000,len:13884
load:0x40080400,len:3340
entry 0x40080618
MicroPython v1.19.1 on 2022-09-16; ESP32 module with ESP32
Type "help()" for more information.
>>> help('modules')
__main__          apa106            uasyncio/__init__ upip_utarfile
_boot             btree             uasyncio/core     uplatform
_onewire          builtins          uasyncio/event    urandom
_thread           cmath             uasyncio/funcs    ure
_uasyncio         dht               uasyncio/lock     uselect
_webrepl          ds18x20           uasyncio/stream   usocket
aio_ble           esp               ubinascii         ussl
aio_ble_bps       esp32             ubluetooth        ustruct
aio_ble_bps_key_services            flashbdev         ucollections      usys
aio_ble_bps_services                framebuf          ucryptolib        utime
aioble/__init__   gc                uctypes           utimeq
aioble/central    inisetup          uerrno            uwebsocket
aioble/client     math              uhashlib          uzlib
aioble/core       micropython       uheapq            webrepl
aioble/device     neopixel          uio               webrepl_setup
aioble/l2cap      network           ujson             websocket_helper
aioble/peripheral ntptime           umachine
aioble/security   onewire           uos
aioble/server     uarray            upip
Plus any modules on the filesystem

[jimmo]

FYI @puppet13th I think this is a bug in the ESP32 build scripts. I ran into it myself the other day. On other ports it just works automatically. (The ESP32 build is extremely complicated due to how we integrate with the ESP IDF and in particular these sorts of custom rules that the IDF doesn't know about are hard to get right)