Every feature should come along with a unit test

[aguaviva]

This is a usual practice in many projects as it helps increase the quality of the code by catching issues very early.

Does this sound like a reasonable request?

[jimmo]

@aguaviva MicroPython has 98% code coverage, the unit testing of the core VM, compiler, emitters, runtime, standard library and more is exhaustive. Occasionally we miss things (Python has a lot of corner cases), but we continue to add new tests as things are found (and as features are added). These tests run continuously in CI on several different configurations of MicroPython on a range of architectures and platforms, and we also run them on a range of hardware before every release.

Can you give an example of a feature that doesn't have a unit test?

We also have a fairly extensive set of integration tests, most of which run on hardware, including multi-device tests with networking and bluetooth.

The main thing we currently lack is completely automated hardware-in-the-loop integration tests. We do this manually before releases at the moment.

[hmaerki]

I collected some ideas for a hardware in the loop test enviroment. Fully automated.

Please have a look at this page: https://github.com/hmaerki/experiment_microoctopus

I started a discussion here: https://github.com/orgs/micropython/discussions/13464. I am very interested to know in how I could collaborate.

@andrewleech:
Don't get me wrong, there are plans to try to start building up a hardware testing platform like this with bench / racks of hardware, but it's a very expensive job
...
To test these on every commit costs a lot of time and effort to set up an automated suite - time and money that no one has right now, perhaps if someone wanted to fund it ???

time and money that no one has right now - I will be retired soon and there is some time and money...
I sketched a proposal which is not so expensive (~USD 20 per tentacle, say every mcu or device).
What are your plans? How would you build a hardware testing platform?

@jimmo
These tests run continuously in CI on several different configurations of MicroPython on a range of architectures and platforms, and we also run them on a range of hardware before every release

How do you run these tests? How much manual work is involved (for example to installing the firmware)? Would you be interested, for me collaborating with automating?

[andrewleech]

@hmaerki thanks for your reply, I saw your other discussion and repo you shared yesterday but hadn't had a chance yet to respond :-)

I've spoken a number of times in the past with @jimmo and @mattytrentini in particular about setting up something just like this, there's definitely interest for sure!

I know @mattytrentini had some ideas and hardware he was planning on using for this purpose too.

I've written automation scripting at work to handle flashing of firmware and starting tests too, though these days the script running would mostly be handled by mpremote I expect.

Are you on the micropython discord? It might be helpful to chat through some of these ideas there too.

[aguaviva]

Hey @hmaerki, you rock ;) I am very glad you are taking on this venture, I worked in 3 HW companies and in all of them unit tests are considered a critical part of the development process. Plus showing the leadership of putting in place a process that will drive the I2C bug count from 70 to 0 is going to look great in your resume.

[hmaerki]

Hi @jimmo

The main thing we currently lack is completely automated hardware-in-the-loop integration tests. We do this manually before releases at the moment.

Could give me a pointer to the hardware-in-the-loop tests you run for every release. I am now working full steam on Octoprobe (formerly Microoctopus) and I am looking for tests where I can test Octoprobe against.

[andrewleech]

Could give me a pointer to the hardware-in-the-loop tests

I don't know if there's more, but the standard test suite in micropython is run on both Unix and hardware platforms. Eg.

STEP: ~/micropython/tests $ ./run-tests.py --target esp32 --device /dev/ttyACM0
pass  basics/0prelim.py
pass  basics/andor.py
pass  basics/annotate_var.py
pass  basics/array1.py
pass  basics/array_add.py
...

STEP: ~/micropython/tests $ ./run-tests.py --target ?
Traceback (most recent call last):
File "./run-tests.py", line 1176, in
main()
File "./run-tests.py", line 1085, in main
raise ValueError("target must be one of %s" % ", ".join(LOCAL_TARGETS

• EXTERNAL_TARGETS))
  ValueError: target must be one of unix, qemu-arm, webassembly, pyboard, wipy, esp8266, esp32, minimal, nrf, renesas-ra, rp2

There's also the multi tests which involve communication between two instances; these can be two copies of Unix port, two devices or one Unix and one device. Generally for things like networking or Bluetooth

@step: ~/micropython/tests $ ./run-multitests.py -i pyb:/dev/ttyACM0 -i pyb:/dev/ttyACM1 multi_net -h
usage: run-multitests.py [-h] [-s] [-t] [-i INSTANCE] [-p PERMUTATIONS]
files [files ...]
Run network tests for MicroPython
positional arguments:
files input test files
optional arguments:
-h, --help show this help message and exit
-s, --show-output show test output after running
-t, --trace-output trace test output while running
-i INSTANCE, --instance INSTANCE
instance(s) to run the tests on
-p PERMUTATIONS, --permutations PERMUTATIONS
repeat the test with this many permutations of the instance order
Supported instance types:
-i pyb: physical device (eg. pyboard) on provided repl port.
-i micropython unix micropython instance, path customised with MICROPY_MICROPYTHON env.
-i cpython desktop python3 instance, path customised with MICROPY_CPYTHON3 env.
-i exec: custom program run on provided path.
Each instance arg can optionally have custom env provided, eg. ,ENV=VAR,ENV=VAR...

Eg.

STEP: ~/micropython/tests $ ./run-multitests.py -i pyb:/dev/ttyACM0 -i pyb:/dev/ttyACM1 multi_net/*

[aguaviva]

Man, today, yet another I2C bug (#13536) that could have been caught using unit tests. Good luck fixing it and touch wood the fix for it wont be breaking another platform.

[robert-hh]

That is not a bug. It's the way the hardware works. Similar behavior for ESP32.

[aguaviva]

If you consider MP a quick hack done by someone in their free time, then it is not a bug. But AFAIK you are asking for funding and this is a business. But most of the funding gets used to fix a same bug multiple times, also you shift focus constantly to support more and new boards,.... I don't think this is sustainable and MP is going to be more and more broken. And being part of this dynamic is not a story you want to tell to your next employer.

Also, there is a back log of 340 PRs... This is the best way to scare away talented contributors.

[stinos]

Looking at the number of github issues (those are issues, not necessarily bugs) or PRs to judge a project is usually rather tricky and contributors should know that. Just go and look for a couple of fairly large and often used projects on github and check the numbers: they scale with code size and number of users but that doesn't mean the overall quality is an actual issue when using the project.

But most of the funding gets used to fix a same bug multiple times,

That just seems unlikely. I mean, unless you do an actual code churn + failure analysis, this is merely how you perceive things based on one particular problem you seem to be having, and I don't think it is actually a correct statement if you'd look at the full history.

[projectgus]

@aguaviva Claiming every new I2C issue report would have been automatically prevented by unit testing is not helpful, respectful, or defensible. Nor is making derogatory claims about the employability of people who contribute to MicroPython. Please refrain from doing this.

You've made your point, which is that more hardware-in-the-loop test coverage of I2C would reduce I2C bugs. Everyone agrees on this general point.

[aguaviva]

Claiming every new I2C issue report would have been automatically prevented by unit testing is not helpful, respectful, or defensible

You need unit tests, a well defined API and a good hardware abstracion layer. And doing it for the I2C would be a good starting point.

[andrewleech]

These are already all done for I2C, how about learning how micropython already works before telling people what "needs" to be done.
machine.I2C is both the well defined API and hardware abstraction layer, each port implements their bindings from this interface to their respective hardware drivers. There are tests for the original pyb.I2C which machine.I2C was based on, if you want to raise a PR to retarget those tests to machine.I2C then go right on ahead, contributions are always welcome.

[andrewleech]

Arguably it's just a new feature request - dynamically chaning pins at runtime while peripheral is running has never been a requested or documented feature before. So even if there were hardware in the loop tests written, no way they'd be covering functionality like this. Again, just because someone raises an issue here relating to a praticular hardware like I2C does not mean it's a bug in the micropython codebase!

[Carglglz]

The main thing we currently lack is completely automated hardware-in-the-loop integration tests. We do this manually before releases at the moment.

@jimmo I've just put together this https://github.com/Carglglz/micropython-localCI, so it should be fairly easy to automate it now. 👍🏼
It's pretty easy to setup and has no requirements other than PyYAML. So far only unix, esp32 and stm32 ports are supported but should be straightforward to add support for other ports...

[victorallume]

We've recently been looking at renode (https://renode.io/about/) which allows simulation of the entire hardware. Has anyone played with it, or considered it for CI in micropython?

[mattytrentini]

Yes, and we've spoken to the Renode folks who have kindly put together this dashboard:

https://zephyr-dashboard.renode.io/dashboard-micropython.html

It shows all of their (nearly 600) supported boards running MicroPython on Zephyr and tests whether a simple REPL command can be executed. For passing boards (~350 of them) you can even play back a video of it occurring; Brilliant!

For the longest time I've been wanting to test MicroPython running on Renode bare metal but, though I've tried a couple of times with a few boards that ought to be supported, I couldn't get to the REPL without errors. I think the problem is that MicroPython initialises a lot of peripherals and they're not all simulated in Renode, but that's a guess based on the register activity - I haven't had a chance to investigate further.

I saw that Renode v1.15 was just released (13 hours ago!) and would like to revisit this again...but if you have time to also try it out then please do so, I could use the help!

[aguaviva]

HW is not an problem (for $20 you could buy 2 ESP32 and have them run a BLE/WIFI/I2C/I2S test suite that covers 90% of the use cases.) The problem is that maintainers believe this is not important.

[scruss]

which 2 ESP32s:WROOMs? WROVERs? C3s? S3s? S2s? Single cores? All are different, all have different peripherals, some entirely different toolchains. Aside from that, most of them need manual intervention to update and upload the latest test firmware.

Okay, now do samd, nrf and stm32. You're now into the hundreds of physical variants, so it would be a team running full time just to test these. You gonna pay for that?

[victorallume]

@aguaviva - you are welcome to fork the repository, and set up such a test suite - this is an open source project. Like any project this one has plenty if competing priorities. If you wish to get something done faster, you can always look into paying/contracting someone to work on a particular feature you need.

[andrewleech]

The problem is that maintainers believe this is not important.

Now you're being insulting, multiple maintainers and regular contributors have replied here with details about the existing testing being done as well as strong interest and plans on how to improve test coverage.

$20 might buy a couple of unbranded unreliable unrepeatable dev boards from AliExpress, but it doesn't buy the rest of the hardware needed to connect them to PC in a way that they can be automated and monitored, let alone pay for the time to write and maintain a suitable test suite.

[aguaviva]

which 2 ESP32s:WROOMs? WROVERs? C3s? S3s? S2s? Single cores? All are different, all have different peripherals, some entirely different toolchains. Aside from that, most of them need manual intervention to update and upload the latest test firmware.

Just start small, and see where that leads. What do you have to lose? Do you really believe this is going to be wasted time?

[scruss]

It's not going to waste any of my time: you're suggesting it, so you do it. You have everything to gain.

[aguaviva]

I'm not investing any of my time or funds until a proper QA process is put in place.