Port from Python to Micropython

[fmafma]

Hi!

A few years ago, I developped a KNX framwork to create virtual devices more or less the same way real devices work:

https://framagit.org/fma38/pKNyX

I have a few devices running for years on an old router (with OpenWRT firmware), and they work great 24 hours a day.

I just ported pKNyX to python3 (still bugs to fix), and I now have new goals:

1. port the framework to Micropython for linux ;
2. implement TPUart protocol;
3. port the framework to Micropython for RP2040/ESP32/others.

For point 1. I guess I will have to remove/simplify things, but I don't really know how to start! Do you have some advices? For example, how to replace threads by asyncio? I never did that...

Thanks,

[fmafma]

In the current implementation, I have 3 methods running as threads:

2 from UDPTranceiver: https://framagit.org/fma38/pKNyX/-/blob/master/pknyx/stack/transceiver/udpTransceiver.py?ref_type=heads
1 from L_DataService: https://framagit.org/fma38/pKNyX/-/blob/master/pknyx/stack/layer2/l_dataService.py?ref_type=heads

Then, I start apscheduler BackgroundScheduler (I suppose it runs in a thread too).

Finally, I just loop forever (this loop can be overwritten if needed, to handle user specific things).

The UDPTransceiver threaded methods _receiverLoop() and _transmitterLoop() manage the multicast sockets (in and out).
The L_DataService threaded method _loop() manage a queue for the incoming messages (in order to transmit them to the upper layer).

[fmafma]

I started to migrate the threads to coroutines, and in the way, I needed to changed some internal things, like queues.

I'm trying to use Peter's implementation, from 'primitives' package, but I have errors I don't understand. The first one is:

RuntimeWarning: coroutine 'Queue.get' was never awaited
  type_, src, gad, priority, data = groupMonitorObject.queue.get()

How do I 'await' the get() method?

The other error (maybe related) is:

    type_, src, gad, priority, data = groupMonitorObject.queue.get()
TypeError: cannot unpack non-iterable coroutine object

Any help welcome.

PS: for now, I run my tests under CPython, as there are a lot of other things to change in order to use Micropython...

[andrewleech]

  type_, src, gad, priority, data = groupMonitorObject.queue.get()

How do I 'await' the get() method?

Try:

    type_, src, gad, priority, data = await groupMonitorObject.queue.get()

PS: for now, I run my tests under CPython, as there are a lot of other things to change in order to use Micropython...

Yep when you've got code that already works on cpython, doing big changes like porting from threads to async is definitely worth
doing under cpython before trying to run it all on micropython, it's really helpful to do this!

[fmafma]

    type_, src, gad, priority, data = await groupMonitorObject.queue.get()

Thanks, it fixed both errors!

[fmafma]

So, next move is to 'async' the multicast sockets, so they don't block while waiting for data:

https://framagit.org/fma38/pKNyX/-/blob/master/pknyx/stack/multicastSocket.py?ref_type=heads#L120

Ho do I do that?

[andrewleech]

I'm not sure about multicast, but most asyncio networking stuff is handled via https://docs.python.org/3/library/asyncio-stream.html

[fmafma]

Ok, I see. It seems I can give it a socket, so it shouldn't be too hard to migrate...

Ths hard stuff is to ensure all tools I use exist on Micropython side.

[fmafma]

Ah, Micropython is much more limited, and I can't give it a socket, and so can't use UDP.
Anyway, I will try with CPython..

[fmafma]

It's may be related to the fact I'm using CPython, but in primitives.condition, I had to modify this line:

    async def __await__(self):
        await self.lock.acquire()
        return self

adding the 'async' keyword, or I got:

  File "condition.py", line 23
    await self.lock.acquire()
    ^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside async function

[andrewleech]

Yeah that's normal, once you start adding async calls it kinda has to spread all the way back up the call stack

[fmafma]

Mmm, but it fails as soon as I import the module, not when running it...

[fmafma]

I confirm, it only fails with CPython, but not with Micropython.

[fmafma]

Ok, I have a working version :o)

But not perfect... I still have an issue with sockets. I'm using the loop.sock_xxx() functions, which should avoid blocking, but it does not work.

In this code:

https://framagit.org/fma38/pKNyX/-/blob/asyncio/pknyx/stack/transceiver/udpTransceiver.py?ref_type=heads#L134

I had to add a await asyncio.sleep(1) (line 169) in order to let the framework handle the incoming frame after it has been received (line 141). Without it, the loop quickly returns to the socket receive, and blocks everything.

Any idea why?

[andrewleech]

I haven't used the sync sockets much myself so not sure. That socket receive function that appears to be blocking, is it definitely an async / stream receive function?

[fmafma]

AFAIU, sockets are normal sockets, and they are wrapped with loop.sock_xxx() functions, to be 'async-ed'.
And I set the socket non blocking (line 136):

https://framagit.org/fma38/pKNyX/-/blob/asyncio/pknyx/stack/multicastSocket.py?ref_type=heads#L121

[fmafma]

The strange thing is I also set a timeout!

[fmafma]

Arf, I just pass, in the timeout exception! If I do:

            except socket.timeout:
                await asyncio.sleep(0)

It works! With a short timeout (0.1s), it is ok. I will try to remove the loop.sock_xxx() encapsulation, to see if it changes anything... It would be nice, as these functions do not exist in Micropython.

[fmafma]

Ok, still works without loop.sock_xxx() encapsulation.

[fmafma]

Ok, I'm now moving to Micropython :o)

Usually, to test my code without installing it, I set PYTHONPATH env dir while executing a script. For example:

$ PYTHONPATH=. python pknyx/scripts/pknyx-group.py ...

The problem is unlike PYTHONPATH with CPython, MICROPYPATH value given on command line replaces the entire content of sys.path, instead of adding the given one!

$ python
>>> import sys
>>> sys.path
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']

$ PYTHONPATH=. python
>>> import sys
>>> sys.path
['', '/home/fma', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']

$ micropython
>>> import sys
>>> sys.path
['', '.frozen', '/home/fma/.micropython/lib', '/usr/lib/micropython']

$ MICROPYPATH=. micropython
>>> import sys
>>> sys.path
['', '.']

And it does not expand the current dir, like PYTHONPATH does...

Is there a way to avoid giving the complete paths with Micropython?

$ MICROPYPATH=/home/fma:.frozen:/home/fma/.micropython/lib:/usr/lib/micropython micropython
>>> import sys
>>> sys.path
['', '/home/fma', '.frozen', '/home/fma/.micropython/lib', '/usr/lib/micropython']

[andrewleech]

Oh I didn'y know PYTHONPATH behaved that way!
I don't know a better way to manage MICROPYPATH sorry, though I often symlink my dev packages into ~/.micropython/lib to avoid setting path manually

[fmafma]

Ah, using symbolic links is a good idea! Thanks.

[fmafma]

Another question: Micropython socket.recvfrom() method is supposed to returns the from address as bytes. But I get strange values, which are not always the same for the given machine! For example:

b'\x02\x00\x86\x0e\xac\x11\x16\xd2\x00\x00\x00\x00\x00\x00\x00\x00'
b'\x02\x00\xaa\xca\xac\x11\x16\xd2\x00\x00\x00\x00\x00\x00\x00\x00'

[fmafma]

No idea?

[andrewleech]

I've never used this function myself.

[robert-hh]

But I get strange values, which are not always the same for the given machine

Could it be an IP6 address? The size of 128 bit matches.

[fmafma]

Nope, I already tried to convert as IPv6, it does not match.

The fact that the value changes at each call is very strange... Some bytes remain the same, though.

[fmafma]

Ok, I tried to deploy upKNyX on a RPi Pico W, and I have a few problems:

• it looks like multicast requests do not reach the Pico :o(
• I don't have enough memory in certain situations

For the second point, does Micropython load docstrings in memory? Should I remove them to save memory?

[fmafma]

Thanks for the edit!

[fmafma]

• it looks like multicast requests do not reach the Pico :o(

There's hope, I can send multicast requests!!! I guess I have to tune the receive socket... I have to admit that I don't understand everything about multicast :o/

[fmafma]

Mmm, according to: #9105
the problem should have been fixed: georgerobotics/cyw43-driver#25

But it still does not work a my RPi Pico W :o(

Could someone make a test with the given examples above?

[fmafma]

I think I found the issue: it seems to only work with "224.1.1.1" multicast address; as soon as I send a multicast message on another address, it does not reach the Pico :o(

That's a problem, as I can't choose this address: it is set by the KNX standard.

[fmafma]

Pff, I was wrong, it works!

But I had to reboot the Pico and send the correct socket settings. With my several tests, I may have set a bad configuration, which may have remained between calls. Or it may be something else, I don't know... I have so much things to debug in parallel, it is not easy...