Same Goal...

[kdschlosser]

I noticed that you opened an issue in the python-openzwave repo and I did some reading about your project. While I was reading I realized that my opinions are not much different then yours with respect to the current Home Automation hot plate. I am using the term Home Automation in the loosest way because this is not an accurate description of what most available software actually does. the term that fits better is Smart Home. There is nothing automatic about having to pick up a phone or tablet to press a button to turn the lights on or speaking a command to turn the lights on. and your software seems better suited to actually be called a Home Automation control software. I have not yet tested your software but your description of it seems to be in line with that.

I have written an object oriented thread safe version of python-openzwave that has the functionality you asked about, I would have to go back and look but I do not believe that the current release supports what you are wanting to do. The large majority of python-openzwave has been rewritten to solve a plethora of problems, some examples of the problems are.. thread safety, network locking, incorrect data from the devices and really poor performance. I also expanded the functionality some examples are association groups, device support, devices having more then a single control point, this last one is the most important. It builds the classes for the devices dynamically adding only code that is relevant. using the library in an application is a snap to do. the library code can be extended to include application code without having to override anything. no monkey patching necessary!. This functionality allows an application to use it's normal structure for devices without having to create or hold reference to any other objects. an applications representation of a device would get constructed at the same time the library builds a python representation of the device. Another feature is a complete new build system and that new system is able to compile python-openzwave much much faster. a 5 minute build time is now 45 seconds on my workstation (6 cores @ 3.5ghz). Performance of the library has increased quite a bit. When starting the pytyhon-openzwave a 20 device network would take over 2 minutes to load. with my changes it takes 0.9 seconds to load the data needed to show the user a representation of the device and 15 seconds until all 20 devices were fully loaded and can be controlled. If you are interested in checking it out let me know. I do not have the code available on github I would have to send it to you some other way.

You might want to take a look at a project I am the administrator of called EventGhost (EG). You may or may not have heard of it, EG has been around since 2005. I do not know of any other open source automation software that is as old and is still being developed. It only runs on Windows at the moment, This is one of the things I am wanting to change but I have not been able to locate someone that knows Python and also proficient with writing code to access the API for Linux type OS's. The time it is going to take me to learn and also code the changes would be fairly long. Someone that has the knowledge can get it done in a fraction of the time.

EG was originally written to run on a Windows XP machine, and resources were not so plentiful back then. Single core machines with 256mb of memory and sub 128gb HDD's were common. EG has a low CPU/memory/disk footprint, using 45mb ram (with the GUI), 76mb disk and fractions of a percent of the CPU. It uses and event/action type of system that is really easy to use. It also has real time python scripting support that is able to have data that persists between script runs. There could be some things you see that spark some ideas for ya.

let me know if you are interested in the zwave library I wrote.

[blacklight]

Thanks so much for the post! platypush aims indeed to solve the problem of home automation with complete customizability in mind. I also started to develop it long ago (I started the initial code of platypush around 2013), but it's nice to see it grow - and if you have any user-side suggestions they're more than welcome :)

I've found your fork of the C++ openzwave library. Does that solve indeed the problem with creating and managing groups? I'd be quite interested because I'm struggling a bit to integrate some features from openzwave (among those, managing features and properly setting values). I know that python-openzwave is currently unmaintained and looking for new maintainers, I have also recently opened a pull request but I don't have many hopes of it being picked up. If you have further insights or a maintained repository that fixes these issues I'd be very glad to update the zwave dependencies of platypush (and maybe it may also be worth to pick the baton of the official python-openzwave repository).

I've also taken a look at EventGhost. Really good job and an impressive number of integrations! I feel however that its limitation to Windows may be a bit of a downside - especially nowadays that a lot of automation runs on RaspberryPi and similar devices, or on Unix-based cloud nodes. Also, the reliance on a GUI app makes it a bit limited, as you'll need a display or a VNC connection to manage it - but, on the other hand, platypush is heavily command-line based and YAML-config-based and the web interface still lacks features when it comes to configuration, so I may definitely take inspiration from your UI.

If you don't mind I'll lurk a bit around the EventGhost plugins and see if I could reuse some of that code (feel welcome to do the same with my codebase :) ).

[kdschlosser]

I do have all of the plugins in a single zip file if you want them. I am not the original author of EventGhost tho there is not much of the original code left. The plugins are scattered about our forum which makes it difficult to sort through them. I am in the process of adding a repository for the plugins to our web server so a user can download them directly from the application. I do also plan on integrating in the ability to access the support forum topic for a plugin from within the application as well with the ability to collect any relevant data that may be needed if a plugin has an error while being used. that information can optionally be sent to the support for the plugin. This will make a plugin authors job a whole lot easier to do having all of the information they need to diagnose an issue right there without having to ask the user to collect it for them.

my fork of openzwave is not what you would need to use. my version of python-openzwave uses the version of openzwave that has been released. I would have to send you a copy of the work I did.

I started writing a modified version of pyozw some 2 years ago. It started with getting the thing to compile on windows. when openzwave released it's new version 1.6 pyozw was not able to use it. There were extensive changes made to ozw and those changes required a lot of changes in pyozw.

pyozw is a rewrite of a far older python binding to openzwave and that was only written to provide a very limited binding. HomeAssistant is the largest project that uses python-openzwave. HASS is slow. it has a single event thread that queries each of the added plugins or extensions for updates. it does this even if there are no updates. so the more you have installed and running in HA the slower it gets. well ZWave is also not a speedy thing to use. and it takes a while to query the devices for status updates. so the more devices you have the slower it gets. this is a bad design. the other thing was in openzwave when a change comes in from a device a callback is made to pyozw when pyozw does what it needs to do a callback is made to the application. the application then processes whatever it needs to process before the thread is able to return to openzwave. while this is going on everything stops in openzwave. so if another device sends in an update it gets lost.

in order to fix this problem openzwave would need to have alot of code changes made. I was able to greatly reduce the problem in the pyozw code. the author of HASS was not willing to make the changes needed in order to use my version. he wanted to have a drop in replacement. He did not want to hear that it is impossible to make a drop in replacement that would work with the new version of openzwave. He stated that it would take to long to make the changes needed.

It took me 1 hour to make the changes to HASS in order to get it running. and that is with me not knowing a thing about how HASS works. HASS has far to much voodoo magic code in it to use an IDE properly when writing anything for it. Help is no where to be found if wanting to write an extension for it. But yet it still only took me an hour. He simply didn't want to make any changes. My thoughts on it were that if HASS is using an external module then HASS needs to be written to work with that external module, that external module should not have to be written to work with HASS. the author of openzwave doesn't seem to care about other people that use python-openzwave or openzwave other then HASS and wants to have everything written to work with HASS. I did not share this same design philosophy and we parted ways. There were other people involved that were working on getting my version to run 100% in HASS and they were running into problem with HASS and it not playing nice because of how HASS was written and because it had incorrectly overridden some parts of the std lib. They to were unable to get any help from the author of HASS to get things running right. They understood why I decided not to help anymore. From my understanding HASS is going to be writing their own binding to openzwave. Because in some way they think that it is going to be easier to do that then to make the changes needed.

You may like this feature I wrote into python-openzwave. It can be installed as a service (Windows, OSX, Linux). and a remote or local connection can be made to the service using python-openzwave. the only change to the whole API is passing an ip address for the device instead of a serial port and supplying a password. the password is actually an AES 256 key that has to match what the service is using. optionally the connection can we wrapped with SSL. The number of simultaneous connections can be set (yes more then one connection can be made). and IP addresses that are allowed to connect can also be set. Security is not a problem, and because no type of password or username is ever shared between the 2 over the connection the possibility of "hacking" into it becomes almost impossible.

my version is object orientated. So if you wanted to add a node to an association group it is done as easily as association_group += node and if you wanted to remove a node. association_group -= node. how about listing all of the association groups a node is added to. associated_to = node.is_associated_to. A node has the ability to have multiple endpoints. so a single zwave interface is able to control more then one "device" there is no handling of this in the current version of pyozw. my version there is. technically speaking a node is not a device. a node is an interface. more then one device can exist for a single node. I have a landscape lighting controller that has 6 zones. I am able to control each of those 6 zones individually. well these are represented as virtual nodes. I added the ability to name and also supply a location for each of them separately.

The whole value system that was originally written went right out the window, if you have some need to access a value directly it's done using friendly names instead of hunting and pecking about for index numbers and labels. node.values.switch_state. there are methods and properties that do the heavy lifting so if you wanted to get the state of a switch print node.switch_state, node.switch_state = True. The properties and methods available are added if the node supports those features. I came up with a really clever way of having the application know what is available and without the need of having to ask or perform any checks.

There is no need to know anything about node id's or group indexes. in fact you do not even need to know what a value is. I removed all of the for and while loops that were used in the original code. The locking of the network when some commands were issued has been removed as well.

I wrote a dynamic thread worker system to handle incoming data. it can be set to use a single thread or it can be set so that it will dedicate threads to each node. when in multi threaded mode the threads will timeout if nothing is taking place. when a node reports a change the thread will start and then handle the processing of data for pyozw and also handle the callback to the application. this releases the openzwave thread to go back to receiving data from the nodes. If the thread for a node is already running the notification gets placed into a queue and the openzwave thread gets released. This type of mechanism keeps all of the notifications in order. It also allows the processing of changes to multiple nodes to happen in parallel. More then one person can turn a light on at the same time, an application should be able to process changes made to multiple nodes in parallel.

There are some things I was not able to overcome these limitations were in openzwave.
While openzwave does work most of it was written before the API was released to the public. There are portions of it that are not correct and there are portions of the API that are not available in ozw. I was told that it would be to much work to fix the incorrect things and to add the missing pieces in and that it would not get done. Kind of a let down.

I did not like that answer so I started writing a pure python ZWave engine. one that is not handicapped in a manner that causes it to stop receiving and sending while an application is processing changes. It will support as close to 100% of the specification as possible. I have found errors in the specification and those errors are slated to be updated and corrected at some point. API documentation for the specification has conflicting spots in it and incorrect information and Silabs has been informed of these as well, and they should be updated at some point. It is going to be a while until I complete this project. the ZWave specification is a monster, and to get it all written and written correctly is no small undertaking.

I am more then willing to share the code I have for python-openzwave. I have to make changes to it and remove the python-openzwave name from it. I do not believe there is any original pyozw code left in it and it is really no longer a fork and it should not be tied to python-openzwave in any way other then me stating that I got inspiration from it. I will create a repository for it if you are still interested. I do have to say that I have been running it personally and I have no issues with it. However I do not have all of the devices available on the market so I am not able to test all of it's functionality. I am sure that there are bugs or things that need some correcting. It is stable and can be run without issue for the majority of the users out there

Let me know if it is something you are interested in looking at. I can provide examples of how to use it. There is documentation available for almost all of it.

[blacklight]

You've hit two very good points when it comes to HASS. Both platypush and HASS were kickstarted around the same time (about 10 years ago). The reasons why I decided not to converge my efforts into HASS were:

• I really dislike the single-threaded, single asyncio-based loop mechanism used by HASS. Although it makes things slightly easier when it comes to development, as you don't have a lot of concurrency issues to handle, it increases the overhead of the whole application because of the internal polling, it makes HASS slow to run on less powerful devices, and it increases the chances of non-deterministic behaviours in case two events need to be processed at the same time. I've tackled these issues in platypush by relying on a completely asynchronous design that completely decouples the message delivery infrastructure from the business logic of the integrations. Internal messages are delivered over Redis and websockets, which makes things both very responsive and easy to integrate with. Each backend, executor and procedure runs in its own thread, it's possible to process new messages while running some procedure or action in the background, and the business logic of the integrations never gets in the way of the core application. The integrations mostly listen for messages on a queue or synchronize on threading events, event hooks are run in their own threads and respond to messages received on the bus, and all the parts of the application do nothing and use no CPU resources unless they're actually expected to do something. This makes platypush the highest performant framework I've seen in home automation so far. At its peak it consumes less than 20% of the CPU on a tiny RaspberryPi Zero while running 15 integrations (included a voice assistant, a web server, a websocket service and a serial backend), and I even run it on a 12-year-old Nokia N900 phone with Linux installed (even there, it takes around 20% of CPU load with many integrations, included GPS, enabled). Real parallelism, high responsiveness, extremely low latency and a support-all-platforms approach were really key elements in my design, and I don't get how HASS could compromise on them. Also, unlike HASS I tried to build platypush as a tool that can be used in many ways. You can subscribe to the events it generates or send messages to it over Redis, websocket, MQTT, JSON-RPC API, Kafka, web panel etc. - I believe that it's the user's decision to pick their favourite way to interface with it. You can even use it as a library in another project or script instead of running it as a stand-alone app, through an interface as simple as a get_plugin('light.hue').on() or get_plugin('zwave').activate_scene(label='MyScene'), or get_bus().on('platypush.message.event.EventType', your_callback). And developing a new plugin is as simple as extending the Plugin class and using the @action decorator on the methods that you want to expose, without dealing with the asyncio bus internalities and the co-routines hell of HASS. A good tool should get as much as possible out of way of the user's use case and just do its job.

• I also don't agree with the "works with HASS" philosophy. It's something I've already struggled with many times, especially when working on integrations that have their most used/maintained library as a HASS-dedicated library. HASS favours libraries that use the asyncio/co-routines/single-threaded approach because that's the paradigm it uses. Instead, I've always believed that a good library should be as agnostic as possible when it comes to the event-handling paradigm, and hide as much as possible its messaging infrastructure and internals from the 3rd-party developer. It should be easy to integrate into applications that use threads, multiprocessing, asyncio or event callbacks without being opinionated on what's the best approach, and it should be thread-safe by design, because people may want to use the library in threaded contexts and they should be able to do so without bothering with internal synchronization mechanisms. Instead, many libraries are now designed to support the HASS use case and that's it: that, in my opinion, sacrifices reusability and defeats the whole purpose of building a library (call it a module for HASS, not a library). Because of it I've had to rewrite some of the integrations of HASS almost from scratch for them to be usable in platypush, and that's an anti-pattern when it comes to open-source libraries.

That being said, I'd be very interested if you could publish your version of pyozw. I like the idea of it running as a separate service (the integration I've built with Zigbee follows a similar pattern, as it relies on zigbee2mqtt), even if I generally prefer a stand-alone library - again, I'm usually quite strict on platypush being as atomic, self-contained, easy to deploy and high-performance as possible; plugins should always be as simple to install as a pip install 'platypush[plugin_name]' without running external services whenever possible.

I'd really appreciate if you could publish it on GitHub - it'd make life simpler to anyone who is struggling with ozw like me and it'd make it very easy for me to handle it as an external dependency. Also, feel free to try and use platypush yourself, I'm pretty sure you'll have plenty of good advice for it.

[kdschlosser]

As far as EventGhost is concerned. I would love for it to be cross platform This is something I have been really wanting to do. I am not proficient in the API available for other OS's. I have been hoping that someone with the knowledge might come along and be willing to help in that aspect. EventGhost (EG) can run without the GUI and can run as a service, there are some portions of EG that would not be available to a user if running this way, things like moving the mouse, or changing an applications window metrics. For these features to be available there would need to be an application that gets run upon user login to a GUI and those abilities would be carried out by that application.

EG is for Home Automation it is not for a Smart Home. a Smart Home has a GUI that a user is able to interact with to turn the lights on or off. or to power a TV on or off. Home Automation does not need a GUI in this respect. it is not something the user interacts with in that type of a way. the GUI is only for setting up the application. In fact the save file is XML and can easily be written and loaded without the need to launch the GUI what's so ever. Home Automation is the next step. It was called Home Automation back in the days of x10 but it wasn't really home automation back then. All of the applications that people like these days most of them are not Home Automation either. There is no difference between flipping a light switch on a wall or picking up a cell phone and pressing a button to turn a light on.. It is still a person doing it. A GUI is able to be created using the webserver plugin if the user has a want to access the devices in that manner. there are loads of dashboard frameworks available that would be able to achieve this and have that dashboard connect to the EG webserver to collect the different metrics and metadata from EG for the various devices attached to the system.

[kdschlosser]

well I can tell you this.. EventGhost in terms of performance would give your project a run for it's money in terms of performance... EG is unable to run on a PI but I can tell you that it was designed to run on a Windows XP machine. Pentium processor single core 233 mhz.. the PI is leaps and bounds faster then that.. LOL..

But for the sake of argument EG only uses a single core of a processor. it is not written to use multiple cores.

On my workstation running 6 cores @ 3.5 GHZ the memory footprint is 45mb with the GUI loaded. I am able to process 1000 events in 428 milliseconds using < 10% of the CPU. This is running the "standard" version of EG so the events are processed one after another and not in parallel. events are able to be created in parallel but not processed in parallel.

I have a "special" version of EG. one that is able to process events in a concurrent/consecutive manner. The reason why it processes the events using both a consecutive and concurrent manner is this..

say you have an extension or a plugin that has it's own thread loop to collect data from a device. or it is an idle loop that only reacts to the device reporting a change. either way it doesn't matter. It is of utmost importance that the plugin is able to go back to getting the next update so we want to release that thread to do so as fast as possible. that thread passes off the information needed to create the event to another thread. the "event bus" then the plugin thread gets release to go back to get more data or to wait for incoming data.

the event bus is responsible for managing the thread workers for each unique event. If an event has never occurred before a thread worker gets created and then stored. It it has taken place before that worker gets collected from the storage. the thread worker (whether new or old) has a run counter that gets incremented by 1. If the thread worker does not have a thread that is already running one gets created and it gets started. the thread will loop and process whatever it needs to each loop it reduces that counter by 1. on the last iteration of the loop (counter at 0) at the end it will wait a specified amount of time, say 3 seconds before exiting. if the counter is changed during that 3 seconds the wait is exited as soon as the counter is changed and the loop starts to run once again. This makes sure there are no idle threads just sitting about doing nothing and consuming resources.

The purpose for the consecutive is we do not want to have multiple threads telling a TV to turn the volume up at the same time. we want it to happen one after another.

That special version of EG is able to process events a HEAP load faster. this does come at a cost of RAM and CPU use. we are talking a bump to about 75mb of ram and the CPU use depends on the number of events. If it is processing 50 events each event 1000 times it takes maybe a second or so and will hit 20% CPU use (maxes out the single core). The time to process 50000 events vs 1000 events is about double. It only consumes resources when they are needed. I have tried to reduce as much resource waste as possible while providing the best possible performance. repeat events are where the speed matters the most. EG handles IR remotes from just about every manufacturer made. SIRC12 IR protocol is able to spit out repeat codes when a button is held every 45 milliseconds. we want to keep that processing to as close to 45 milliseconds as possible so no "lag" or continued processing of events is taking place after the button gets released that is the ideal goal. I times it once to see how bad that lag was. with a version of the IR decoding that I wrote along with a pure python MCE IR Receiver interface running from the time the code was first received until the event was completely processed took < 2 milliseconds. which is really good due to the IR decoding having to test almost 50 different protocols to see which one it is receiving. If I keyed it to only the single protocol I would imagine it would be sub 1 millisecond.

EG at idle when it is not processing any events sits at < 1% for 90% of the time. and may spike at no higher then 3%

It sounds like we share the same thoughts on how an event system should be written. with keeping resource use as priority one.

The Windows specific components of EventGhost can be separated in a manner that would allow EventGhost to be cross platform. My lack of knowledge as far as the other OS's is what the issue is. It would take me a long time to get familiar with the API of Debian and Darwin at a minimum. The GUI is already cross platform so there would be no issue there. I do not know how I would go about making the equivalent of a windows executable for Darwin or Debian.

[kdschlosser]

Running my version of pyozw as a service is an option not a requirement. There are several benefits of doing this. the first is on a multi core system running the engine in a different process you will be able to specify the core to run on. This would help load balance the program across multiple cores. as you already know I am sure is that Python is a single core program unless it is instructed to be otherwise. This has a level of complexity and overhead that make it almost not worth doing. But if one piece can run standalone and an application can run in a different process and then connect to it you now have a solution that is using multiple cores.

The other is this.. ZWave's network topology is a mesh. it deals in having multiple known "neighbors". because each device is also a repeater it is able to make the network cover a large physical area using low output radios. While this is a great thing it does have a single horrible flaw.. The radios are not full duplex, and the devices lack the CPU power to be able to do more then a single thing at a time. So if you have the controller at one end of the house all of the network traffic is going to have to run through a handful of nodes that are closest to the controller. if this is say 3 nodes and you have 200 devices.... there is going to be a problem and packets are going to get lost.

with the version I made a ZStick can be plugged into a raspberry pi and run only python-openzwave. the pi can then be connected to by the application and interface with the ZWave network that way. Using the exact same API that would be used if the ZStick was plugged into the local machine running the application. This would allow a user to be able to install the application on a more powerful computer that is not located in the center of the home. and with only a cost of 35.00 dollars to buy the raspberry pi they can place the ZWave radio in a central location inside of the home. this would solve the problem of lost packets. It is also going to greatly speed up the ZWave network because it is less hops that have to be taken between a device and the controller.

That was the original design idea I had and why I made that option available. The other main reason why i did this is because it does take some time for the zwave to becomes 100% usable. by having the ZWave engine run as a service the application can be restarted without the ZWave engine having to. This greatly reduces the application startup time to only a few milliseconds for the ZWave portion.

in theory the ZWave engine should never need to be restarted, it should be able to run forever.

[blacklight]

@kdschlosser the way EG processes events and messages sounds quite similar to what I do in platypush - except that you pre-allocate the worker threads per event - see __init__.py, request module and Redis bus module. It's always nice to see different people who didn't communicate before converge on similar solutions for the same problem.

When it comes to the GUI, I've solved the problem of platform inter-compatibility in platypush by providing a web panel and a web dashboard, so any system can access it as long as it has a browser or a web rendering engine.

Unfortunately, I can't try EG because I don't run Windows on any of my machines, and I'd have to install a VM for the purpose (and for the same reason I wouldn't really use it in my environment, as I run mostly RaspberryPis and other Linux-based devices). But feel free to try out platypush to check yourself whether it matches your expectations. Otherwise, we keep on this "my system is so cool" talk without really looking outside of our comfort zone :) there's a wiki for kick-starting and a list of the available integrations, all extensively documented. I've put so much work into platypush over the years, built so many integrations and worked so much to make it cross-platform that I'm very unlikely to drop it to contribute to another project.

When it comes to pyozw, again, I'm very interested in your version, especially after I've already hit several walls using the current version - and, being the library unmaintained, I already know that it's unlikely for them to be fixed soon. Again, I'd be very interested if you were keen to upload your project on Github, so I can take a better look and understand what needs to be done to migrate the code to it.