How to organize polling of a large number of IP addresses via Modbus TCP (for example, 30)? (IDFGH-9251)

[iamalek]

A maximum of 5 IP addresses with associated UIDs can be specified in the table.
Or 1 IP address with many descriptors in which you can set any UID and all of them will be tied to only one IP address.

[alisitsyn]

Hello @iamalek ,

The esp-modbus communication approach assumes that the master connects to all defined slaves first before performing any communication with slaves. This approach follows the Modbus TCP application implementation document from the Modbus Association. This is because the connection timeout may be up to ~3 seconds. This is what can cause the response timeout management issue.The master supports the maximum CONFIG_FMB_TCP_PORT_MAX_CONN defined in kconfig. However, it is not recommended to make more than five connections simultaneously because it takes up enough resources and memory and can cause latency and other issues.
The resources of the ESP32 chip and its memory are significantly constrained compared to a Linux-based device or other similar PLC.
Because of this, please consider using fewer slaves with your master.

[iamalek]

Thanks for the answer! There is one more unpleasant drawback of this implementation of the library - if at least one connection with the slave cannot be established at the time of initialization by the master, then the master hangs on it, waiting for the connection and making it impossible to work with the rest of the available slaves. This is a very significant shortcoming, please advise how to organize the work of the master, regardless of the availability of connecting slaves during initialization! Thank you!

[alisitsyn]

Thank you for your feedback. I agree that this approach is not very good in some projects. For the mentioned functionality, the code shall include event-based state machine and according to the slave state will go through all the required phases and do re-connection as required. I implement this behavior and going to include this in some update of Modbus.
However for existing approach we can do as below:
It is possible to start communication while the connection cycle is not completed, but it requires some modification of port layer code. The simple approach is to change the connection cycle to pass just one asynchronous connect command xMBTCPPortMasterConnect() for each slave config in xMbPortConfig.pxMbSlaveInfo[slave] despite the errors: ERR_CONN - connection can not be established and ERR_INPROGRESS - connection in progress. Then start polling cycle even while the connection is not established. The read/write functions will return the errors for these broken slaves, and if the ERR_CON is returned, the connection command can be issued again for the appropriate slave. Also, the stop polling and re-connection functions here should be removed to continue polling in spite of errors.

see also here

[alisitsyn]

Hi @iamalek ,

This feature request is implemented in modbus stack v2.0.0_beta1. This version will be able to handle the connection and resolving stages for each slave individually.

[pr200sd]

Thank you for your feedback. I agree that this approach is not very good in some projects. For the mentioned functionality, the code shall include event-based state machine and according to the slave state will go through all the required phases and do re-connection as required. I implement this behavior and going to include this in some update of Modbus. However for existing approach we can do as below: It is possible to start communication while the connection cycle is not completed, but it requires some modification of port layer code. The simple approach is to change the connection cycle to pass just one asynchronous connect command xMBTCPPortMasterConnect() for each slave config in xMbPortConfig.pxMbSlaveInfo[slave] despite the errors: ERR_CONN - connection can not be established and ERR_INPROGRESS - connection in progress. Then start polling cycle even while the connection is not established. The read/write functions will return the errors for these broken slaves, and if the ERR_CON is returned, the connection command can be issued again for the appropriate slave. Also, the stop polling and re-connection functions here should be removed to continue polling in spite of errors.

see also here

Good afternoon.
You can make changes to the Master TCP, I tried to make sure that if one of the slaves loses connection, the polling of the second one continues, but I can’t make it so that at startup, if there is no connection with the slave, the polling of the second one starts correctly, and after it appears on the network, a reconnection occurs.
port_tcp_master.zip
This is my file from version: "1.0.15"

[alisitsyn]

Hi @pr200sd,

I just took a look to your change and it seems the change you made

pucMBTCPFrame[MB_TCP_UID] = pxInfo->ucSlaveAddr;

is not needed. When the xMBMasterTCPPortSendResponse() is called the UID field in the frame buffer is already set correctly by transport layer. I think the change you made might help in your case but it is not required for stack. Please let me know if I miss something and you have some specific details. I will try to verify as soon as I can.
The v1 stack starts the connection right after initialization of the stack as it recommended by implementation guide from association. The version v2 is able to configure network and start polling immediately. Each slave stage such as connection, address resolution, error tracking, re connection and communication is controlled independently for each slave node. I have the updated implementation of v2 stack but the test infrastructure does not adequately test this yet. I think you will need to use the new stack implementation for your project.

[pr200sd]

Hi @pr200sd,

I just took a look to your change and it seems the change you made

pucMBTCPFrame[MB_TCP_UID] = pxInfo->ucSlaveAddr;

is not needed. When the xMBMasterTCPPortSendResponse() is called the UID field in the frame buffer is already set correctly by transport layer. I think the change you made might help in your case but it is not required for stack. Please let me know if I miss something and you have some specific details. I will try to verify as soon as I can. The v1 stack starts the connection right after initialization of the stack as it recommended by implementation guide from association. The version v2 is able to configure network and start polling immediately. Each slave stage such as connection, address resolution, error tracking, re connection and communication is controlled independently for each slave node. I have the updated implementation of v2 stack but the test infrastructure does not adequately test this yet. I think you will need to use the new stack implementation for your project.

Thanks for the answer, yes, I saw your version V2, but everything has been changed there, and I’m not sure that I can run my project simply by replacing the esp-modbus folder with V2, and since with esp-modbus I use both modbus RTU and TCP and everything works almost perfectly. with the exception of not starting master tcp if at the initial stage there is no one of the configured Slaves, so I would like to get this version, according to the recommendations you gave above. I tried to make the edits myself, but couldn't figure it out completely. As for this edit, yes, I made it back in 22, then apparently it didn’t exist and wouldn’t work without it, and then I moved it to a more recent version. I'll check without her.

[alisitsyn]

Thanks for the answer, yes, I saw your version V2, but everything has been changed there, and I’m not sure that I can run my project simply by replacing the esp-modbus folder with V2, and since with esp-modbus I use both modbus RTU and TCP and everything works almost perfectly.

The v2, has changes but the adoption of your Modbus code will be pretty straight-forward. I need to complete test implementation the verification of this V2 version and will try to share it to get some feedback as soon as possible.
Unfortunately, I am very limited on what can be updated or implemented in V1. The V2 will be the only version where the features will be implemented.

As for this edit, yes, I made it back in 22, then apparently it didn’t exist and wouldn’t work without it, and then I moved it to a more recent version. I'll check without her.

The code was changed long time ago to set the corresponded UID in the frame on transport layer.

[alisitsyn]

Hi @pr200sd,

The stack v2 version is updated and supports the multi-instance TCP Slave and Master with its own communication options. The pre-release esp-modbus component version 2.0.0-beta.1 can be selected by idf_component.yml in main folder like below:

dependencies:
  idf:
     version: ">=4.1.0"
  espressif/esp-modbus:
    version: "^2.0.0-beta"
  espressif/mdns:
    version: "^1.0.0"
    rules:
      - if: "idf_version >=5.0"
  mb_example_common:
    path: "../mb_example_common"
  protocol_examples_common:
    path: ${IDF_PATH}/examples/common_components/protocol_examples_common

[alisitsyn]

@iamalek,
@pr200sd,

with the exception of not starting master tcp if at the initial stage there is no one of the configured Slaves, so I would like to get this version, according to the recommendations you gave above. I tried to make the edits myself, but couldn't figure it out completely.

The stack can be easily updated. For more information you can take a look to the examples of esp-modbus v2. They kept with minimal changes required for V2. The start_disconnected option in the tcp configuration structure allows to start polling cycle while some slave are still disconnected. For the rest of slaves the IP addresses will be resolved and be connected when they become alive and connected.

Please try the update and let me know if this solves your issue. Any feedback is appreciated.

[pr200sd]

@alisitsyn Thank you, I tried to replace the contents of the old esp-modbus-master folder in my idf with what is in the new one, the assembly did not work, the first error is related to esp_timer_is_active, then mbs_ascii_transp_delete
As I understand it, it’s not possible to simply replace one folder with another manually? I have ESP-IDF v4.4

[alisitsyn]

@pr200sd,

Thank you for the feedback.

I tried to replace the contents of the old esp-modbus-master folder in my idf with what is in the new one, the assembly did not work, the first error is related to esp_timer_is_active, then mbs_ascii_transp_delete

Ok, I will take a look and will prepare the example for v4.4. Please send your log if possible. I can help you with the transition to esp-modbus v2.0.0, if it applicable for you.

As I understand it, it’s not possible to simply replace one folder with another manually? I have ESP-IDF v4.4

Unfortunately, it is not enough. The API calls should be fixed. For example thembc_tcp_master_setup() call is removed from new version and the options are being set in the constructor of concrete instance.
The below code describes the changes that are required:
For example the original code from v1.0.x:

static void* master_handler = NULL;

// ... other code 
// TCP master initialization:
    mb_communication_info_t comm_info = { 0 };
    comm_info.ip_port = MB_TCP_PORT;
    comm_info.ip_addr_type = ip_addr_type;
    comm_info.ip_mode = MB_MODE_TCP;
    comm_info.ip_addr = (void*)slave_ip_address_table;
    comm_info.ip_netif_ptr = (void*)get_example_netif();
    esp_err_t err = mbc_master_init_tcp(&master_handler);

    MB_RETURN_ON_FALSE((master_handler != NULL), ESP_ERR_INVALID_STATE,
                                TAG,
                                "mb controller initialization fail.");
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE,
                            TAG,
                            "mb controller initialization fail, returns(0x%x).",
                            (int)err); 

    err = mbc_master_setup((void*)comm_info);
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE,
                            TAG,
                            "mb controller setup fail, returns(0x%x).",
                            (int)err);
    err = mbc_master_set_descriptor(&device_parameters[0], num_device_parameters);
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE,
                                TAG,
                                "mb controller set descriptor fail, returns(0x%x).",
                                (int)err);
    ESP_LOGI(TAG, "Modbus master stack initialized...");

err = mbc_master_get_cid_info(cid, &param_descriptor);
....
// the read code:
err = mbc_master_get_parameter(param_descriptor->cid, (char *)param_descriptor->param_key, (uint8_t *)pinst, &type);

shall be changed to the code below for esp-modbus v2.0:

static void* master_handle = NULL; // keep this pointer to the created master instance, this will be used in all API calls.
// ..... other code

    mb_communication_info_t tcp_master_config = {
        .tcp_opts.port = MB_TCP_PORT,
        .tcp_opts.mode = MB_TCP,
        .tcp_opts.addr_type = ip_addr_type,
        .tcp_opts.ip_addr_table = (void *)slave_ip_address_table,
        .tcp_opts.uid = 0,
        .tcp_opts.start_disconnected = false,
        .tcp_opts.response_tout_ms = CONFIG_FMB_MASTER_TIMEOUT_MS_RESPOND,
        .tcp_opts.ip_netif_ptr = (void*)get_example_netif()
    };
    // Create the master instance with the options in one call (Note: setup call is removed)
    esp_err_t err = mbc_master_create_tcp(&tcp_master_config, &master_handle);
    MB_RETURN_ON_FALSE((master_handle != NULL), ESP_ERR_INVALID_STATE,
                                TAG,
                                "mb controller initialization fail.");
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE,
                            TAG,
                            "mb controller initialization fail, returns(0x%x).",
                            (int)err);
    err = mbc_master_set_descriptor(master_handle, &device_parameters[0], num_device_parameters);
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE,
                                TAG,
                                "mb controller set descriptor fail, returns(0x%x).",
                                (int)err);

mb_parameter_descriptor_t* param_descriptor = NULL;

// The below is the read code:
err = mbc_master_get_cid_info(master_handle, CID_HOLD_FLOAT_ABCD, &param_descriptor);
... 
float float_data[3];
err = mbc_master_get_parameter(master_handle, param_descriptor->cid, (uint8_t *)float_data[i], &type);

The possible ways to integrate the v2 into your project:

1. Update the API calls to according to esp-idf V2 changes (See above, required for both options below).
2. Copy the context of esp-modbus v2.0.0 into components folder of your project.
3. Add the idf_component.yml manifest file into main component folder of your project with the below content:

dependencies:
  idf:
     version: ">=4.1.0"
  espressif/esp-modbus:
    version: "^2.0.0-beta"
  espressif/mdns:
    version: "^1.0.0"
    rules:
      - if: "idf_version >=5.0"
  mb_example_common:
    path: "../mb_example_common"
  protocol_examples_common:
    path: ${IDF_PATH}/examples/common_components/protocol_examples_common

This option will allow to download the esp-modbus V2 component by component manager and use it instead of previous one.

Let me know if you have any questions.

[pr200sd]

@alisitsyn
The fact that the calls in my code will need to be adjusted is understandable, I’m talking about the esp-idf assembly itself, I wouldn’t want to change everything globally, it’s important for me to be able to quickly return to the previous working version of esp-modbus.

Add the idf_component.yml manifest file into main component folder of your project with the below content:
Should the idf_component.yml with its contents be placed in the components folder of my project?
and then the folder from esp-idf/components/esp-modbus can not be replaced/deleted? Will the 2.0.0-beta build be used as long as the idf_component.yml file is present?

[alisitsyn]

The fact that the calls in my code will need to be adjusted is understandable, I’m talking about the esp-idf assembly itself, I wouldn’t want to change everything globally,

I think it is possible to make the code adjustable using macros for example to allow to use old or new component. I will not make a pressure to you to migrate to v2.0, it is up to you. The above description will be useful for other users later.
The original idea is to merge the v2.0.0 to master on final stage to replace v1.0..x as an obsolete.

Should the idf_component.yml with its contents be placed in the components folder of my project?

The idf_component.yml should be placed into main folder in the root of your project that corresponds to main library of the project.

and then the folder from esp-idf/components/esp-modbus can not be replaced/deleted? Will the 2.0.0-beta build be used as long as the idf_component.yml file is present?

Yes, once the manifest is set the context of components/your_modbus_comp_name can be removed. The component manager will download the latest v2 component into proj_root/managed_components and it will be compiled by idf.py as dependent component. This will work as long as the manifest file present in your project.

[pr200sd]

Okay, I'll check it out. Do I understand correctly that now I should wait for you to build v2.0.0 for esp-idf 4.4?

[alisitsyn]

Okay, I'll check it out. Do I understand correctly that now I should wait for you to build v2.0.0 for esp-idf 4.4?

I will check possible issues for v4.4 but propose you to migrate to esp-idf v5.0 at least or to latest release (better). The v4.4 is out of support period now. Is this possible for you to migrate?

[pr200sd]

Okay, I'll check it out. Do I understand correctly that now I should wait for you to build v2.0.0 for esp-idf 4.4?

I will check possible issues for v4.4 but propose you to migrate to esp-idf v5.0 at least or to latest release (better). The v4.4 is out of support period now. Is this possible for you to migrate?

I plan to in the future, but since my project is complex with support for ethernet and wi-fi with a large number of libraries added manually, I’m not sure what to quickly adapt to 5.0

[alisitsyn]

I plan to in the future, but since my project is complex with support for ethernet and wi-fi with a large number of libraries added manually, I’m not sure what to quickly adapt to 5.0

I was trying to update the v2.0.0-beta to work with esp-idf v4.4. I successfully solved some issues but there are many of them related to other component compatibility (mdns for example). I still have some other issues to solve. Unfortunately, I need to postpone this for now. Will take a look later. Once you have opportunity please try to update your project to esp-idf v5.0. This will allow to solve a lot of issues in future and the libraries will be able to be updated automatically.

[pr200sd]

I have been working with idf espressif since 2015, and every transition from one version to another is a headache for at least a week, something is always not going to work, the Python is not the same, or the paths do not match. I decided to install it on one PC, but since W7 there, none of the options ended in success. As a result, the old assembly no longer works. On the second PC with W10, I managed to install v5.3 on the 4th attempt, the standard example was assembled, when I tried to replace it with v2.0.0, errors began to appear, most likely I got confused with the placement of the files. But this is not the most important thing, v5 is radically different in everything from v4.4, and even if I check modbus_tcp_master v2.0, I will not be able to adapt my project to v5, which has been developing since 2021.

[alisitsyn]

I clearly understand your concern and understand if you would keep it as is.
The most important process is migration to esp-idf v5 from v4.4. This update move many of the components out of esp-idf and also has some other changes, this may make some regression and this process should be performed carefully. After the first update to any of v5 the further changes should be much easy. I don't expect any serious issues to migrate from esp-modbus v1 to v2. The esp-idf v4.4 to v5.0 migration guide for components may help.