Is the caffe2 model in caffe2_inference.py thread safe?

[hpanwar08]

I am using a slightly modified version of caffe2 model https://github.com/facebookresearch/detectron2/blob/master/detectron2/export/caffe2_inference.py#L17 as flask rest api. It works of single request, but when multiple request tries to do inference then some of inferences fails with the error
RuntimeError: [enforce fail at ..\caffe2\python\pybind_state.cc:210] ws->HasBlob(name). Can't find blob: bbox_nms

I am loading only 1 instance of the model.

Below is the modified caffe2 model class. And inference fails at

c2_outputs = [ws.FetchBlob(b) for b in self.net.Proto().external_output]

class DLAC2Model:
    _ids = count(0)

    def __init__(self, predict_net, init_net) -> None:
        """Initialize model"""
        self.ws_name = "__tmp_ProtobufModel_{}__".format(next(self._ids))
        self.net = core.Net(predict_net)

        with ScopedWS(self.ws_name, is_reset=True, is_cleanup=False) as ws:
            ws.RunNetOnce(init_net)
            uninitialized_external_input = []
            for blob in self.net.Proto().external_input:

                if blob not in ws.Blobs():
                    uninitialized_external_input.append(blob)
                    ws.CreateBlob(blob)
            ws.CreateNet(self.net)

        self._error_msgs = set()
        self._input_blobs = uninitialized_external_input

    def __call__(self, inputs):
        """Run model on the inputs (batched image)"""
        with ScopedWS(self.ws_name, is_reset=False, is_cleanup=False) as ws:
            for b, tensor in zip(self._input_blobs, inputs):
                ws.FeedBlob(b, tensor)

            try:
                ws.RunNet(self.net.Proto().name)
            except RuntimeError as e:
                if not str(e) in self._error_msgs:
                    self._error_msgs.add(str(e))
                    logger.warning("Encountered new RuntimeError: \n{}".format(str(e)))
                logger.warning("Catch the error and use partial results.")
                logger.info(e)

            c2_outputs = [ws.FetchBlob(b) for b in self.net.Proto().external_output]

            # Remove outputs of current run, this is necessary in order to
            # prevent fetching the result from previous run if the model fails
            # in the middle.
            for b in self.net.Proto().external_output:
                # Needs to create uninitialized blob to make the net runable.
                # This is "equivalent" to: ws.RemoveBlob(b) then ws.CreateBlob(b),
                # but there'no such API.
                ws.FeedBlob(b, f"{b}, a C++ native class of type nullptr (uninitialized).")

        return c2_outputs

TIA

[ppwwyyxx]

We have not used python inference like this. My guess is that different threads have to use different instances of DLAC2Model to be thread safe.

[hpanwar08]

Thanks for the information.
If I use detectron2's DefaultPredictor, it seems to be thread safe, it can handle multiple request without failing. Is there a way to convert the model in pure pytorch without the need for detectron2, caffe2 dependencies?

[ppwwyyxx]

caffe2 is (often) part of pytorch already. If this dependency is not accepted, there are other options: https://detectron2.readthedocs.io/en/latest/tutorials/deployment.html#deployment-with-tracing-or-scripting - which require torchvision dependency instead.

[hpanwar08]

Thanks for prompt reply. I see that torchscript option is available. Does it include pre and post processing?