[ServerSide] Move the Practitioner Details endpoint to plugins on the FHIR Gateway

[dubdabasoduba]

Context

• We currently have a Custom Practitioner Details endpoint used to fetch all the Details for a practitioner. The details are the Practitioner, Organizations, CareTeams, Groups, Locations & LocationHierarchy.
• This endpoint is added as an extension on the JPA server.
• The idea is to move this to the plugins targetting the gateway server.

Implementation

• Create the ability to define an endpoint on the Gateway server
• The endpoint will do the following
  • Accept a GET request
  • Interpret the request and build sub-requests to target the HAPI FHIR instance.
  • Aggregate the response from the HAPI sub-requests then return the aggregate as the response.

Acceptance Crirteria

• The Practitioner Details endpoints should work as before. It should return the following data points tied to the practitioner
  • Practitioner
  • Groups
  • CareTeams
  • Organizations
  • Locations
  • PractitionerRoles
  • LocationHierarchy

Linked issues

• https://github.com/opensrp/fhircore/issues?q=is%3Aopen+is%3Aissue+label%3A%22Practitioner+Details%22
• https://github.com/opensrp/fhircore/issues?q=is%3Aissue+label%3A%22Practitioner+Details%22+is%3Aclosed

[rehammuzzamil]

In order to add a new end-point on the Gateway, it would be necessary to conduct R&D to figure out the actual design implementation. Following that investigation, the appropriate LOE would be shared.
cc: @f-odhiambo @dubdabasoduba

[rehammuzzamil]

Initial R&D results on Moving Practitioner Details end-point to the Plugin module on the FHIR Gateway:
1- The code is written in such a way that once each request passes the basic criteria (valid auth token) and the checks defined in the active Access Checker, then request is bypassed to the FHIR Server/GCP Server based on the environmental variable defined.
2- There is a method HttpResponse handleRequest(ServletRequestDetails request) defined in the HttpClient in the server module that updates the URI to the FHIR Server.
3- To exempt updating the URI when the PractitionerDetails or LocationHeirarchy end-point is called, we need to add an additional check to do not update when the request path is equal to the PractitionerDetails or LocationHeirarchy.
4- This would require changes in the Server module (That is not the best approach, as we do not intend to make any OpenSRP specific changes to the core/generic functionality).
5- Apart from this, there are also changes required on the handling of response too (that part is under R&D).
6- The implementation of the API will also be altered as we will not be having a same way to register custom endpoints on the JPA Server, we also do not have a required dependency present to inject DAOs, due to which we have to explicitly call all the sub-requests via the HttpClient and then aggregate all the responses.
7- I have a doubt that this might be performance intensive.
@dubdabasoduba @f-odhiambo
What are your thoughts on this?

[rehammuzzamil]

Found another finding during the R&D:

• There is a @Hook defined inside the BearerAuthorizationInterceptor that is immediately called before the handling method is selected. This method is triggered whenever a request is made to the same base url.
• This is causing issues in the case when we are trying to access end-points on the base URL of the Gateway for the custom end-points ("/PractitionerDetails" or "/LocationHeirarchy").
• Since we have added a check not to alter the end-points/URI when the request path equals the "PractitionerDetails" or "LocationHeirarchy", therefore on the execution statement of the request, the request again reaches back to the hook starting point.
• This is causing an issue because at this time point header of Authorization is equal to an empty string, which throws an exception of "Authorization header is not a valid Bearer token!".
• The actual flow works well when we hit any other end-point, FHIR Resource, because the first hit directly hits this immediate hook handler and redirects the request to the FHIR Server, hence does not cause any issue.

Here is the reference on the Hook documentation.

I will look around for a solution for this and see if there is a way to skip certain end-points from intercepting the request.
@dubdabasoduba any feedback or thoughts is appreciated.

[dubdabasoduba]

@rehammuzzamil I have reviewed the comments above and this is how I thought the workflow would look like

1. The Gateway server receives the "/LocationHeirarchy" request.
2. It checks permissions and ignores the filtering since we do not want the data from this endpoint filtered.
3. We then modify the request to hit the "/Location" endpoint on the JPA server. The method that this is the JPA server.
   • The method that hits the JPA server is recursive and only stops the location return from the "/Location" endpoint has a null/empty partOf attribute.
   • The first "/Location" request will use the user-assigned location as the request param.
   • All the subsequent requests will use the reference on the returned location partOf attribute
4. We then built the "/LocationHierarchy" Response on the Gateway server using the Responses from the "/Location"

Pros

1. We don't have to maintain a custom version of the JPA server.
2. The "LocationHierarchy" endpoint would automatically work regardless of the server FHIR server the gateway is pointing to.

Cons

1. The recursive method pauses a performs degradation risk and would need to be tested heavily with huge locations data sets.

[dubdabasoduba]

I don't think the above description will help with the infinite loop created when the request is returned from the JPA server.

I will review this further and leave comments

[rehammuzzamil]

There is an issue encountered while returning the aggregated response on the custom end-points (Testing it with LocationHeirarchy/). The issue lies in the method that is responsible to read the content from the FHIR store response entity, replaces any FHIR store URLs by the corresponding proxy URLs, and write the modified response to the proxy response writer. It fails when it tries to read the data from the Reader instance passed as a paramter to the method, the stream gets closed until reached here.

However, the same flow works fine on the FHIR end-points (For example: /Patient). I have observed, both custom and existing end-point of FHIR follows the same flow of logic and lands on the same if/else condition blocks. I have also added logs to monitor how many times the same method is being executed to ensure that stream is accessed only once for the reading, and observed the same that the method is triggered only once in both the cases.

It fails with the following exception and stack trace:

17:53:57.607 [http-nio-8081-exec-9] ERROR c.g.f.g.BearerAuthorizationInterceptor [BearerAuthorizationInterceptor.java:333] Exception for resource LocationHeirarchy/ method GET with error: java.io.IOException: Stream closed 17:53:57.608 [http-nio-8081-exec-9] ERROR c.g.f.g.BearerAuthorizationInterceptor [ExceptionUtil.java:35] Stream closed java.io.IOException: Stream closed at java.base/java.util.zip.GZIPInputStream.ensureOpen(GZIPInputStream.java:63) at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:114) at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70) at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181) at java.base/java.io.BufferedReader.fill(BufferedReader.java:161) at java.base/java.io.BufferedReader.read(BufferedReader.java:182) at com.google.fhir.gateway.BearerAuthorizationInterceptor.replaceAndCopyResponse(BearerAuthorizationInterceptor.java:371) at com.google.fhir.gateway.BearerAuthorizationInterceptor.authorizeRequest(BearerAuthorizationInterceptor.java:331) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService$HookInvoker.invoke(BaseInterceptorService.java:602) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.doCallHooks(BaseInterceptorService.java:311) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:299) at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:1135) at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:416) at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1870) at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) 17:53:57.609 [http-nio-8081-exec-9] ERROR c.u.f.r.s.i.ExceptionHandlingInterceptor [ExceptionHandlingInterceptor.java:194] Failure during REST processing: java.lang.RuntimeException: Stream closed java.lang.RuntimeException: Stream closed at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at com.google.fhir.gateway.ExceptionUtil.throwRuntimeExceptionAndLog(ExceptionUtil.java:38) at com.google.fhir.gateway.ExceptionUtil.throwRuntimeExceptionAndLog(ExceptionUtil.java:57) at com.google.fhir.gateway.BearerAuthorizationInterceptor.authorizeRequest(BearerAuthorizationInterceptor.java:337) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService$HookInvoker.invoke(BaseInterceptorService.java:602) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.doCallHooks(BaseInterceptorService.java:311) at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:299) at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:1135) at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:416) at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1870) at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829)

Please let me know if you have any thoughts @dubdabasoduba
cc: @ndegwamartin @ekigamba @Wambere

[ndegwamartin]

@rehammuzzamil @dubdabasoduba identified the issue as being caused by invocation of this helper method:
EntityUtils.toString(entity, "UTF-8");

Internally in its implementation it closes the reader after it finishes processing

I've stripped out the invocations on my PR here #39 , we can find some other alternative way to process the response

[rehammuzzamil]

Thanks @ndegwamartin
PR looks good to me, however we need to find a way to parse HttpResponse to relevant FHIR Entities, as mentioned before the use of gson and Jackson libraries were not working. I will troubleshoot it again.

[rehammuzzamil]

Detailed Issue for reference on the custom plugins repo.
PR for reference.