WIP: Added es and cerebro services, rudimentary unstract-metrics lib added

[chandrasekharan-zipstack]

What

• Added es and cerebro services as part of the optional profile unstract-metrics
• Added a simple unstract-metrics library that wraps elasticsearch_dsl to add and query metrics
• Added some pytests for the new library methods

NOTE: Understood now that index creation need not happen through the regular code flow so some of the changes here would be invalid, will continue my development assuming that the index is already created and focus on exposing methods to aid easier collection of metrics for each sub-process / process type

How

• Made use of elasticsearch_dsl which is a high level lib that helps interact with es

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• No, new changes added under optional profile and not used anywhere

Env Config

• Added below envs to new library

ES_URL=http://localhost:9200
ES_CLOUD_ID=
ES_API_KEY=
COLLECT_UNSTRACT_METRICS=False

Relevant Docs

• elasticsearch_dsl

Dependencies Versions

• elasticsearch-dsl~=8.14.0 but not included in any existing services yet

Notes on Testing

• Tried setting it up locally and able to view the index and its mapping after inserting the metrics

Screenshots

1. Docker compose stats with the new services
   
2. Able to access the es instance through cerebro
   
3. Added some unit tests
   

Checklist

I have read and understood the Contribution Guidelines.

[jaseemjaskp]

@chandrasekharan-zipstack Create a confluence page for Cerebro no need to include here.

[hari-kuriakose]

@jaseemjaskp Wondering if we should mention it in README so that users can install for admin tasks if they wish? Are there any other recommended tools?

[chandrasekharan-zipstack]

@hari-kuriakose kibana maybe?

[jaseemjaskp]

@hari-kuriakose @chandrasekharan-zipstack Elasicsearch official one is Kibana. we can mention that in readme if required. https://github.com/lmenezes/cerebro is not up to date with official ES.

[chandrasekharan-zipstack]

@jaseemjaskp if cerebro is not upto date with official ES do you think we should also try kibana?

[jaseemjaskp]

@chandrasekharan-zipstack try out Kibana. but no need into include in docker-compose file

[sonarcloud]

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[github-actions]

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

[sonarcloud]

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud