Latest '1.0.2' tag won't install grafana operator on v3.11 Openshift

[paoloyx]

Hi all,

we're trying to install the operator on a production v3.11 cluster and Grafana operator won't install.
The make cluster/install goes fine, and all relevant CRDs are present

➜  application-monitoring-operator git:(master) oc project       
Using project "application-monitoring" on server "https://openshift-cluster.[DOMAIN]:8443"

➜  application-monitoring-operator git:(master)  oc get crds       
NAME                                                           CREATED AT
alertmanagers.monitoring.coreos.com                            2019-08-30T14:07:24Z
applicationmonitorings.applicationmonitoring.integreatly.org   2020-01-14T17:12:44Z
blackboxtargets.applicationmonitoring.integreatly.org          2020-01-14T17:12:46Z
bundlebindings.automationbroker.io                             2019-08-30T14:10:38Z
bundleinstances.automationbroker.io                            2019-08-30T14:10:38Z
bundles.automationbroker.io                                    2019-08-30T14:10:39Z
grafanadashboards.integreatly.org                              2020-01-14T17:12:47Z
grafanadatasources.integreatly.org                             2020-01-14T17:12:48Z
grafanas.integreatly.org                                       2020-01-14T17:12:46Z
podmonitors.monitoring.coreos.com                              2020-01-13T14:19:10Z
prometheuses.monitoring.coreos.com                             2019-08-30T14:07:24Z
prometheusrules.monitoring.coreos.com                          2019-08-30T14:07:24Z
servicemonitors.monitoring.coreos.com                          2019-08-30T14:07:24Z

➜  application-monitoring-operator git:(master) oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift-cluster..[DOMAIN]::8443
openshift v3.11.135
kubernetes v1.11.0+d4cacc0
➜  application-monitoring-operator git:(master) 

Other resources are correctly deployed

➜  application-monitoring-operator git:(master) oc get pods
NAME                                               READY     STATUS    RESTARTS   AGE
alertmanager-application-monitoring-0              3/3       Running   0          4m
application-monitoring-operator-749d9b6b54-mhj9s   1/1       Running   0          5m
prometheus-application-monitoring-0                5/5       Running   1          4m
prometheus-operator-86467cc6d8-l8cx4               1/1       Running   0          4m

We can see this error in application-monitoring-operator logs:

{"level":"info","ts":1579022331.038997,"logger":"controller_applicationmonitoring","msg":"Phase: Install GrafanaOperator"}
{"level":"info","ts":1579022331.0712292,"logger":"controller_applicationmonitoring","msg":"Error in InstallGrafanaOperator, resourceName=grafana-operator-role : err=error creating resource: roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 14b6a2a5-36f1-11ea-a98e-005056920bc0 [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services services/finalizers endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments deployments/finalizers daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]"}
{"level":"error","ts":1579022331.0713165,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"applicationmonitoring-controller","request":"application-monitoring/example-applicationmonitoring","error":"error creating resource: roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 14b6a2a5-36f1-11ea-a98e-005056920bc0 [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services services/finalizers endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments deployments/finalizers daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]","errorVerbose":"roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 14b6a2a5-36f1-11ea-a98e-005056920bc0 [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services services/finalizers endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments deployments/finalizers daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]\nerror creating resource\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).createResource\n\tapplication-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:516\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).installGrafanaOperator\n\tapplication-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:468\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).Reconcile\n\tapplication-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/dkirwan/bin/applications/go/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tapplication-monitoring-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tapplication-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tapplication-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
W0114 17:18:51.623456       1 reflector.go:302] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:204: watch of *v1.Secret ended with: The resourceVersion for the provided watch is too old.

Can anybody help? Thanks a lot

[langemar]

Hi,

this is also true for Release Version 1.0.0 in combination with OKD 3.11.

{"level":"info","ts":1579185611.6341345,"logger":"controller_applicationmonitoring","caller":"applicationmonitoring/applicationmonitoring_controller.go:465","msg":"Phase: Install GrafanaOperator"}
{"level":"info","ts":1579185611.667897,"logger":"controller_applicationmonitoring","caller":"applicationmonitoring/applicationmonitoring_controller.go:469","msg":"Error in InstallGrafanaOperator, resourceName=grafana-operator-role : err=error creating resource: roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [apps] [deployments/finalizers] [] []} {[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 86fcbb4c-e431-11e9-a1c0-0050568cd90a [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]"}
{"level":"error","ts":1579185611.6680799,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"applicationmonitoring-controller","Request":"application-monitoring/example-applicationmonitoring","error":"error creating resource: roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [apps] [deployments/finalizers] [] []} {[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 86fcbb4c-e431-11e9-a1c0-0050568cd90a [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]","errorVerbose":"roles.rbac.authorization.k8s.io \"grafana-operator-role\" is forbidden: attempt to grant extra privileges: [{[*] [apps] [deployments/finalizers] [] []} {[*] [integreatly.org] [grafanadashboards/status] [] []} {[*] [integreatly.org] [grafanadatasources/status] [] []} {[*] [integreatly.org] [grafanas/status] [] []}] user=&{system:serviceaccount:application-monitoring:application-monitoring-operator 86fcbb4c-e431-11e9-a1c0-0050568cd90a [system:serviceaccounts system:serviceaccounts:application-monitoring system:authenticated] map[]} ownerrules=[{[get] [ user.openshift.io] [users] [~] []} {[list] [ project.openshift.io] [projectrequests] [] []} {[get list] [ authorization.openshift.io] [clusterroles] [] []} {[get list watch] [rbac.authorization.k8s.io] [clusterroles] [] []} {[get list] [storage.k8s.io] [storageclasses] [] []} {[list watch] [ project.openshift.io] [projects] [] []} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/healthz /healthz/*]} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[create] [ authorization.openshift.io] [selfsubjectrulesreviews] [] []} {[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[list watch get] [servicecatalog.k8s.io] [clusterserviceclasses clusterserviceplans] [] []} {[get] [] [] [] [/healthz/ready]} {[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[create] [ build.openshift.io] [builds/docker builds/optimizeddocker] [] []} {[create] [ build.openshift.io] [builds/jenkinspipeline] [] []} {[create] [ build.openshift.io] [builds/source] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]} {[delete] [ oauth.openshift.io] [oauthaccesstokens oauthauthorizetokens] [] []} {[get] [] [] [] [/version /version/* /api /api/* /apis /apis/* /oapi /oapi/* /openapi/v2 /swaggerapi /swaggerapi/* /swagger.json /swagger-2.0.0.pb-v1 /osapi /osapi/ /.well-known /.well-known/* /]} {[impersonate] [authentication.k8s.io] [userextras/scopes.authorization.openshift.io] [] []} {[create get] [ build.openshift.io] [buildconfigs/webhooks] [] []} {[*] [] [pods services endpoints persistentvolumeclaims events configmaps secrets serviceaccounts] [] []} {[*] [apps] [deployments daemonsets replicasets statefulsets] [] []} {[*] [monitoring.coreos.com] [alertmanagers prometheuses prometheusrules servicemonitors] [] []} {[*] [applicationmonitoring.integreatly.org] [applicationmonitorings applicationmonitorings/finalizers blackboxtargets blackboxtargets/finalizers] [] []} {[*] [integreatly.org] [grafanadatasources grafanadashboards grafanas grafanas/finalizers grafanadatasources/finalizers grafanadashboards/finalizers] [] []} {[*] [route.openshift.io] [routes routes/custom-host] [] []} {[*] [rbac.authorization.k8s.io] [rolebindings roles] [] []} {[*] [extensions] [ingresses] [] []} {[create] [authentication.k8s.io] [tokenreviews] [] []} {[create] [authorization.k8s.io] [subjectaccessreviews] [] []} {[get] [ image.openshift.io] [imagestreams/layers] [] []}] ruleResolutionErrors=[]\nerror creating resource\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).createResource\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:516\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).installGrafanaOperator\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:468\ngithub.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring.(*ReconcileApplicationMonitoring).Reconcile\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/pkg/controller/applicationmonitoring/applicationmonitoring_controller.go:158\ngithub.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207\ngithub.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.10.8.linux.amd64/src/runtime/asm_amd64.s:2361","stacktrace":"github.com/integr8ly/application-monitoring-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/src/github.com/integr8ly/application-monitoring-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1579185612.6687596,"logger":"controller_applicationmonitoring","caller":"applicationmonitoring/applicationmonitoring_controller.go:115","msg":"Reconciling 

[byroncollins]

see the same issue here on an OpenShift 3.11 Cluster

[byroncollins]

I fixed the issue by including the following resources to the application-monitoring-operator role and restarting the application-monitoring-operator pod

- apiGroups:
  - integreatly.org
  attributeRestrictions: null
  resources:
    - grafanadashboards/status
    - grafanadatasources/status
    - grafanas/status
  verbs:
  - '*'

➜  deploy git:(674aca8) oc get pods
NAME                                               READY     STATUS    RESTARTS   AGE
alertmanager-application-monitoring-0              3/3       Running   0          25m
application-monitoring-operator-749d9b6b54-bkhrv   1/1       Running   0          19m
grafana-deployment-6c4cb975b8-wswdq                2/2       Running   0          18m
grafana-operator-66c44cc44c-sdnm7                  1/1       Running   0          19m
prometheus-application-monitoring-0                5/5       Running   1          25m
prometheus-operator-86467cc6d8-lr6q5               1/1       Running   0          25m
➜  deploy git:(674aca8) 

[paoloyx]

@byroncollins thanks, your fix worked for me too

...
...
- apiGroups:
  - integreatly.org
  attributeRestrictions: null
  resources:
  - grafanadashboards/status
  - grafanadatasources/status
  - grafanas/status
  verbs:
  - '*'

➜  ~ kgp -n application-monitoring                                                
NAME                                               READY   STATUS    RESTARTS   AGE
alertmanager-application-monitoring-0              3/3     Running   0          19d
application-monitoring-operator-749d9b6b54-php27   1/1     Running   0          2m
grafana-deployment-6c4cb975b8-5kxpv                2/2     Running   0          2m
grafana-operator-66c44cc44c-vmbzt                  1/1     Running   0          2m
prometheus-application-monitoring-0                5/5     Running   1          19d
prometheus-operator-86467cc6d8-stvkw               1/1     Running   0          19d