Microclimate build deployment fails on non-default namespace

[fctoibm]

Issue type: Bug/Docs/Enhancement/Question/UI
Reported in:
Internal issue:
Fixed in:
Description of issue:
Workaround (if possible):

Status updates:

[johnmcollier]

@fctoibm Can you post any messages from the failure? Is it the Jenkins pipeline build that fails?

[fctoibm]

When I select a different namespace to deploy the build, it fails. I have been talking to many folks on the slack channel, but they have pointed me to opening a GitHub issue.

here is the slack conversation
Harmeet Singh [7 days ago]
Thanks for the reply, do you have any documentation or pointer on how to update the pipeline configuration?

Harmeet Singh [7 days ago]
I followed you guideline but still not successful

[harsingh@fiscicpclient ~]$ docker login https://stagecluster.icp:8500
Username: harsingh2
Password:
Login Succeeded
created secret for the icp4d namespace
[harsingh@fiscicpclient ~]$ kubectl create secret docker-registry microclimate-pipeline-secret \

--docker-server=stagecluster.icp:8500
--docker-username=harsingh2
--docker-password=passw0rd$HS
--docker-email=harsingh@us.ibm.com
--namespace=icp4d
secret/microclimate-pipeline-secret created

Step 2 patched it
kubectl patch sa default -n icp4d --type=json -p=“[{\“op\“:\“add\“,\“path\“:\“/imagePullSecrets/0\“,\“value\“:{\“name\“: \“microclimate-pipeline-secret\“}}]”
serviceaccount/default patched
[harsingh@fiscicpclient ~]$ kubectl describe serviceaccount default --namespace icp4d
Name: default
Namespace: icp4d
Labels:
Annotations:
Image pull secrets: microclimate-pipeline-secret
dockerregistrydevlogin
Mountable secrets: default-token-x75dq
Tokens: default-token-x75dq
Events:

Step 3 created helm secert for ICP4D namespace
kubectl create secret generic microclimate-helm-secret --from-file=cert.pem=$HELM_HOME/cert.pem --from-file=ca.pem=$HELM_HOME/ca.pem --from-file=key.pem=$HELM_HOME/key.pem
secret/microclimate-helm-secret created

Step 4 gave Helm repository and namespace to access to the user/team
Pasted image at 2019-05-08, 2:08 PM

Harmeet Singh [7 days ago]
but its still failed to deploy the build to different namespace

Harmeet Singh [7 days ago]
Pasted image at 2019-05-08, 2:10 PM

Harmeet Singh [7 days ago]
i was able to fix the issue by combing through logs

Harmeet Singh [7 days ago]
time=“2019-05-08T19:41:10Z” level=info msg=“Release “ipc4d-f278f795-a700-483b-a324-cb121ea1cd67” does not exist. Installing it now.
Error: release ipc4d-f278f795-a700-483b-a324-cb121ea1cd67 failed: Internal error occurred: admission webhook “trust.hooks.securityenforcement.admission.cloud.ibm.com” denied the request:
Deny “stagecluster.icp:8500/microclimate-install-deployments/fromlocal:b4a672c”, no matching repositories in the ImagePolicies
”
time=“2019-05-08T19:41:10Z” level=error msg=“Error executing upgrade --install --wait --namespace icp4d --values chart-036184813/pipeline.yaml ipc4d-f278f795-a700-483b-a324-cb121ea1cd67 chart-036184813/chart/fromlocal: exit status 1"
time=“2019-05-08T19:41:10Z” level=debug msg=“Status update must be sent”

Harmeet Singh [7 days ago]
just included the default cluster in my imagepolicy which was part of the namespace scope

Harmeet Singh [7 days ago]
well maybe I am not able to fully crack the problem yet, I am getting closer but not sure why no pod exist in Helm deployment

Harmeet Singh [7 days ago]
Pasted image at 2019-05-08, 4:30 PM

Harmeet Singh [7 days ago]
Any input why i am unable to pull image during microclimate build deployment

Harmeet Singh [7 days ago]
Failed to pull image “stagecluster.icp:8500/microclimate-install-deployments/testicp4d:e4bcac0”: rpc error: code = Unknown desc = Error response from daemon: Get https://stagecluster.icp:8500/v2/microclimate-install-deployments/testicp4d/manifests/e4bcac0: unauthorized: authentication required

Arjun [6 days ago]
@harmeet Singh can you docker do docker login as user harsingh2 and try to pull the image stagecluster.icp:8500/microclimate-install-deployments/testicp4d:e4bcac0

Arjun [6 days ago]
i guess the user harsingh2 doesnt have access to the namespace microclimate-install-deployments

Arjun [6 days ago]
you may need to add the namespace resource to harsingh2

Harmeet Singh [6 days ago]
GM Arjun

Arjun [6 days ago]
halo

Arjun [6 days ago]
gm

Harmeet Singh [6 days ago]
Yesterday, I came to conclusion that the the new namespace ICP4D required following POD security ibm-anyuid-hostpath-psp, ibm-restricted-psp in order to get this far.
My next question is why Jenkins pipeline upload the new docker/helm content to

https://stagecluster.icp:8500/v2/ microclimate-install-deployments /testicp4d/manifests/e4bcac0

Arjun [6 days ago]
after adding psp it worked ?

Harmeet Singh [6 days ago]
Nope, i got further and now pod is unable to pull the image.

Harmeet Singh [6 days ago]
I don’t like the fact in jenkins build log file states
“[Pipeline] echo
Only one child directory found, setting realChartFolder to: chart/testicp4d
[Pipeline] sh

• echo image:
  repository: stagecluster.icp:8500/microclimate-install-deployments/testicp4d
  tag: “e4bcac0”
  [Pipeline] fileExists
  [Pipeline] sh
• echo commitID=e4bcac0\nfullCommit=e4bcac0e9b157a6b0c3e4fa7deb13b3d8fb80f48\ncommitMessage=Microclimatizing testicp4d

Which means MC only deploy to default pipeline and changing the pipeline will make global change for all the users.

any ways even giving the harsing2 access to NAMESPACE microclimate-install-deployments
I am still seeing following issue
Screen Shot 2019-05-09 at 8.49.49 AM.png

Arjun [6 days ago]
i dont think psp is related here

Arjun [6 days ago]
can you try docker login and try manual pull

Harmeet Singh [6 days ago]
if I didn’t update the psp, then pod was not even getting created

Harmeet Singh [6 days ago]
now least pod is getting creating, that was my finding, again I will try again on new machine once I have it working on this one.

Harmeet Singh [6 days ago]
sure let me try docker

Harmeet Singh [6 days ago]
[harsingh@fiscicpclient ~]$ docker login stagecluster.icp:8500
Username (harsingh2):
Password:
Login Succeeded
[harsingh@fiscicpclient ~]$ docker pull stagecluster.icp:8500/v2/microclimate-install-deployments/testicp4d/manifests/e4bcac0clear
Using default tag: latest
Error response from daemon: Get https://stagecluster.icp:8500/v2/v2/microclimate-install-deployments/testicp4d/manifests/e4bcac0clear/manifests/latest: unauthorized: authentication required
[harsingh@fiscicpclient ~]$

Harmeet Singh [6 days ago]
unable to pull the image

Harmeet Singh [6 days ago]
never mind i had a typo

so i am able to pull the image manually using harsingh

Harmeet Singh [6 days ago]
[harsingh@fiscicpclient ~]$ docker login stagecluster.icp:8500
Username (admin): harsingh2
Password:
Login Succeeded
[harsingh@fiscicpclient ~]$ docker pull stagecluster.icp:8500/microclimate-install-deployments/testicp4d:e4bcac0
e4bcac0: Pulling from microclimate-install-deployments/testicp4d
c5e155d5a1d1: Download complete
221d80d00ae9: Download complete
4250b3117dca: Download complete
3b7ca19181b2: Downloading [=====================================> ] 37.08MB/50.07MB
425d7b2a5bcc: Downloading [=========> ] 40.55MB/215.1MB
69df12c70287: Download complete
2a68245de447: Waiting
4f61e9705839: Waiting
0cf6437a8109: Waiting
14fc56622baa: Waiting
944b0f90b2ce: Waiting
a9d321f11184: Waiting
f344ca4fc393: Waiting

Arjun [6 days ago]
so your user have access to the namespace..I too tryed deploying to various namespace and it worked after patching the secret..seems like some auth pod issues .. @jagraj can you help here

Arjun [6 days ago]
https://github.com/microclimate-dev2ops/microclimate-dev2ops.github.io/issues

Arjun [6 days ago]
you can raise an issue there and microclimate team might help

Harmeet Singh [6 days ago]
[harsingh@fiscicpclient privateReg]$ cloudctl login -a https://10.75.30.171:8443 --skip-ssl-validation

Username> harsingh2

Password>
Authenticating...
OK

Targeted account stagecluster Account (id-stagecluster-account)

Select a namespace:

1. icp4d
2. microclimate-install-deployments
   Enter a number> 1
   Targeted namespace icp4d

Configuring kubectl ...
Property “clusters.stagecluster” unset.
Property “users.stagecluster-user” unset.
Property “contexts.stagecluster-context” unset.
Cluster “stagecluster” set.
User “stagecluster-user” set.
Context “stagecluster-context” created.
Switched to context “stagecluster-context”.
OK

Configuring helm: /home/harsingh/.helm
OK
[harsingh@fiscicpclient privateReg]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
testicp4d-deployment-66d9fdc8bc-bjm29 0/1 ImagePullBackOff 0 7m9s
[harsingh@fiscicpclient privateReg]$ kubectl logs testicp4d-deployment-66d9fdc8bc-bjm29
Error from server (BadRequest): container “testicp4d” in pod “testicp4d-deployment-66d9fdc8bc-bjm29” is waiting to start: trying and failing to pull image
[harsingh@fiscicpclient privateReg]$ kubectl get secrets
NAME TYPE DATA AGE
default-token-vknvj kubernetes.io/service-account-token 3 15h
microclimate-helm-secret Opaque 3 15h
microclimate-pipeline-secret kubernetes.io/dockerconfigjson 1 15h
microclimate-registry-secret kubernetes.io/dockerconfigjson 1 14h

Harmeet Singh [6 days ago]
i even have secret like you suggested it yesterday.

Harmeet Singh [6 days ago]
NAME TYPE DATA AGE
default-token-8hxnq kubernetes.io/service-account-token 3 9d
devops-sa-token-6jvkb kubernetes.io/service-account-token 3 6d18h
ibm-microclimate-mc-tls-secret kubernetes.io/tls 2 9d
ibm-microclimate-tls-secret kubernetes.io/tls 2 9d
mc-hs-ibm-microclimate Opaque 3 5d21h
mc-hs-jenkins Opaque 1 6d18h
mc-hs-mc-tls-secret kubernetes.io/tls 2 6d18h
mc-hs-tls-secret kubernetes.io/tls 2 6d18h
micro-sa-token-sl6hx kubernetes.io/service-account-token 3 6d18h
microclimate-helm-secret Opaque 3 9d
microclimate-registry-secret kubernetes.io/dockerconfigjson 1 5d18h
kubernetes.io/dockerconfigjson 1 5d21h
ufpxctcut5d3tmxqagetleyu5hql1vf0 kubernetes.io/basic-auth 4 6d
w0lyatn12lliamiytcfgegqvzu4h3kdz kubernetes.io/basic-auth 4 5d21h
[root@fiscicpclient harsingh]#

Harmeet Singh [6 days ago]
what is the purpose of
sa-microclimate-install-deployments

Harmeet Singh [6 days ago]
I think the problem is where this image resides
[root@fiscicpclient harsingh]# kubectl get image testicp4d -o yaml
apiVersion: icp.ibm.com/v1
kind: Image
metadata:
creationTimestamp: 2019-05-08T20:49:02Z
generation: 1
name: testicp4d
namespace: microclimate-install-deployments
resourceVersion: “6369560”
selfLink: /apis/icp.ibm.com/v1/namespaces/microclimate-install-deployments/images/testicp4d
uid: b5ef21dd-71d2-11e9-a551-005056ad5623
spec:
scope: namespace

icp4d namespace will never able to to this image?

Harmeet Singh [6 days ago]
my question, if you select namespace option during the deployment, why does it defaults to microclimate-install-deployments.
seems like a bug @jagraj

[rajivnathan]

There's a lot going on in the previous comment. It may help to specify the error you are currently seeing that is blocking you.

I'm not too familiar with the pipeline setup but if the current issue is the image fails to pull then it's likely a problem with the secret itself or the service account you patched. I noticed above you patched the default service account. You may want to check whether that is the correct service account for the pipeline deployment. One way to verify which service account is being used is to list the deployment for the devops pod using kubectl describe deploy <devops pod name> and the service account it is associated with should be listed. That will be the service account you need to patch.

If it is default then you may want to ensure the secret itself is correct. You can try recreating it and patching it again.

[arjunsbabu]

@fctoibm I too have a similar setup. I have two infra one is dev and other is prod.
In dev i used admin ID to create helm secret and registry secret
ie i have followed the steps in dev

kubectl create secret docker-registry microclimate-registry-secret --docker-server=watson.dst.ibm.com:8500 --docker-username=admin --docker-password=Gpasssdfa@ --docker-email=admin@test.local -n microclimate
cloudctl login -u admin https://watson.dst.ibm.com:8500 -c id-mycluster-account -n microclimate
kubectl create secret generic microclimate-helm-secret --from-file=cert.pem=$HELM_HOME/cert.pem --from-file=ca.pem=$HELM_HOME/ca.pem --from-file=key.pem=$HELM_HOME/key.pem -n microclimate
kubectl create secret docker-registry microclimate-pipeline-secret --docker-server=watson.dst.ibm.com:8500 --docker-username=admin --docker-password=Gpasssdfa@ --docker-email=admin@test.local --namespace=namespace2
kubectl patch serviceaccount default --namespace namespace2 -p "{\"imagePullSecrets\": [{\"name\": \"microclimate-pipeline-secret\"}]}"

````

The above steps are enough since i used admin id and i can deploy into non default namespace namespace2

[arjunsbabu]

In Prod i used a different way

kubectl create secret docker-registry microclimate-registry-secret --docker-server=icp-cluster.test.local:8500 --docker-username=Icpldap --docker-password=ASCnXze#v --docker-email=icpldap@test.local -n devops
cloudctl login -u Icpldap -a https://icp-cluster.test.local:8443  -n devops  --skip-ssl-validation

kubectl create secret generic microclimate-helm-secret --from-file=cert.pem=$HELM_HOME/cert.pem --from-file=ca.pem=$HELM_HOME/ca.pem --from-file=key.pem=$HELM_HOME/key.pem -n devops

kubectl create secret docker-registry microclimate-pipeline-secret --docker-server=icp-cluster.test.local:8500 --docker-username=Icpldap  --docker-password=ASCenX71 --docker-email=icpldap@test.local --namespace=namespace2
kubectl patch serviceaccount default --namespace namespace2 	 -p "{\"imagePullSecrets\": [{\"name\": \"microclimate-pipeline-secret\"}]}"

microclimate is the namespace used for MC deployment in dev and devops is the namespace used for prod. I added Icpldap user as an administrator to devops and namespace2 namespace. So that Icpldap can pull image from devops andm deploy on namespace2
With these steps i am able to deploy. You may be need to share the build log and log of devops pod. And i dont think pushing the image to microclimate hosted namespace is a bug. Its the way of the work.
And we are patching the default service account in namespace2 to pull image from devops namespace