Reputation: 75
I'm getting below error in GKE logging cluster on PROD ENV
I have 2 GKE cluster running in GCP
1 service is running in dev env same is security-test that has domain name and endpoint security-test (this service name as running as container/pod in dev env) Domain name :- https://privacy.ai.dev.test.ai (register in godaddy as A record can say subdomain) endpoint :- https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec (this is 100% working as told by developer)
Below error is showing in prod env of another service name called TEST test (this service name as running as container/pod in prod env) endpoint :- https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec (this endpoint added in github repo in .ENV from there is fetching to dev env of security-test (anyway its given by developer in code they know it)
Here is the log showing in PROD env
ERROR 2024-07-09T17:45:38.167685925Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.167691406Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.176754883Z [resource.labels.containerName: test] Error updating field details with AI service: https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {
ERROR 2024-07-09T17:45:38.176931732Z [resource.labels.containerName: test] maxBodyLength: -1,
ERROR 2024-07-09T17:45:38.176936968Z [resource.labels.containerName: test] env: { FormData: [Function], Blob: null },
ERROR 2024-07-09T17:45:38.176942716Z [resource.labels.containerName: test] validateStatus: [Function: validateStatus],
ERROR 2024-07-09T17:45:38.176967489Z [resource.labels.containerName: test] headers: Object [AxiosHeaders] {
ERROR 2024-07-09T17:45:38.176972838Z [resource.labels.containerName: test] Accept: 'application/json, text/plain, */*',
ERROR 2024-07-09T17:45:38.176978217Z [resource.labels.containerName: test] 'Content-Type': 'application/json',
ERROR 2024-07-09T17:45:38.176983247Z [resource.labels.containerName: test] 'User-Agent': 'axios/1.6.7',
ERROR 2024-07-09T17:45:38.176988581Z [resource.labels.containerName: test] 'Content-Length': '1140',
ERROR 2024-07-09T17:45:38.176997428Z [resource.labels.containerName: test] 'Accept-Encoding': 'gzip, compress, deflate, br'
ERROR 2024-07-09T17:45:38.177003006Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.177008558Z [resource.labels.containerName: test] method: 'post',
ERROR 2024-07-09T17:45:38.177045615Z [resource.labels.containerName: test] request: <ref *1> ClientRequest {
ERROR 2024-07-09T17:45:38.176853594Z [resource.labels.containerName: test] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-09T17:45:38.176859304Z [resource.labels.containerName: test] config: {
ERROR 2024-07-09T17:45:38.176865215Z [resource.labels.containerName: test] transitional: {
ERROR 2024-07-09T17:45:38.176869804Z [resource.labels.containerName: test] silentJSONParsing: true,
ERROR 2024-07-09T17:45:38.176876185Z [resource.labels.containerName: test] forcedJSONParsing: true,
ERROR 2024-07-09T17:45:38.176881243Z [resource.labels.containerName: test] clarifyTimeoutError: false
ERROR 2024-07-09T17:45:38.178548164Z [resource.labels.containerName: test] transitional: [Object],
ERROR 2024-07-09T17:45:38.178554160Z [resource.labels.containerName: test] adapter: [Array],
ERROR 2024-07-09T17:45:38.178559991Z [resource.labels.containerName: test] transformRequest: [Array],
ERROR 2024-07-09T17:45:38.178566239Z [resource.labels.containerName: test] transformResponse: [Array],
ERROR 2024-07-09T17:45:38.178572641Z [resource.labels.containerName: test] timeout: 0,
ERROR 2024-07-09T17:45:38.178578788Z [resource.labels.containerName: test] xsrfCookieName: 'XSRF-TOKEN',
ERROR 2024-07-09T17:45:38.178588035Z [resource.labels.containerName: test] xsrfHeaderName: 'X-XSRF-TOKEN',
ERROR 2024-07-09T17:45:38.178594880Z [resource.labels.containerName: test] maxContentLength: -1,
ERROR 2024-07-09T17:45:38.178601129Z [resource.labels.containerName: test] maxBodyLength: -1,
ERROR 2024-07-09T17:45:38.178606872Z [resource.labels.containerName: test] env: [Object],
ERROR 2024-07-09T17:45:38.178612398Z [resource.labels.containerName: test] validateStatus: [Function: validateStatus],
ERROR 2024-07-09T17:45:38.178618445Z [resource.labels.containerName: test] headers: [Object [AxiosHeaders]],
ERROR 2024-07-09T17:45:38.178624425Z [resource.labels.containerName: test] method: 'post',
ERROR 2024-07-09T17:45:38.178630320Z [resource.labels.containerName: test] url: 'https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178639363Z [resource.labels.containerName: test] data: '{"initialEndpoints":[{"name":"business_id","location":"Request Field.business_id","label":"business_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"customer_id","location":"Request Field.customer_id","label":"customer_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"request_id","location":"Request Field.request_id","label":"request_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_access_token","location":"Request Field.vendor_access_token","label":"vendor_access_token","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_customer_id","location":"Request Field.vendor_customer_id","label":"vendor_customer_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_institution_id","location":"Request Field.vendor_institution_id","label":"vendor_institution_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_public_token","location":"Request Field.vendor_public_token","label":"vendor_public_token","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"}]}'
ERROR 2024-07-09T17:45:38.178646406Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178652737Z [resource.labels.containerName: test] request: <ref *1> ClientRequest {
ERROR 2024-07-09T17:45:38.178658793Z [resource.labels.containerName: test] _events: [Object: null prototype],
ERROR 2024-07-09T17:45:38.178664457Z [resource.labels.containerName: test] _eventsCount: 7,
ERROR 2024-07-09T17:45:38.178670377Z [resource.labels.containerName: test] _maxListeners: undefined,
ERROR 2024-07-09T17:45:38.178676401Z [resource.labels.containerName: test] outputData: [],
ERROR 2024-07-09T17:45:38.178682602Z [resource.labels.containerName: test] outputSize: 0,
ERROR 2024-07-09T17:45:38.178688339Z [resource.labels.containerName: test] writable: true,
ERROR 2024-07-09T17:45:38.178694513Z [resource.labels.containerName: test] destroyed: false,
ERROR 2024-07-09T17:45:38.178700556Z [resource.labels.containerName: test] _last: true,
ERROR 2024-07-09T17:45:38.178706554Z [resource.labels.containerName: test] chunkedEncoding: false,
ERROR 2024-07-09T17:45:38.178712375Z [resource.labels.containerName: test] shouldKeepAlive: false,
ERROR 2024-07-09T17:45:38.178718126Z [resource.labels.containerName: test] maxRequestsOnConnectionReached: false,
ERROR 2024-07-09T17:45:38.178724127Z [resource.labels.containerName: test] _defaultKeepAlive: true,
ERROR 2024-07-09T17:45:38.178730216Z [resource.labels.containerName: test] useChunkedEncodingByDefault: true,
ERROR 2024-07-09T17:45:38.178736233Z [resource.labels.containerName: test] sendDate: false,
ERROR 2024-07-09T17:45:38.178742273Z [resource.labels.containerName: test] _removedConnection: false,
ERROR 2024-07-09T17:45:38.178748093Z [resource.labels.containerName: test] _removedContLen: false,
ERROR 2024-07-09T17:45:38.178753683Z [resource.labels.containerName: test] _removedTE: false,
ERROR 2024-07-09T17:45:38.178760256Z [resource.labels.containerName: test] strictContentLength: false,
ERROR 2024-07-09T17:45:38.178766740Z [resource.labels.containerName: test] _contentLength: '1140',
ERROR 2024-07-09T17:45:38.178772550Z [resource.labels.containerName: test] _hasBody: true,
ERROR 2024-07-09T17:45:38.178779103Z [resource.labels.containerName: test] _trailer: '',
ERROR 2024-07-09T17:45:38.178785017Z [resource.labels.containerName: test] finished: true,
ERROR 2024-07-09T17:45:38.178790888Z [resource.labels.containerName: test] _headerSent: true,
ERROR 2024-07-09T17:45:38.178804555Z [resource.labels.containerName: test] _closed: false,
ERROR 2024-07-09T17:45:38.177940275Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.177946381Z [resource.labels.containerName: test] upgrade: false,
ERROR 2024-07-09T17:45:38.177952468Z [resource.labels.containerName: test] url: '',
ERROR 2024-07-09T17:45:38.177958267Z [resource.labels.containerName: test] method: null,
ERROR 2024-07-09T17:45:38.177964449Z [resource.labels.containerName: test] statusCode: 502,
ERROR 2024-07-09T17:45:38.177970059Z [resource.labels.containerName: test] statusMessage: 'Bad Gateway',
ERROR 2024-07-09T17:45:38.177976151Z [resource.labels.containerName: test] client: [TLSSocket],
ERROR 2024-07-09T17:45:38.177981960Z [resource.labels.containerName: test] _consuming: false,
ERROR 2024-07-09T17:45:38.177987765Z [resource.labels.containerName: test] _dumped: false,
ERROR 2024-07-09T17:45:38.177993847Z [resource.labels.containerName: test] req: [Circular *1],
ERROR 2024-07-09T17:45:38.178000128Z [resource.labels.containerName: test] responseUrl: 'https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178015296Z [resource.labels.containerName: test] redirects: [],
ERROR 2024-07-09T17:45:38.178021831Z [resource.labels.containerName: test] [Symbol(kCapture)]: false,
ERROR 2024-07-09T17:45:38.178027843Z [resource.labels.containerName: test] [Symbol(kHeaders)]: [Object],
ERROR 2024-07-09T17:45:38.178034523Z [resource.labels.containerName: test] [Symbol(kHeadersCount)]: 12,
ERROR 2024-07-09T17:45:38.178040646Z [resource.labels.containerName: test] [Symbol(kTrailers)]: null,
ERROR 2024-07-09T17:45:38.178050602Z [resource.labels.containerName: test] [Symbol(kTrailersCount)]: 0,
ERROR 2024-07-09T17:45:38.178056327Z [resource.labels.containerName: test] [Symbol(RequestTimeout)]: undefined
ERROR 2024-07-09T17:45:38.178062331Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178068224Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.178810323Z [resource.labels.containerName: test] socket: [TLSSocket],
ERROR 2024-07-09T17:45:38.178816800Z [resource.labels.containerName: test] _header: 'POST /api-analyzer/fetch-api-spec HTTP/1.1\r\n' +
ERROR 2024-07-09T17:45:38.178822068Z [resource.labels.containerName: test] 'Accept: application/json, text/plain, */*\r\n' +
ERROR 2024-07-09T17:45:38.178827145Z [resource.labels.containerName: test] 'Content-Type: application/json\r\n' +
ERROR 2024-07-09T17:45:38.178833020Z [resource.labels.containerName: test] 'User-Agent: axios/1.6.7\r\n' +
ERROR 2024-07-09T17:45:38.178838987Z [resource.labels.containerName: test] 'Content-Length: 1140\r\n' +
ERROR 2024-07-09T17:45:38.178844833Z [resource.labels.containerName: test] 'Accept-Encoding: gzip, compress, deflate, br\r\n' +
ERROR 2024-07-09T17:45:38.178850525Z [resource.labels.containerName: test] 'Host: privacy.ai.dev.test.ai\r\n' +
ERROR 2024-07-09T17:45:38.178856171Z [resource.labels.containerName: test] 'Connection: close\r\n' +
ERROR 2024-07-09T17:45:38.178861452Z [resource.labels.containerName: test] '\r\n',
ERROR 2024-07-09T17:45:38.178867493Z [resource.labels.containerName: test] _keepAliveTimeout: 0,
ERROR 2024-07-09T17:45:38.178873614Z [resource.labels.containerName: test] _onPendingData: [Function: nop],
ERROR 2024-07-09T17:45:38.178879514Z [resource.labels.containerName: test] agent: [Agent],
ERROR 2024-07-09T17:45:38.178885497Z [resource.labels.containerName: test] socketPath: undefined,
ERROR 2024-07-09T17:45:38.178891154Z [resource.labels.containerName: test] method: 'POST',
ERROR 2024-07-09T17:45:38.178897283Z [resource.labels.containerName: test] maxHeaderSize: undefined,
ERROR 2024-07-09T17:45:38.178903044Z [resource.labels.containerName: test] insecureHTTPParser: undefined,
ERROR 2024-07-09T17:45:38.178909214Z [resource.labels.containerName: test] path: '/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178915164Z [resource.labels.containerName: test] _ended: true,
ERROR 2024-07-09T17:45:38.178920714Z [resource.labels.containerName: test] res: [IncomingMessage],
ERROR 2024-07-09T17:45:38.178926597Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.178932395Z [resource.labels.containerName: test] timeoutCb: null,
ERROR 2024-07-09T17:45:38.178938568Z [resource.labels.containerName: test] upgradeOrConnect: false,
ERROR 2024-07-09T17:45:38.178971909Z [resource.labels.containerName: test] parser: null,
ERROR 2024-07-09T17:45:38.178979560Z [resource.labels.containerName: test] maxHeadersCount: null,
ERROR 2024-07-09T17:45:38.178985241Z [resource.labels.containerName: test] reusedSocket: false,
ERROR 2024-07-09T17:45:38.178991521Z [resource.labels.containerName: test] host: 'privacy.ai.dev.test.ai',
ERROR 2024-07-09T17:45:38.178449005Z [resource.labels.containerName: test] [Symbol(kUniqueHeaders)]: null
ERROR 2024-07-09T17:45:38.178455189Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178461168Z [resource.labels.containerName: test] response: {
ERROR 2024-07-09T17:45:38.178467312Z [resource.labels.containerName: test] status: 502,
ERROR 2024-07-09T17:45:38.178473740Z [resource.labels.containerName: test] statusText: 'Bad Gateway',
ERROR 2024-07-09T17:45:38.178479670Z [resource.labels.containerName: test] headers: Object [AxiosHeaders] {
ERROR 2024-07-09T17:45:38.178486053Z [resource.labels.containerName: test] 'content-type': 'text/html; charset=UTF-8',
ERROR 2024-07-09T17:45:38.178492197Z [resource.labels.containerName: test] 'referrer-policy': 'no-referrer',
ERROR 2024-07-09T17:45:38.178498332Z [resource.labels.containerName: test] 'content-length': '332',
ERROR 2024-07-09T17:45:38.178504108Z [resource.labels.containerName: test] date: 'Tue, 09 Jul 2024 17:45:38 GMT',
ERROR 2024-07-09T17:45:38.178511Z [resource.labels.containerName: test] 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
ERROR 2024-07-09T17:45:38.178529028Z [resource.labels.containerName: test] connection: 'close'
ERROR 2024-07-09T17:45:38.178996766Z [resource.labels.containerName: test] protocol: 'https:',
ERROR 2024-07-09T17:45:38.179001897Z [resource.labels.containerName: test] _redirectable: [Writable],
ERROR 2024-07-09T17:45:38.179006948Z [resource.labels.containerName: test] [Symbol(kCapture)]: false,
ERROR 2024-07-09T17:45:38.179012617Z [resource.labels.containerName: test] [Symbol(kBytesWritten)]: 0,
ERROR 2024-07-09T17:45:38.179018157Z [resource.labels.containerName: test] [Symbol(kEndCalled)]: true,
ERROR 2024-07-09T17:45:38.179023423Z [resource.labels.containerName: test] [Symbol(kNeedDrain)]: false,
ERROR 2024-07-09T17:45:38.179028620Z [resource.labels.containerName: test] [Symbol(corked)]: 0,
ERROR 2024-07-09T17:45:38.179034587Z [resource.labels.containerName: test] [Symbol(kOutHeaders)]: [Object: null prototype],
ERROR 2024-07-09T17:45:38.179040173Z [resource.labels.containerName: test] [Symbol(kUniqueHeaders)]: null
ERROR 2024-07-09T17:45:38.179046285Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.179052728Z [resource.labels.containerName: test] data: '\n' +
ERROR 2024-07-09T17:45:38.179058145Z [resource.labels.containerName: test] '<html><head>\n' +
ERROR 2024-07-09T17:45:38.179063778Z [resource.labels.containerName: test] '<meta http-equiv="content-type" content="text/html;charset=utf-8">\n' +
ERROR 2024-07-09T17:45:38.179073355Z [resource.labels.containerName: test] '<title>502 Server Error</title>\n' +
ERROR 2024-07-09T17:45:38.179079448Z [resource.labels.containerName: test] '</head>\n' +
ERROR 2024-07-09T17:45:38.179085177Z [resource.labels.containerName: test] '<body text=#000000 bgcolor=#ffffff>\n' +
ERROR 2024-07-09T17:45:38.179090962Z [resource.labels.containerName: test] '<h1>Error: Server Error</h1>\n' +
ERROR 2024-07-09T17:45:38.179107509Z [resource.labels.containerName: test] '<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n' +
ERROR 2024-07-09T17:45:38.179113834Z [resource.labels.containerName: test] '<h2></h2>\n' +
ERROR 2024-07-09T17:45:38.179119609Z [resource.labels.containerName: test] '</body></html>\n'
ERROR 2024-07-09T17:45:38.179125141Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.179130953Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.194095673Z [resource.labels.containerName: test] Error updating field details with AI service: https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Here is the YAML of security-test
apiVersion: apps/v1
kind: Deployment
metadata:
name: security-ai
namespace: test
spec:
replicas: 2
selector:
matchLabels:
app: security-test
template:
metadata:
labels:
app: security-test
spec:
containers:
- name: security-test
image: australia-southeast1-docker.pkg.dev/test-dev/security-test/security-test:2024-07-04-0856
imagePullPolicy: Always
ports:
- containerPort: 3000
readinessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
livenessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 1
startupProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 60
failureThreshold: 0
periodSeconds: 10
timeoutSeconds: 1
resources:
requests: # Minimum amount of resources requested
cpu: 500m
memory: 1Gi
limits: # Maximum amount of resources requested
cpu: 1000m
memory: 2Gi
---
apiVersion: v1
kind: Service
metadata:
name: security-test
namespace: test
labels:
app: security-test
spec:
type: ClusterIP
selector:
app: security-test
ports:
- port: 3000
targetPort: 3000
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-security-test
namespace: test
annotations:
kubernetes.io/ingress.global-static-ip-name: ingress-security-test
networking.gke.io/managed-certificates: managed-cert-security-test
ingressClassName: "gce"
spec:
rules:
- host: "privacy.ai.dev.test.ai"
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: security-test
port:
number: 3000
---
# apiVersion: networking.k8s.io/v1
# kind: Ingress
# metadata:
# name: ingress-security-test
# namespace: test
# annotations:
# kubernetes.io/ingress.global-static-ip-name: ingress-security-test
# networking.gke.io/managed-certificates: managed-cert-security-test
# ingressClassName: "gce"
# spec:
# defaultBackend:
# service:
# name: security-test
# port:
# number: 3000
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: managed-cert-security-test
namespace: test
spec:
domains:
- privacy.ai.dev.test.ai
What im missing in ingress or deployment file please do let me know its high priority issue im facing since 4days
Even sometimes i get liveliness and readlines fail due to client.timeout issue
i trying many times changing ingress rule, loadbalancer setting but nothing work to solve 502 bad gateway
just now i describe pods i got readiness and liveness probe fail
Limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 1Gi
Requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 1Gi
Liveness: http-get http://:3000/ delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:3000/ delay=30s timeout=1s period=10s #success=1 #failure=3
Startup: http-get http://:3000/ delay=60s timeout=1s period=10s #success=1 #failure=3
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 28m (x10 over 13h) kubelet Readiness probe failed: Get "http://10.91.128.102:3000/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 28m (x9 over 13h) kubelet Liveness probe failed: Get "http://10.91.128.102:3000/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Upvotes: 0
Views: 65