eks alb 접속 장애문제와 해결과정 (보안그룹)

개발팀에서 개발서버 접근시 504 Gateway Time Out 오류가 발생하고 있다고 연락이 오고
ALB에 연결된 Argocd Dashboard가 접속이 안되는 상황 발생

 

kubectl get event -n argocd

에서는 특별한 이상한 점을 못찾았고

kubectl logs argocd-server-1234567 -n argocd

아르고 서버 파드의 로그에서 아래와 같은 에러를 확인했습니다.

 

time="2023-05-20T03:39:59Z" level=warning msg="Failed to resync revoked tokens. retrying again in 1 minute: dial tcp: lookup argocd-redis: i/o timeout"
time="2023-05-20T03:40:01Z" level=info msg="Alloc=16204 TotalAlloc=294391674 Sys=77265 NumGC=67037 Goroutines=404"
time="2023-05-22T07:00:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2023-05-22T07:00:42Z" grpc.time_ms=14.227 span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2023-05-22T07:00:42Z" span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2023-05-22T07:00:42Z" grpc.time_ms=10.996 span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.content= grpc.service=version.VersionService grpc.start_time="2023-05-22T07:00:42Z" span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2023-05-22T07:00:42Z" grpc.time_ms=11.49 span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2023-05-22T07:00:42Z" span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2023-05-22T07:00:42Z" grpc.time_ms=13.177 span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.content= grpc.service=session.SessionService grpc.start_time="2023-05-22T07:00:42Z" span.kind=server system=grpc
time="2023-05-22T07:00:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2023-05-22T07:00:42Z" grpc.time_ms=6.664 span.kind=server system=grpc
time="2023-05-22T07:00:43Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: Token is expired" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2023-05-22T07:00:43Z" grpc.time_ms=8.417 span.kind=server system=grpc
time="2023-05-22T07:00:43Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: Token is expired" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2023-05-22T07:00:43Z" grpc.time_ms=7.033 span.kind=server system=grpc

 

문제의 원인은 개발팀에서 ALB에 보안그룹(Security Group)을 등록했는데, 개발팀 IP만 인바운드 규칙에 있었기 때문이었습니다.

VPC의 Cidr, 제 자리의 IP를 추가해서 해결하였습니다.