DevOps Live Interview | DevOps | Cloud | DevSecOps | SRE

[00:00:04]Hi Jadeja.

[00:00:05]>> Yeah, Akash.

[00:00:08]>> So, let me open your resume once. Just give me a sec.

[00:00:13]Okay.

[00:00:15]So, till then till then please start by introducing yourself.

[00:00:20]>> Yeah. So, my name is Jadeja Rajpalsinh and I'm a DevOps engineer with 7 year plus experience in designing, building, operating scalable and highly available cloud platform.

[00:00:29]My core experience lies in implementing the key pillars of DevOps that is automation, continuous integration delivery, infrastructure as a code, monitoring, and security.

[00:00:38]Throughout my journey, I have worked on multiple cloud platform, but most extensively in the side of AWS and a bit of GCP and Azure. I have worked on designing cloud native platform, Kubernetes based environment, and end-to-end CCD ecosystem for enterprise application.

[00:00:54]And if I talk about on the side of governance, then I have implemented secure multi-cloud architecture, centralized access management solution aligned with compliance standard like GDPR and HIPAA. I also been involved into the client discussion and pre-sales activity where I need to translate the business requirement into the technical solutions.

[00:01:14]Along with the hands-on engineering, I have certification like AWS certified solution architect, Red Hat certified system administration, and Red Hat Ansible automation exam. So, I think yeah, this is just a brief about me.

[00:01:27]>> Okay.

[00:01:28]So, currently in your organization, are you working independently or as a part of DevOps team?

[00:01:34]>> Um like we are a service-based company, so right now I'm working on two projects simultaneously. In one project, I'm working as an individual contributor where I independently manage the DevOps activity. And in another project, I am leading a team of three DevOps engineer where I'm responsible for task planning and technical guidance.

[00:01:53]>> Okay. Okay.

[00:01:55]So, let's start with the questions.

[00:01:58]Suppose you need to host a new application that is expected to handle around 5 million concurrent users.

[00:02:06]What How would you design that? And what all services and tools would you use?

[00:02:12]>> Okay, so 5 million concurrent users. So, before designing the architecture, I would first understand the application requirement, whether it will be stateless, stateful. We need to also understand the expected traffic pattern, the database requirement, compliance, and latency expectation. And based on that, I would design a highly available and secure platform. So, like if I talk about the upper layer architecture, then it would be a it would have a route 53 for DNS DNS, then we we would have CloudFront for CDN and caching. We could use a web and shield for security.

[00:02:44]Then to distribute the traffic, we can use load balancer. And if you're going with the containerization approach, then either we can go ahead with ECS or EKS.

[00:02:51]I would choose ECS if it like if the architecture is not that much complex, and ECS has lower operational overhead.

[00:02:59]But if you want some sort of advanced feature of Kubernetes like GitHubs, some sort of custom networking, or advanced deployment strategy, then in that case, I would use EKS one. And also at the same time, we can use Argo CD for the deployment.

[00:03:14]And for the backend, we can go ahead with Aurora and DynamoDB for the database, Redis for the caching, S3 for the static content, SQS Kafka for asynchronous processing. And there are multiple tools for the monitoring and CCD that we can set it up over there.

[00:03:29]So, it usually depends on the budget.

[00:03:31]Like if the client can't like can't pay it, then they can't have it. So, we need to understand the cost factor as well there over there. So, but we can go ahead with some sort of open source tools like Prometheus, Grafana, CloudWatch. We can go ahead with some sort of CCD tools like Jenkins, GitHub Action, GitLab CI. Yeah, so I think this would be some of my checklist that I would follow.

[00:03:52]>> Okay. Cool. Suppose you successfully build this application and your application is already in the running state, right? And you what you notice is sudden traffic spikes up to 10x, right?

[00:04:06]How would you determine whether it's the genuine traffic or a DDoS attack?

[00:04:11]>> Okay. So, as a debugging point of view, firstly I would analyze the access log to look at the source IP, request pattern, geolocation, and the request path. If the traffic is coming from diverse real users with normal behavior and valid application request, it's likely a genuine traffic spike. So, over there I would simply scale up the infrastructure using the auto scaling. But, if I see a large number of repeated requests from a suspicious IP or maybe some unusual user agent, and also at the same time if they are requesting at or targeting a single endpoint, so I would suspect a DDoS attack over there. So, to mitigate it, I would use a AWS WAF, AWS Shield, or like we can use some sort of rate limiting functions over there, and it will block the malicious IP. Like, also at the same time, we need to continuously monitor the application to ensure the legitimate users are not getting impacted over there.

[00:05:01]>> Makes sense, yeah.

[00:05:03]Um okay. So, suppose as you mentioned auto scaling in your answer, that reminds me of Kubernetes. So, have you worked on Kubernetes, yes?

[00:05:13]>> Yeah, yeah. Actually, we worked on the Kubernetes side.

[00:05:16]>> Okay. That's cool. So, suppose you have three name spaces in a Kubernetes in the same Kubernetes cluster, how would you prevent the dev name space from communicating with the prod name space?

[00:05:28]>> Okay. So, like in our case, ideally we create a different cluster for like different cluster for different environment.

[00:05:36]But, let's say if we are having all the environment into a single cluster, then over there we could use the Kubernetes network policy to restrict the cross name space communication. By default, like I would deny all the ingress and egress traffic, then explicitly we can allow only the required communication within the namespace that is required over there.

[00:05:56]>> Got it. Got it.

[00:05:57]So, have you created any, you know, deployments or services or ingress out there from the YAML file? Have you ever >> like we are creating it from the scratch even though we are working into the helm chart format, so yes.

[00:06:10]>> Okay. So, just let me know what happens internally when you run a kubectl apply hyphen f the supposed deployment.yaml.

[00:06:18]What happens internally?

[00:06:20]>> So, whenever we apply the kubectl command, so basically kubectl reads the YAML for manifest, then it sends the request to the kube API server, and then API server validates the request, some sort of authentication and authorization, and then it store the data into the etcd.

[00:06:36]Then the scheduler assign the port to a suitable worker node, then the kubelet on that specific node receives the instructions and ask the container runtime to create the container.

[00:06:45]Finally, the port start running and the container, like, sorry, the controller continuously ensure that the desired state is maintained, like, whatever the replica that you are mentioning inside the deployment file.

[00:06:54]So, usually this is how the flow works.

[00:06:57]>> Okay. So, based on your previous answer, um I assume that you're pretty much well versed with the Kubernetes architecture, no? Like, you have Have you heard about etcd on the master plane?

[00:07:12]>> Yes, yes. So, basically it's a database where it store the data of everything.

[00:07:16]>> Okay. So, like, answer this, like, why did Kubernetes choose etcd instead of any other database? They could have chosen any database, but why etcd?

[00:07:29]>> Um like, to be honest, like, I haven't looked into that depth, but my understanding is that etcd is built for distributed system, and it provides a consistency and leader election, specially the leader election.

[00:07:41]So, which are important for Kubernetes, so that is what my current understanding, but I would uh need to check into a bit more. Yeah.

[00:07:49]>> So, yeah, fine, fine, fine. Makes sense.

[00:07:51]So, >> [clears throat] >> your your you must have hosted, you know, microservice uh applications or the websites out there, right?

[00:07:59]>> Mhm.

[00:08:00]>> So, what do you think like how does the request reach your pod when a user accesses your website?

[00:08:07]>> So, like whenever user sends a request to our website, the request first reaches our reaches to the DNS. So, it can be route 53, it can be GoDaddy, it can be anything, and then it resolves the domain from there. And then it reaches like if you are using the application load balancer, then it will firstly go to the application load balancer in our case.

[00:08:24]Then it forwards the request to the Kubernetes ingress controller. And the ingress controller the ingress routes the request to the appropriate Kubernetes service, and then the service forward it it to the one of the healthy pod using the kube proxy.

[00:08:38]>> Okay. Okay.

[00:08:40]Uh let me just allow me a sec. Let me have a look at your resume.

[00:08:45]Okay. So, in your achievement section, you mentioned that you migrated 500 TBs of S3 bucket of data of S3 bucket from one AWS account to another.

[00:08:58]Mhm. Um like from the point of accomplishments, isn't that straightforward? You you could have just used AWS data sync, right?

[00:09:08]Can you just elaborate on that?

[00:09:10]>> So, even though to perform that migration activity, we used the AWS data sync. So, the migration part was never a bigger challenge for us. The real challenge was to how to very re-verify that all the data has been migrated successfully without missing any object because since the data was into the 500 TB of data, so it was very difficult for us to re-verify the objects whether everything has been successfully migrated or not. So, to solve that specific issue, I I the some more services of AWS like I used the S3 inventory. It basically generates a CSV format containing all the objects in a in the bucket. And I then loaded that inventory into a DynamoDB table and like then simply compared the source and the destination inventory to identify any missing object with the help of that specific DynamoDB database.

[00:09:55]>> Okay. Okay. Makes sense.

[00:09:58]Um okay. So, how like uh in your 6-7 years of DevOps career, I hope that you must have built a lot of pipelines out there, right?

[00:10:09]>> Correct.

[00:10:10]>> Uh so, can you just walk me through the best CI/CD pipeline that you built?

[00:10:16]>> So, actually we I worked on multiple pipelines with different types of tools and different different type of technology, but in my recent project that currently where I'm working on, so let me walk you through over there. So, basically the approach would be almost similar for almost all the pipeline all the tools. So, usually whenever a developer pushes the code, the pipeline will automatically get triggered. It will firstly pull the latest code from the Git. It will run some unit test cases. Then it will perform some sort of code quality analysis using the SonarQube. Then while creating the image, we also use some sort of vulnerability scanning tool like Trivy and everything. Then it pushes the image to the ECR.

[00:10:50]Then for the CD part, we are having different repositories. So, over there we update the image tag in the GitHub repository.

[00:10:57]Then Argo CD automatically detect the change and it synchronize it with the our Kubernetes cluster.

[00:11:03]And then the application was uh in our in our current project, the application was deployed using the rolling update strategy. And post deployment, health checks were performed before making before marking the deployment as successful.

[00:11:15]>> Uh you mentioned uh the application was deployed using rolling update, right?

[00:11:20]>> Correct.

[00:11:21]>> Uh so, why didn't you use a canary deployment and instead you used rolling update? Because canary deployment is too much in fashion, no?

[00:11:28]>> Correct.

[00:11:29]>> So, why?

[00:11:30]Yeah.

[00:11:30]>> So, basically the thing is that we are at the stage where we are enhancing the infrastructure.

[00:11:35]So, like in the current project, like we are using the rolling update because like the application was backward compatibility, and the deployment risk was very low. So, it provided us a zero-downtime deployment with minimal operational overhead.

[00:11:47]But let's say if we have to set up a canary deployment for some sort of business-critical application or major releases, we could really implement the canary deployment. It just like we need to just do some sort of basic testing. Like the basic implementation, it will just allow us to allow give us a feature to release a new version to a small percentage of user. Then we can simply monitor the key matrices like the error rate, latency, and then we can simply gradual gradually increase the traffic.

[00:12:12]And if any issue is detected over there, then we can simply roll back into the previous version. So, if if needed, then we can also implement the canary deployment. But we are at the stage where we are enhancing the infrastructure.

[00:12:24]>> Got it. Got it. Makes sense. Yeah.

[00:12:27]So, like while creating CI/CD deployments out there, there you might have faced a lot of challenges.

[00:12:34]Sometimes some havoc, some issue in the configuration. So, let me give you a scenario. Let's say a developer accidentally commits an AWS access key and the secret key to the Git Git repository.

[00:12:49]What And that that's a blunder, no? What immediate actions would you take and how would you prevent this from happening in future?

[00:12:58]>> So, firstly, what I will do, I will just simply disable or rotate the compromised access key so that it cannot be used anymore. And then I will go to the AWS, and then we can we could check the cloud trail to just to identify if any suspicious activity has been done with the those compromised key or not.

[00:13:16]And then for the future, we can integrate some sort of Git secret or Git leak into the CI pipeline to detect whether any of the secret is getting pushed or not before the code is getting merged.

[00:13:26]And then like at the like the best practices is we should store some sort of sensitive credential into the AWS secret manager or we could save it to HashiCorp Vault as well.

[00:13:36]And also, at the same time, we need to educate the developers on secure credential management as a part of the development workflow.

[00:13:44]>> Okay. Okay, makes sense.

[00:13:47]So, sup- let's suppose like by using your CI/CD pipeline or something, uh you deployed a new version of the application, right? The deployment is showing successful, that means the green deployment, right? So, it's green on your side, but the customers are, you know, getting a lot of 502 bad gateway errors.

[00:14:09]Right? So, how would you troubleshoot that?

[00:14:12]>> So, firstly, uh okay, the deployment goes green, but still the users are impacting or getting 502 errors.

[00:14:18]>> Yeah, 502. Yeah, yeah.

[00:14:19]>> So, maybe firstly, I will check the logs of the pod, and maybe I will check whether that pods are up and running or not. Maybe I would also check some sort of readiness and liveness probe as well.

[00:14:29]And then, I will verify whether the service is correctly pointing or forwarding the traffic to the healthy pods or not.

[00:14:34]Then, even though if that is also correct, then I will check the ingress controller or the application at the LB logs to identify whether the from where the 502 is being generated. Then, also I will at the same time, I will I will try to check some sort of application logs for any sort of startup or runtime error.

[00:14:50]Then, like, I will also ensure that the the application is listening on the correct port and the service target port matches the container port that we defined inside the deployment file.

[00:15:00]And even though if the issue is still persistent over there, so my first preference would be to roll back it to the previous stable version to restore the services while investigating the root cause. And we can we could do it onto the lower environment. So, firstly, I will roll roll back it to the previous version, so that at least the end customer would not impact anymore.

[00:15:19]>> Got it. So, like, suppose as you answered, you will check this you will check a lot of things. So, like during your investigation of the checks, you find that the suppose you during your investigation of the checks, you find that the 502 error is caused by the database, right? And the database CPU is consistently above 90% since you made the deployment. And one there's one query that is being executed hundreds of times per minute because of the because of the particular application functionality that's assume. So, how would you solve this?

[00:15:57]>> So, the firstly I would do is to confirm whether the repeated query is actually necessary or it's an application issue.

[00:16:04]Let's say if the data doesn't change frequently like whenever user is requesting that specific query. So, I would introduce Redis as a caching layer. So, instead of hitting the database directly for every request, the application firstly it would go to the Redis and if the data is available it would return the response directly from there. If the data is not present in Redis, it would query the database and then store the result in Redis for the subsequent request. So, like this would significantly reduce the number of database queries. It will also reduce the CPU utilization. So, maybe this is how we could improve the response time for the request.

[00:16:39]>> Okay.

[00:16:41]Okay.

[00:16:42]So, have you worked on Terraform?

[00:16:44]>> Yes, I'm working on Terraform. Yeah.

[00:16:46]>> Okay. So, how would you structure Terraform code for large enterprise project with multiple environments like dev, prod, and stage?

[00:16:56]>> So, we can create a modular structure in which like we can create separate module folder for component like VPC, EKS, RDS, or whatever the services that we are using into our current project. And then there would be a separate environment folder such as dev, stage, prod in which where each environment will call the same module with the different variable values.

[00:17:15]So, it would like also at the same time each environment would have its own back end and state file to keep the interest isolated. And then we could simply integrated that Terraform with a CCD pipeline to so that every infrastructure changes goes through review, validation, and approval being before being getting applied over there.

[00:17:31]So this is what we can do here.

[00:17:33]>> Okay. So while working with Terraform, Terraform being too sensitive and you know, you you need to do your work in a good fashion, right? So you must have faced some kind of you know, issue while issue as in like suppose I'll give you an example. Like what like what would happen if someone manually changes the infrastructure in AWS outside of Terraform? What would what like what happens in that case?

[00:18:01]>> Let's say if they made some sort of changes into the AWS cloud and if we are not having that code into our Terraform, then whenever we will try to run the Terraform plan, the Terraform will detect the drift and shows the difference between the desired state and the current state. So like it depend on the change. Like if the manual change is not part of the Terraform code, Terraform apply may revert it back to the desired state. But let's say if the manual change is needed into our current code right now. So I would firstly review it and then update the Terraform code so that the infrastructure and the code remains in sync. So this is what this is how we can solve it.

[00:18:33]>> Okay.

[00:18:34]So when do you use depends on? Can you just give me an example?

[00:18:39]>> So like I use depends on when I want to make one resource is created only after another resource is ready. So there is a dependency. Over there we could get So if I give you a small example, then let's say we create a private route while creating VPC, right? So I want to ensure that the net gateway because inside the route we attach the net gateway to the routes. So I want to ensure that the net gateway is fully created before the route points to it.

[00:19:04]So in such case I can accidentally use the depends on to guarantee that the creation order.

[00:19:10]>> Okay. Okay.

[00:19:12]So let's come back to AWS maybe just a few questions on AWS.

[00:19:17]What would happen if an entire AWS region goes down and how would you recover your application?

[00:19:25]>> Like we are already deploying our application with the help of multi-region. But even though in our current project since we are working with US based client, so we perform some sort of disaster recovery drills every 3 months to ensure our recovery process actually works. So, there are multiple steps that we need to follow. So, we are keeping a backup of our database like we use the database snapshot. It will automatically copy like it will automatically create the snapshot and there is lambda function which will move the snapshot from one region to another one another one, sorry. And then we also enable the S3 cross region application that is simply CRR so that our application data is already available in the DR region. And since our infrastructure is managed through Terraform, so we can quickly recreate the infrastructure in the secondary region. That won't be an issue much more.

[00:20:09]And once the application is up and ready, like we can simply update the route 53 to route the traffic to the healthy region.

[00:20:15]And we have already tested this DR process multiple times in our current project. So, like right now like we are much confident that we could cover the application like we can recover the application within the agreed RTO or the RPO instead of figuring it out at the run time.

[00:20:31]>> Okay.

[00:20:33]Okay, okay.

[00:20:34]I was just going through your resume. I find one thing very interesting.

[00:20:40]You worked on >> [clears throat] >> HIPAA and GDPR compliances, right?

[00:20:46]>> Correct.

[00:20:47]>> So, what was your exact role in this?

[00:20:50]Can you just mention?

[00:20:51]>> So, basically I was working for a client and they are into the health care sector. So, since they are applying around the data of the patient. So, they need to be a bit more concerned about the security part. So, like currently they They into the Europe under the GDPR compliances. So, when they expanded their business to the US so they also needed to comply with the HIPAA requirement. So, as a part of the initiative, our our focus was on strengthening the infrastructure from a security and compliance perspective. So, if I give you a small example, there were a lot a lot of checklist over there, but if I give you a small example, then uh we ensured the private S3 bucket were accessed through VPC endpoint instead of the public internet.

[00:21:30]Then we are implementing some sort of governance policy in our community cluster to uh to enforce the secure deployment uh standards. Then we make sure that the PI and the PHI data was never written to the application logs by masking or removing the sensitive information from that logs.

[00:21:45]Then we are also like we also enforce the encryption. We also use some sort of least privileges I am access and regularly we review the security and everything, yeah.

[00:21:55]>> Okay. Okay.

[00:21:57]So, let [clears throat] me give you a real-life scenario, right?

[00:22:01]>> Mhm.

[00:22:01]>> So, uh right now FIFA is uh too much trending and the ads and the banners in the FIFA. You must have heard that the in different countries they show different kind of ads or different kind of pages, right? So, >> [clears throat] >> like I would like to question on that in CloudFront, if you want to show different pages based on the user's country, how would you implement that?

[00:22:25]>> Um different pages based on the their country, right?

[00:22:29]>> Yeah.

[00:22:30]In CloudFront.

[00:22:31]>> Oh, CloudFront, right.

[00:22:32]So, there would be multiple ways of doing it. Uh like the firstly, the simplest would be like we can write up a small logic inside the application that will handle the uh logic which will uh move the traffic from one page to different one according to the user's country. But let's if you are specifically going with the CloudFront, so there is a there is one more option that we can do it uh CloudFront provides us the CloudFront function.

[00:22:57]So, we could use the CloudFront functions over there and there is a uh header like we can use uh, viewer country header, like I don't exactly remember the header, but there is some sort of CloudFront viewer country, some sort of header like that.

[00:23:07]>> Yeah.

[00:23:08]>> And then we can write up a small CloudFront function, and that it will automatically inspect that specific header, and rewrite the request URI or the route the users to a different uh, content over there. Let's say if the user is coming from India, so it will uh, access on the server page like with some sort of landing page. So we could create a CloudFront function. Then there are multiple ways of doing it. So even though we could implement some sort of Lambda function through which Lambda@Edge which will handle this specific logic. So there are multiple ways of doing it.

[00:23:36]>> Mhm. Okay.

[00:23:38]Okay.

[00:23:40]So uh, have you worked in multi-cloud environments?

[00:23:44]>> Yeah, so like uh, like within this uh, 7 or 6 years of my cloud journey, so I got a chance to work on different different projects. I worked on AWS, GCP, Azure. I did a migration of like multiple times I did a migration of from GCP to AWS as well. I did a lot of uh, I have created a lot of pipeline in terms of Azure DevOps as well. So like it's it's like they have worked on almost all the cloud provider, but most extensively on the side of AWS.

[00:24:12]>> Most ex- And I also see that you have uh, AWS certification, right?

[00:24:18]>> Correct. Correct. So yeah, it's a professional one.

[00:24:21]>> Yeah, it's a professional one. So when did you get this?

[00:24:25]>> So I did that in the year somewhere around 2023. So maybe it would be in some sort of expiration state or maybe like it would be a bit closer to the expiration.

[00:24:35]>> Yeah, it's like fine. It's like Okay.

[00:24:39]Uh, let me go through your resume once more.

[00:24:43]Just allow me a second.

[00:24:45]Okay. So I was Yeah, I was asking that Mhm. Like, [clears throat] can you just tell me more about Kibana, like a little about Kibana?

[00:24:55]>> So right now we have used the Kibana in our current project, so it is helping us to uh, strengthening the security of the cabin.

[00:25:01]So, let me give you a small example. So, we have like it's just a YML file in which you could create any sort of policy that you wanted to use in your Kubernetes cluster. In our case, like I'm using like this is one of the policy that we are using.

[00:25:12]So, that we have mentioned the account ID of the AWS account account AWS account ID. So, that whatever the image that is present into that specific AWS account ID, only those deployment would be able to deploy. They won't they won't be able to deploy any sort of basic like Ubuntu, Nginx, Python. They won't be able to deploy anything. Only the images that is present on that specific AWS account ID. So, this is how we are restricting to use secure images that is present on our AWS account. Yeah.

[00:25:40]>> Okay.

[00:25:42]Okay, makes sense. Um yes, I guess I covered most of the points.

[00:25:49]Um yes, so yep, I'm good from my side.

[00:25:53]Um maybe if you want to have some questions, please spam them.

[00:25:56]>> So, actually I don't have any specific question, but just wanted to understand that what kind of project or business domain that your organization is working on and also maybe some bit idea about the tools and the technology.

[00:26:07]>> Okay. So, in our organization, we work with multiple clients across multiple domains such as, you know, health care, fintech, a lot of e-commerce clients out there, retail clients out there. So, the technology stack varies like depending on the client based on the client requirement. But most of our project revolve around AWS, Kubernetes, Terraform, Docker, GCP, and monitoring tools.

[00:26:31]So, as a DevOps engineer, you'll likely work on different projects and, you know, get exposure to multiple technologies over the same time. Yeah.

[00:26:41]I guess I answered your question.

[00:26:43]>> Yeah, yeah. Yep, perfect. I think so, yeah, that is the only question that I have right now.

[00:26:48]>> Okay. Okay.

[00:26:50]So, nice meeting you, Jadeja. I'll just pass on my feedback to the HR. She'll get back to you.

[00:26:55]>> Sure. Thank you so much. Thank you so much. Yeah, have a nice day.

[00:26:58]>> Thank you.